Advanced usage overview¶

The standard Neptoon workflow is configuration-first: sensor YAML describes the station and input data, process YAML describes scientific processing, and ProcessWithConfig runs the complete pipeline. Advanced use begins when an application needs to inspect intermediate state, compose lower-level APIs, register a scientific extension, or integrate Neptoon into a larger system.

This page maps those extension points and explains when to use configuration, the public Python API, or a source-code contribution.

Choose the right level of control¶

Goal	Recommended interface
Process a station reproducibly from raw files	Sensor/process YAML with `ProcessWithConfig` or the `neptoon stationary` CLI.
Inspect and control individual processing stages	`DataHubFromConfig` followed by `CRNSDataHub` methods.
Start from an already prepared pandas table	Construct `CRNSDataHub` directly.
Run one scientific equation without a hub	Import a public function from `neptoon.corrections`, `neptoon.calibration`, or `neptoon.products`.
Change source column names	Configure `time_series_data.key_column_info` in YAML.
Change canonical runtime/output labels	Use `ColumnInfo.relabel()` with process-wide-state precautions.
Add a site-specific correction in application code	Subclass `Correction` and register it with the correction factory.
Add a reusable correction, provider, product, or schema field	Contribute it to the source tree with tests and documentation.
Process many station configurations together	Use the multi-sensor workflow.
Convert one prepared observation	Use the single-row workflow.
Download external data without running the CRNS pipeline	Use the standalone NMDB or provider APIs.

The API reference lists all public imports and links to generated signatures.

Core architecture for advanced users¶

The developer architecture page provides the canonical diagrams and an interactive, searchable dependency map. The summary below focuses on the contracts most relevant to advanced API use.

A complete station run can be viewed as four cooperating layers:

Configuration models
        |
        v
Raw/external data acquisition and formatting
        |
        v
CRNSDataHub + shadow flag DataFrame
        |
        v
Scientific corrections, products, figures, reports, and exports

Configuration models¶

Pydantic models validate sensor and process YAML before processing begins. Unknown keys are rejected in the main user-facing models so spelling errors do not silently change behavior. Use ConfigurationManager when an application needs to load, inspect, or supply configuration objects programmatically.

`CRNSDataHub`¶

CRNSDataHub is the managed state of a station workflow. It holds:

the CRNS time-series DataFrame;
the parallel quality-flag DataFrame;
sensor and calibration metadata;
the correction factory and selected correction sequence;
storage configuration and output helpers;
preparation and product methods that keep data and flags aligned.

Do not bypass the hub by mutating internal frames between managed operations unless the application also maintains index, column, flag, unit, and metadata consistency.

Canonical columns and units¶

Internal processing expects canonical labels from ColumnInfo and the units documented in Internal units. Raw logger names are mapped in YAML; they do not need to match the canonical output schema.

Use Changing expected column names only when a Python application truly needs runtime relabelling.

Programmatic workflow patterns¶

Run the complete YAML pipeline¶

from neptoon.workflow import ProcessWithConfig

processor = ProcessWithConfig(
    path_to_sensor_config="sensor.yaml",
    path_to_process_config="process.yaml",
)
processor.run_full_process()

if not processor.completed_successfully:
    raise RuntimeError(processor.exit_reason)

processed = processor.data_hub.crns_data_frame
flags = processor.data_hub.flags_data_frame

This is the preferred application API when the complete operation can be expressed through configuration. It preserves the documented operation order, metadata, audit behavior, and output semantics.

Create a hub and process step by step¶

from neptoon.workflow import DataHubFromConfig

hub = DataHubFromConfig(
    path_to_sensor_config="sensor.yaml",
).create_data_hub()

hub.attach_nmdb_data(station="JUNG")
hub.prepare_static_values()
# Register QA checks and select corrections as required.
# Then run correction, calibration, products, figures, and saving deliberately.

Use this pattern in notebooks and interactive research where intermediate data must be inspected. The caller is responsible for invoking operations in a scientifically valid order. The complete method list is documented under CRNSDataHub.

Construct a hub from a prepared DataFrame¶

import pandas as pd

from neptoon import CRNSDataHub
from neptoon.columns import ColumnInfo

frame = pd.DataFrame(
    {
        str(ColumnInfo.Name.EPI_NEUTRON_COUNT_CPH): [1050.0, 1032.0],
        str(ColumnInfo.Name.AIR_PRESSURE): [1008.2, 1008.7],
        str(ColumnInfo.Name.AIR_TEMPERATURE): [12.4, 12.1],
        str(ColumnInfo.Name.AIR_RELATIVE_HUMIDITY): [71.0, 73.0],
    },
    index=pd.date_range(
        "2026-01-01",
        periods=2,
        freq="1h",
        tz="UTC",
    ),
)

hub = CRNSDataHub(frame)

The input must already follow Neptoon's schema, timestamp, and unit expectations. Keep validation enabled unless a controlled test specifically requires bypassing it.

Advanced input discovery and parsing¶

raw_data_parse_options supports more than a single local CSV directory:

one file, directory, ZIP archive, or TAR archive;
an ordered list of local data_location entries;
recursive or direct-child-only local directory search;
prefix, suffix, include-regex, and exclude-regex filters;
deterministic multi-file parsing;
malformed-row, extra-column, and schema-mismatch policies;
optional parallel parsing;
optional SFTP acquisition with its own remote recursion setting;
per-column ranges, dtypes, and temporal aggregation hints.

Example:

raw_data_parse_options:
  data_location:
    - ./raw/2026-01
    - ./raw/2026-02
  recursive: false
  filename_include_regex: 'station_A/.+\\.csv$'
  invalid_data_policy: drop_required
  extra_columns_policy: trim

Local raw_data_parse_options.recursive and sftp.recursive control different searches. See Importing data and Sensor configuration.

For unknown logger layouts, the Config wizard can detect a file structure and generate a starter sensor YAML.

External-data integrations¶

Advanced workflows can attach or retrieve external observations independently of raw sensor parsing.

NMDB¶

The public standalone downloader returns a UTC-indexed DataFrame and reuses the existing cache:

from neptoon import NMDBDataDownloader

nmdb = NMDBDataDownloader.download(
    station="JUNG",
    start_date="2025-01-01",
    end_date="2025-02-01",
    resolution="10Min",
)

The hub and configuration workflows use the same NMDB infrastructure for incoming-neutron correction.

DWD and international meteorology¶

Sensor configuration can attach DWD data or provider data from supported international meteorological services. Provider output is mapped into shared Neptoon weather columns and source metadata is retained where available.

See External data and the provider sections in the API reference.

TimeIO / SensorThings¶

TimeIO can act as an input source and can receive selected processed outputs as SensorThings observations. Treat credentials as runtime secrets; rewritten configuration files omit protected token values.

See TimeIO / SensorThings.

Time-series preparation¶

Neptoon exposes lower-level preparation APIs for:

timestamp validation and time-zone conversion;
completion of missing timestamps;
missing-value interpolation;
timestamp alignment;
temporal aggregation;
rolling and Savitzky-Golay smoothing;
rate-first handling of absolute neutron counts with explicit live time;
Kalman filtering of corrected neutron counts.

The order matters. In the standard workflow, raw counts are formatted before correction, correction precedes soil-moisture conversion, and temporal features are placed where their uncertainty and scientific semantics are defined.

Do not copy the order from an isolated helper example into production without checking Workflow description and the process configuration documentation.

Quality assessment and flags¶

Quality flags are stored in a shadow DataFrame aligned with the measurement DataFrame. Advanced users can construct QualityCheck objects, register them with a DataQualityAssessor, create flags, and apply those flags through the hub.

Important invariants:

data and flag indexes must remain aligned;
not every metadata or diagnostic column is flaggable;
removing or renaming a data column may require the corresponding flag-table update;
applying flags may mask values, so inspect flags before and after application when debugging.

See Data quality checks and the neptoon.quality_control reference.

Corrections and scientific extension points¶

Built-in correction types use a factory/builder architecture. For normal use, select correction methods in process YAML. For experimental application code, register a custom Correction subclass at runtime.

See Write your own corrections for runtime registration and Adding a correction to neptoon for a permanent contribution.

Direct scientific functions are also public. They are appropriate for controlled calculations and validation, but they do not automatically provide:

pipeline ordering;
canonical DataFrame column management;
shadow flags;
input unit conversion;
uncertainty propagation outside the called function;
audit metadata or output files.

Use a hub or full workflow when those responsibilities matter.

Calibration and soil-moisture products¶

Advanced calibration workflows can supply calibration samples, choose a method-specific calibration profile, resolve a parameter set, or invoke lower- level calibration functions directly.

The active calibration parameter is resolved from the selected conversion method and parameter set before legacy fallback values are considered. Preserve that relationship when building configurations programmatically.

Relevant guides:

Relevant APIs are indexed under Calibration APIs and Soil-moisture product APIs.

Kalman parameter estimation and filtering¶

The filtering package supports two related tasks:

estimating Kalman parameters from representative corrected-neutron data;
applying the configured filter during normal processing.

The filter writes a separate final corrected-neutron column and can optionally write diagnostic columns. The original correction-only signal remains available. Parameter estimation should be performed on representative periods and reviewed before deploying the resulting parameters to routine processing.

See Kalman parameter estimation and the neptoon.filtering reference.

Output composition¶

Advanced output configuration can combine several independent features:

CSV and/or JSON table output;
full, selected, or both processed-data schemas;
explicit ordering with data_storage.output_columns;
static-pivot compact tables for fully constant columns;
PDF report generation and report-only downsampling;
figures and additional plots;
YAML/audit artifacts and archive creation.

Selected output and static pivot solve different problems:

data_storage:
  output_mode: both
  output_columns:
    - date_time
    - smv
    - crns_measurement_depth_cm

  static_pivot_enabled: true
  static_pivot_mode: sidecar

output_columns is an explicit user-selected schema. Static pivot automatically extracts constant columns into a static-values table. They may be enabled together.

See Saving data and PDF output.

Multi-sensor and single-row workflows¶

Multi-sensor processing¶

Use the multi-sensor runner when several sensor/process configurations should be processed as a batch with harmonised comparison outputs. Per-sensor failures are separated so one failing station does not have to hide the outcome for all other stations.

See Multi-sensor processing.

Single-row conversion¶

Use the single-row workflow for an already prepared observation that must be converted without a complete historical file. It supports NMDB-backed or fixed incoming-intensity operation and returns structured conversion results.

See Single-row conversion.

Errors, logging, and application integration¶

Expected user-facing failures use the public neptoon.exceptions hierarchy. Catch the narrowest recoverable exception in an application; do not suppress unexpected programming errors.

Optional progress output can be controlled through the public logging/output helpers. Library integrations should prefer normal return values and structured exceptions over parsing console text.

processor.run_full_process()

if not processor.completed_successfully:
    # Expected configuration, data, external-service, calibration, and output
    # failures are converted to a concise user-facing exit reason.
    handle_expected_failure(processor.exit_reason)

Lower-level APIs may raise the public exception subclasses directly. Provider- specific DWD, NMDB, and external-meteorology errors are also instances of NeptoonExternalDataError; filesystem failures from output writers are NeptoonOutputError.

from neptoon.exceptions import (
    NeptoonExternalDataError,
    NeptoonInputDataError,
    NeptoonOutputError,
)

try:
    hub = create_or_update_hub()
    hub.save_data()
except NeptoonInputDataError as exc:
    handle_invalid_input(exc)
except NeptoonExternalDataError as exc:
    retry_or_use_cached_data(exc)
except NeptoonOutputError as exc:
    choose_another_output_location(exc)

See the Errors and Logging and user output sections of the API reference.

Secrets and reproducible configurations¶

SFTP passwords, TimeIO tokens, and supported meteorological API keys use Pydantic secret types. Runtime clients reveal a secret only at the connection boundary, and rewritten YAML omits protected values.

For reproducible deployments:

inject credentials from environment-specific configuration;
do not commit secrets to sensor YAML;
keep scientific settings in version-controlled YAML;
retain generated configuration and audit artifacts with results;
record the Neptoon version and active calibration profile.

Configuration, API, or contribution?¶

Use this decision rule:

Change	Best home
Different station values, paths, columns, QA limits, correction methods, or outputs	YAML configuration
Different orchestration using existing public objects	Application code using the public API
Temporary experimental correction	Runtime custom-correction registration
New reusable scientific theory	`neptoon.corrections` plus factory/config integration
New external meteorological service	Provider implementation plus registry/config integration
New canonical output product	`ColumnInfo`, hub/product code, flags, saving, docs, and tests
New user-facing YAML option	Pydantic model, workflow wiring, examples, docs, and tests
Private one-off presentation label	Rename a copy of the final DataFrame

When changing source code, follow Develop with Neptoon and the repository conventions under agent/conventions/.

Stability and compatibility¶

Neptoon follows a stable 1.x release line, in which breaking configuration changes should be limited to major versions. Public package exports and documented configuration fields are the safest integration points, but scientific models, provider behavior, and lower-level implementation modules may evolve.

For application code:

import from documented package-level paths where available;
pin an explicit Neptoon version;
validate output columns and units at the application boundary;
keep regression datasets for scientifically important workflows;
review the changelog before upgrading;
avoid relying on private names beginning with _.

Advanced-user reference map¶

Area	Guide	API reference
Canonical and custom column labels	Changing expected column names	Column registry and aliases
Custom corrections	Write your own corrections	Correction APIs
Hub operations	CRNSDataHub	Hub and column APIs
Raw import	Importing data	Raw-data input APIs
External data	External data	External-data APIs
QA	Data quality checks	Quality-control APIs
Calibration	Sensor calibration	Calibration APIs
Filtering	Kalman parameter tool	Filtering and Kalman APIs
Output	Saving data	Output and persistence APIs
Multi-sensor	Multi-sensor processing	Workflow APIs
Single-row	Single-row conversion	Workflow APIs
Complete public surface	—	API reference