Multi-sensor processing¶

Neptoon can process several independent sensor/process configuration pairs and then create one combined comparison report. The multi-sensor workflow keeps each sensor run isolated: every sensor keeps its own raw neutron columns, local meteorological inputs, QA rules, calibration and correction settings. After the single-sensor runs finish, Neptoon combines the processed data frames with sensor-prefixed column names and creates a combined PDF report.

Use this when several CRNS sensors should be compared on the same plots while still allowing shared reference data such as NMDB, DWD or public meteorological providers.

A guided validation and opt-in run is available in examples/notebooks/04_multi_sensor_processing.ipynb. It uses only the sample files tracked in this repository.

Reusing existing YAML files¶

The multi-sensor workflow intentionally reuses the existing single-sensor YAMLs. A sensor_config must be a normal config: sensor file and a process_config must be a normal config: process file. You do not need to merge sensor YAMLs or rewrite working single-sensor configurations.

If all sensors use the same processing method, define exactly one process config and omit the rest. The single process config can be written either as top-level process_config or as shared.process_config:

config: multi_sensor
process_config: configuration_files/Hydroinnova_processing.yaml

sensors:
  - name: Hydroinnova_UP26
    sensor_config: configuration_files/Hydroinnova_CRS2Probe.yaml
  - name: Hydroinnova_CRSProbe
    sensor_config: configuration_files/Hydroinnova_CRS4Probe.yaml

If exactly one process_config appears anywhere in the file, Neptoon treats it as the default for sensors that do not specify one. When several process configs are present, sensors without their own process_config use the shared/default one. If no shared/default exists and at least one sensor is missing a process_config, validation fails early.

Use per-sensor process configs only for sensors that really need different processing settings:

config: multi_sensor

shared:
  process_config: configuration_files/Hydroinnova_processing.yaml
  save_location: ./multi_sensor_outputs
  cache_dir: ./multi_sensor_outputs/cache
  create_individual_reports: false
  continue_on_error: true
  # Keep processor objects only for debugging/advanced API inspection.
  retain_processors: false

sensors:
  - name: Hydroinnova_UP26
    sensor_config: configuration_files/Hydroinnova_CRS2Probe.yaml

  - name: StyxHC
    sensor_config: configuration_files/StyxHC_sensor_dwd.yaml
    process_config: configuration_files/StyxHC_processing.yaml

Output isolation and combined outputs¶

By default, individual sensor outputs are written below <shared.save_location>/individual/<sensor-name>/. This avoids folder-name conflicts if several sensor YAMLs have the same sensor_info.name, or if append_timestamp_to_folder_name is disabled. Set shared.isolate_individual_outputs: false only if you deliberately want all individual runs to write directly below shared.save_location.

The combined CSV uses prefixed column names such as Hydroinnova_UP26__corrected_epithermal_neutrons and StyxHC__smv to avoid collisions between sensors. The combined report plots comparable columns from each successful sensor run on the same page.

Combined-output harmonisation¶

Harmonisation affects only the combined CSV/report. Individual sensor runs and individual outputs are not modified.

The default mode is auto:

all combined outputs are put on one clock, UTC by default;
the combined time window is the union of all successful sensor runs, not the intersection;
missing sensor periods remain visible as gaps;
all combined series are resampled to the coarsest native resolution found among the sensor result time series;
raw neutron comparison plots use counts per hour (cph) by default, using elapsed-time/time-step information or timestamp spacing when conversion is possible; epithermal and thermal count streams remain separate labelled lines;
muon count streams, when present, are detected separately and can be plotted on the muons page instead of being mixed into neutron plots;
different NMDB stations, DWD stations or external meteo sources remain separate sensor-prefixed series;
common meteorological display-unit mismatches are normalized for combined outputs only: Pa-looking pressure to hPa, Kelvin-looking temperature to degrees Celsius, fractional relative humidity to percent, and daily MJ/m2-like solar radiation to mean W/m2 for the solar comparison plot.

Example:

harmonization:
  mode: auto
  time:
    timezone: UTC
    report_window: union
    report_resolution: coarsest
  neutron_counts:
    comparison_unit: cph
  reference_data:
    keep_distinct_sources: true
  meteo:
    normalize_pressure_to_hpa: true
    normalize_temperature_to_celsius: true
    normalize_relative_humidity_to_percent: true
    solar_radiation_comparison_unit: w_m2
    convert_daily_solar_energy_to_mean_flux: true

To force a fixed report period or resolution, use explicit manual settings:

harmonization:
  mode: auto
  time:
    report_window: explicit
    manual_start: 2024-01-01T00:00:00Z
    manual_end: 2024-02-01T00:00:00Z
    manual_resolution: 1h

harmonization.mode: off keeps the legacy native-time behaviour for combined outputs. strict can be used in CI-like settings when enabled plot pages should fail if a sensor lacks the required series.

Reference data¶

shared.cache_dir is applied to DWD and external_meteo entries that do not already set their own cache directory. This makes repeated downloads avoidable when several sensors use the same reference weather source. NMDB already uses a shared cache by default.

The cache does not force reference sources to be the same. If one sensor uses NMDB: JUNG and another uses NMDB: OULU, both series remain separate in the combined outputs. The same applies to different DWD, Météo-France, NOAA-USCRN, NOAA-GHCN or other external-provider stations.

Example with comparison plots¶

config: multi_sensor
process_config: configuration_files/Hydroinnova_processing.yaml

shared:
  save_location: ./multi_sensor_outputs
  isolate_individual_outputs: true
  cache_dir: ./multi_sensor_outputs/cache
  create_individual_reports: false
  continue_on_error: true

sensors:
  - name: Hydroinnova_UP26
    sensor_config: configuration_files/Hydroinnova_CRS2Probe.yaml
  - name: StyxHC
    sensor_config: configuration_files/StyxHC_sensor_dwd.yaml
    process_config: configuration_files/StyxHC_processing.yaml

harmonization:
  mode: auto
  time:
    report_window: union
    report_resolution: coarsest
  neutron_counts:
    comparison_unit: cph
  meteo:
    solar_radiation_comparison_unit: w_m2

report:
  create_combined_report: true
  save_combined_data: true
  output_name: combined_sensors
  plots:
    raw_neutrons: true
    corrected_neutrons: true
    muons: true
    soil_moisture: true
    air_pressure: true
    precipitation: true
    solar_radiation: true

Run from Python:

from neptoon.workflow import run_multi_sensor_process

result = run_multi_sensor_process("examples/configuration_files/multi_sensor_processing.yaml")
print(result.combined_report_path)

Run from the CLI:

neptoon multi examples/configuration_files/multi_sensor_processing.yaml

Override the multi-sensor worker count at runtime without editing the YAML:

neptoon multi examples/configuration_files/multi_sensor_processing.yaml --workers 4

Use --no-parallel to force a sequential run even when shared.parallel is enabled in the YAML.

Combined JSON output¶

The combined multi-sensor table is written as CSV by default. You can also write JSON, or JSON only, from the report section:

report:
  save_combined_data: true
  save_combined_csv: true
  save_combined_json: true
  json_orient: records
  json_date_format: iso

For JSON-only combined output:

report:
  save_combined_data: true
  save_combined_csv: false
  save_combined_json: true