Workflows and data hub¶
Stable reference pages for the main package exports, end-to-end workflows, and the managed CRNS data hub.
This committed category page is the canonical mkdocstrings location, so links remain valid in GitLab and on the rendered site. The generated module index mirrors the Python package tree and links back to these canonical API sections without rendering objects twice.
neptoon¶
neptoon ¶
neptoon.runtime¶
runtime ¶
Per-run runtime state for configured Neptoon processing.
The configured workflow previously changed several module or third-party globals.
This module keeps mutable state on an explicit :class:ProcessingContext and
activates it with :mod:contextvars, so nested calls use the correct run-local
settings. The external magazine package still exposes a process-global
switch; Neptoon scopes and serialises that switch only while decorated report
functions execute.
__all__
module-attribute
¶
__all__ = ['ColumnRegistry', 'ProcessingContext', 'RuntimeCache', 'activate_processing_context', 'get_processing_context', 'create_magazine_reporting_decorator', 'create_magazine_reporting_figure']
ColumnRegistry
dataclass
¶
RuntimeCache
dataclass
¶
ProcessingContext
dataclass
¶
ProcessingContext(column_labels=ColumnRegistry(), output_enabled=False, report_enabled=False, cache=RuntimeCache(), status_callback=None)
Explicit mutable state owned by one configured processing run.
column_labels
class-attribute
instance-attribute
¶
column_labels = field(default_factory=ColumnRegistry)
get_processing_context ¶
Return the context active in the current thread/task, if any.
activate_processing_context ¶
Activate context and restore the previous context afterwards.
create_magazine_reporting_decorator ¶
Context-aware replacement for Magazine.reporting.
create_magazine_reporting_figure ¶
Context-aware replacement for Magazine.reporting_figure.
neptoon.server_runtime¶
ServerRunMonitor is the opt-in operational boundary for scheduled workflows.
It owns the exclusive run lock, atomic status document, freshness checks, soft
stage deadline, and graceful SIGTERM status transition. Application wrappers
should use the status file as the completion signal and still configure an
external hard timeout for calls that do not return control to Python.
server_runtime ¶
Machine-readable run monitoring for unattended Neptoon processing.
ServerRunMonitor
dataclass
¶
ServerRunMonitor(root, sensor_name, config, run_id=(lambda: hex)(), started_monotonic=monotonic(), started_at=_get_utc_now(), current_stage='initialising', previous_status=dict(), lock_acquired=False, details=dict(), previous_sigterm_handler=None, terminal_status_written=False)
Own one run lock and an atomically updated status document.
started_monotonic
class-attribute
instance-attribute
¶
update ¶
Update the machine-readable status and enforce the soft deadline.
finish_error ¶
Write a terminal status with a stable nested error payload.
Existing top-level error fields are retained for compatibility while
the nested error object matches the language-neutral Neptoon error
contract used by workflow and service adapters.
validate_input_frame ¶
Check row count, freshness and incremental progress for server runs.
neptoon.hub.crns_data_hub¶
crns_data_hub ¶
CRNSDataHub ¶
CRNSDataHub(crns_data_frame, flags_data_frame=None, sensor_info=None, quality_assessor=None, validation=True, calibration_samples_data=None, data_storage=None, keep_raw_data_copy=False, enable_output=False, column_dtypes=None, column_aggregations=None, processing_context=None)
The CRNSDataHub is used to manage the time series data throughout the processing steps. Some key features:
- It stores a DataFrame for a site
- As we progress through the steps, data can be added to the DataFrame and the shadow DataFrame's updated.
Raw data is checked against the RawDataSchema which is a first line of defense against incorrectly formatted tables. Should a fail happen here data must be either reformatted using one of the provided routines or manually formatted to match the standard.
Inputs to the CRNSDataHub.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
crns_data_frame
|
DataFrame
|
CRNS data in a dataframe format. It will be validated to ensure it has been formatted correctly. |
required |
flags_data_frame
|
DataFrame
|
Existing quality-flag table aligned with |
None
|
sensor_info
|
SensorInfo
|
Sensor metadata associated with the data. |
None
|
quality_assessor
|
DataQualityAssessor
|
Neptoon quality assessor used to create and retrieve data-quality flags. |
None
|
validation
|
bool
|
Toggle for whether to have continued validation of data tables during processing (see data_management>data_validation_tables.py for examples of tables being validated). These checks ensure data is correctly formatted for internal processing. |
True
|
calibration_samples_data
|
DataFrame
|
The sample data taken during the calibration campaign. |
None
|
data_storage
|
DataStorageConfig
|
Data storage configuration from the sensor config. When provided, save_data() uses save_location as the default output directory instead of cwd(). |
None
|
keep_raw_data_copy
|
bool
|
Store an immutable copy of the input data for debugging or audit workflows. Disabled by default because large CRNS tables can otherwise be duplicated in memory at hub creation. |
False
|
enable_output
|
bool
|
Print optional user-facing progress messages from core methods. Disabled by default for library-style use. |
False
|
processing_context
|
ProcessingContext
|
Explicit run-local state shared with a configured workflow. |
None
|
processing_context
instance-attribute
¶
processing_context = processing_context or ProcessingContext(output_enabled=bool(enable_output))
column_dtypes
property
¶
Configured per-column dtype hints parsed from YAML short syntax.
column_aggregations
property
¶
Configured per-column temporal aggregation hints.
enable_output
property
writable
¶
Whether optional user-facing output is enabled for this hub.
raw_data
property
¶
Return the optional immutable input-data copy.
Raw-data copies are opt-in to avoid duplicating large time-series tables for normal processing runs.
quality_events_data_frame
property
¶
Return row-level QA events accumulated across processing stages.
quality_actions_data_frame
property
¶
Return per-value QA actions used when masking processed data.
validate_dataframe ¶
Validates the dataframe against a pandera schema See data_validation_table.py for schemas.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
schema
|
str
|
The name of the schema to use for the check. |
required |
complete_time_series ¶
complete_time_series(target_frequency='auto', create_missing_timestamps=True, duplicate_timestamp_policy='first', defer_grid_creation=False)
Insert missing timestamp rows on a regular target grid.
attach_nmdb_data ¶
attach_nmdb_data(station='JUNG', new_column_name=str(INCOMING_NEUTRON_INTENSITY), resolution='60', nmdb_table='revori', reference_value=None, request_timeout=30, request_retries=0, proxies=None, try_without_proxy=True, allow_cache_fallback=True, max_nearest_timedelta='3D')
Utilises the NMDBDataAttacher class to attach NMDB incoming intensity data to the crns_data_frame. Collects data using www.NMDB.eu
See NMDBDataAttacher documentation for more information.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
station
|
str
|
The station to collect data from, by default "JUNG" |
'JUNG'
|
new_column_name
|
str
|
The name of the column were data will be written to, by default "incoming_neutron_intensity" |
str(INCOMING_NEUTRON_INTENSITY)
|
resolution
|
str
|
The resolution in minutes, by default "60" |
'60'
|
nmdb_table
|
str
|
The table to pull data from, by default "revori" |
'revori'
|
reference_value
|
float
|
Fixed positive reference intensity for the neutron monitor. If omitted, the station-specific value from Neptoon's NMDB reference registry is used. |
None
|
Report
Neutron monitor data from NMDB.eu. Station: {station_label}; temporal resolution: {resolution} min.
attach_dwd_data ¶
attach_dwd_data(latitude, longitude, station='nearest', fill_pressure=True, fill_temperature=True, fill_relative_humidity=True, include_precipitation=False, include_solar_radiation=False, precipitation_daily_fallback=True, precipitation_product=None, prefer_dwd_values=False, max_nearest_timedelta='1h', max_distance_km=None, request_timeout=30, request_retries=0, cache_dir=None, use_cache=True, required=False)
Attach or fill meteorological data from a DWD weather station.
Report
Meteorological boundary data from the German Weather Service (DWD) were used for {requested_variables}. Selected station(s): {selected_station_label}.
attach_external_meteo_data ¶
Attach or fill public non-DWD meteorology.
Preferred usage is to pass an ExternalMeteoAttachmentConfig. Keyword
arguments remain supported for backwards compatibility with the former
long signature.
Supported providers include NOAA station products, Meteostat, Open-Meteo, E-OBS, MeteoSwiss, KNMI, Météo-France, RMI Belgium, VMM Waterinfo, the Met Office DataHub, FMI, IPMA Open Data, ECCC GeoMet, BoM, GeoSphere Austria, MET Norway Frost, DMI Open Data, CHMI and ARSO.
Report
External meteorological data from {provider_label} were harmonised with the CRNS time series. Selected source: {selected_station_label}. Variables requested: {requested_variables}.
register_column_aggregations ¶
Declare how named columns aggregate to a coarser output resolution.
Aggregating to a coarser resolution is a normal configured step, and columns attached after the hub was built travel in the same frame, so they need a way to state their method. A flag is never averaged: a marker aggregates as a logical OR and an observation age takes the age of the oldest contributing observation, while a class code takes the mode because the mean of two class codes is a third, unrelated class.
attach_remote_sensing_data ¶
Attach gridded Earth-observation reference series.
Suitability is checked against the product catalog before anything is downloaded, so a station outside a product's extent fails with an error naming the products that do cover it, rather than producing a column of nodata that reads as measurement.
The attached columns are reference series. They are written, reported and audited, and they enter no neutron correction, no N0 calibration and no soil-moisture product.
Report
Remote-sensing reference data were checked against the product catalog and joined to the CRNS time series. {selection_label} Catalog version: {catalog_version}. Requested variables: {requested_variables}.
attach_external_soil_moisture_data ¶
attach_external_soil_moisture_data(sensor_config, processing_config, site_bulk_density_g_cm3=None, site_mean_pressure_hpa=None, horizontal_footprint_model=None)
Attach independently logged soil-moisture reference time series.
Every configured file is parsed and aggregated independently before the generated source and derived columns are joined to the current CRNS time index. Imported values are normalised to m3/m3 fractions.
Report
External soil-moisture reference series were normalised, aggregated independently and joined to the final CRNS time axis.
interpolate_missing_values ¶
interpolate_missing_values(columns, method='time', max_interpolation_hours=12, missing_column_policy='skip', non_numeric_policy='skip')
Interpolate missing values for explicitly configured columns.
Leading/trailing gaps are not extrapolated. Consecutive missing-data
gaps longer than max_interpolation_hours remain missing.
add_quality_flags ¶
Add QualityChecks to undertake on the dataframe
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
custom_flags
|
QualityAssessmentFlagBuilder
|
user can build a QualityAssessmentFlagBuilder with checks and attach this as a whole, by default None |
None
|
add_check
|
Check
|
user can add individual Checks, or a list of Checks. These will be then added to the QualityAssessmentFlagBuilder, by default None |
None
|
apply_quality_flags ¶
Flags data based on quality assessment. A user can supply a QualityAssessmentFlagBuilder object that has been custom built, they can flag using the config file (if supplied), or they can choose a standard flagging routine.
Everything is off by default so a user must choose.
summarize_quality_events ¶
Return row-level QA events, optionally limited to selected columns.
summarize_quality_flags ¶
Return counts of non-UNFLAGGED QA flags by data column.
The summary is intentionally compact and user-facing: it reports how many values were flagged per column/flag class, and the percentage of available rows that this represents. Empty summaries mean no values were flagged.
select_correction ¶
select_correction(correction_type='empty', correction_theory=None, reference_pressure_value=None, input_biomass_units=None, above_ground_biomass_column_name=None, **correction_kwargs)
Method to select corrections to be applied to data.
Individual corrections can be applied using a CorrectionType and CorrectionTheory. If a user assigns a CorrectionType without a CorrectionTheory, then the default correction for that CorrectionType is applied.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
correction_type
|
CorrectionType
|
A CorrectionType, by default "empty" |
'empty'
|
correction_theory
|
CorrectionTheory
|
A CorrectionTheory, by default None |
None
|
reference_pressure_value
|
float
|
Site reference pressure for the pressure correction. |
None
|
input_biomass_units
|
str
|
Units of the biomass input column, for the above-ground biomass correction only. |
None
|
above_ground_biomass_column_name
|
str
|
Column carrying the biomass input, for the above-ground biomass correction only. The source is a configuration decision, a column from the raw file, a static site value, or the column derived from a satellite product, so it is named here rather than assumed. |
None
|
**correction_kwargs
|
Any
|
Correction-specific parameters forwarded to the factory
(e.g. |
{}
|
correct_neutrons ¶
Create correction factors and corrected epithermal neutrons.
reuse_existing_factor_columns is reserved for replay layers that
have independently established that selected correction factors remain
valid. Normal processing leaves it unset and recalculates all factors.
smooth_data ¶
smooth_data(column_to_smooth, smooth_method='rolling_mean', window='12h', alignment='centered', min_proportion_good_data=0.7, poly_order=4, auto_update_final_col=True, insufficient_data_policy='raise')
Applies a smoothing method to a series of data in the crns_data_frame using the SmoothData class.
A column_to_smooth attribute must be supplied, and should be
written using the "str(ColumnInfo.Name.COLUMN)" format. The two
most likely to be used are:
- str(ColumnInfo.Name.SOIL_MOISTURE)
- str(ColumnInfo.Name.EPI_NEUTRONS)
When optional parameters are omitted, the rolling mean uses a 12-hour
centered window. Use alignment="trailing" for causal processing;
centred is accepted as an input alias for centered.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
column_to_smooth
|
str(VALUE)
|
The column in the crns_data_frame that needs to be smoothed. |
required |
alignment
|
(centered, centred, trailing)
|
Rolling-window placement. |
"centered"
|
Report
The final time series {column_to_smooth} was smoothed with {smooth_method} using a {window} {alignment} window to suppress high-frequency counting noise while preserving the timestamp grid.
calibrate_station ¶
Calibrate the sensor
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
CalibrationConfiguration
|
Config file which contains all the required info for calibration, by default None |
None
|
Raises:
| Type | Description |
|---|---|
ValueError
|
When no calibration data provided |
Report
A site calibration was performed. The fitted calibration parameter was {n0} using {neutron_conversion_method_label}. For Desilets/Köhli 2021 this is N_0 in cph; for Köhli 2025 it is the detector-specific normalisation intensity I_D in cph, used directly in I_N = I/I_D*C_h. The sample- derived site means were dry bulk density = {avg_dry_soil_bulk_density} g/cm³, soil organic carbon = {avg_soil_organic_carbon} g/g, and lattice water = {avg_lattice_water} g/g.
write_calibration_sample_location_map ¶
Write an interactive HTML map for the last calibration samples.
The map is available when calibration profiles carry optional WGS84 latitude/longitude metadata. It is a presentation helper only and does not change calibration weights or fitted detector parameters.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output_path
|
str | Path
|
Destination HTML file, by default
|
'calibration_sample_locations.html'
|
Returns:
| Type | Description |
|---|---|
Path
|
Path to the written HTML page. |
Raises:
| Type | Description |
|---|---|
NeptoonCalibrationError
|
If no calibration has been prepared on this hub yet. |
align_time_stamps ¶
Align timestamps onto a regular grid at the target frequency, e.g. 15:04 -> 15:00, using the TimeStampAligner.
Numeric product columns are interpolated onto the grid; observed meteorological columns are snapped to the nearest real observation.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
align_method
|
str
|
Interpolation method passed to the aligner, by default "time". |
'time'
|
max_gap_hours
|
float | None
|
Longest observed-data gap (in hours) that may be interpolated across. Longer gaps are left as NaN. None means no limit, by default None. |
None
|
target_frequency
|
str | None
|
Explicit alignment grid frequency. When omitted, the cadence is inferred from the input data. |
None
|
aggregate_data_frame ¶
Aggregate a crns data frame to a new resolution.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output_resolution
|
str
|
Desired output resolution (e.g., '1h' or '1day') |
required |
max_na_fraction
|
float
|
fraction of acceptable nan values in aggregation period, by default 0.3 |
0.3
|
aggregate_method
|
str
|
description, by default "bagg" |
'bagg'
|
produce_soil_moisture_estimates ¶
produce_soil_moisture_estimates(n0=None, conversion_theory='desilets_etal_2010', dry_soil_bulk_density=None, lattice_water=None, soil_organic_carbon=None, soil_organic_carbon_conversion=None, n0_uncertainty=None, dry_soil_bulk_density_uncertainty=None, lattice_water_uncertainty=None, soil_organic_carbon_uncertainty=None, soil_moisture_systematic_uncertainty=None, correction_factor_relative_uncertainty=0.0, processing_missing_policy='keep_nan', koehli_parameters=None, koehli_2025_humidity_correction='uts_2025', koehli_2025_reference_abs_air_humidity=0.0, horizontal_footprint_model=None)
Produces SM estimates with the NeutronsToSM class. If values for n0, dry_soil_bulk_density, lattice_water, or soil_organic_carbon are not supplied, the values are taken from the internal sensor_info class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
n0
|
float
|
Method-specific calibration intensity, by default None. For
Desilets 2010 and Köhli 2021 this is |
None
|
dry_soil_bulk_density
|
float
|
given in g/cm³, by default None |
None
|
lattice_water
|
float
|
given as decimal percent e.g., 0.01, by default None |
None
|
soil_organic_carbon
|
float
|
Given as decimal percent, e.g., 0.001, by default None |
None
|
Report
Soil moisture was reconstructed with {conversion_method_label}. The reconstruction used calibration intensity = {default_params[n0]} cph, dry bulk density = {default_params[dry_soil_bulk_density]} g/cm³, lattice water = {default_params[lattice_water]} g/g, and soil organic carbon = {default_params[soil_organic_carbon]} g/g.
mask_flagged_data ¶
Return a DataFrame where flagged data has been replaced with NaN.
The method aligns the flag table to the provided data frame and only masks common columns. Missing flag columns are treated as UNFLAGGED and logged instead of causing a shape/index alignment crash.
prepare_static_values ¶
Attaches the static values from the SensorInfo Pydantic model as columns of values in the crns_data_frame.
This method: 1. Converts the Pydantic model to a dictionary 2. Checks if each key already exists in the DataFrame 3. Skips None values 4. Adds the remaining values as new columns
The method preserves existing column values if they are already present in the DataFrame to avoid accidental overwrites.
prepare_additional_columns ¶
Prepares and adds additional columns required for processing.
Such as: - absolute humidity
create_figures ¶
create_figures(create_all=True, ignore_sections=None, selected_figures=None, show_figures=False, additional_plots=None, soil_moisture_sensor_comparison=None, precipitation_resolution='auto')
Handles creating the figures using the FigureHandler.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
create_all
|
bool
|
Default to create all figures in the FigureHandler._register, by default True |
True
|
ignore_sections
|
list
|
Ignore a whole topic section of figure names, by default [] |
None
|
selected_figures
|
list
|
A list of the figures to be created if not using create_all. See FigureHandler._figure_registry for the names of possible figures, by default [] |
None
|
show_figures
|
bool
|
Turn to False to not show Figures in the kernel, by default True |
False
|
save_data ¶
save_data(folder_name=None, save_folder_location=None, use_custom_column_names=False, custom_column_names_dict=None, append_timestamp=True)
Save processed data, flags, figures, and configured sidecar files.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
folder_name
|
str
|
Name of the output folder. Defaults to the sensor name. |
None
|
save_folder_location
|
str or Path
|
Parent output directory. Defaults to the configured save location. |
None
|
use_custom_column_names
|
bool
|
Apply configured custom output labels. |
False
|
custom_column_names_dict
|
dict
|
Mapping of canonical output columns to custom labels. |
None
|
append_timestamp
|
bool
|
Append a timestamp to the output folder name. |
True
|
neptoon.workflow.multi_sensor¶
multi_sensor ¶
Multi-sensor processing facade and execution orchestration.
Configuration, harmonisation, and reporting implementations live in focused modules while this module preserves the historical public import surface.
RECOVERABLE_SENSOR_RUN_ERRORS
module-attribute
¶
RECOVERABLE_SENSOR_RUN_ERRORS = (MultiSensorConfigError, NeptoonUserFacingError, OSError, RuntimeError, ValueError, YAMLError)
ATMOSPHERIC_PLOT_KEYS
module-attribute
¶
PLOT_COLUMN_CANDIDATES
module-attribute
¶
PLOT_COLUMN_CANDIDATES = {'raw_neutrons': (str(EPI_NEUTRON_COUNT_CPH), str(EPI_NEUTRON_COUNT_FINAL), str(EPI_NEUTRON_COUNT_RAW), str(THERM_NEUTRON_COUNT_CPH), str(THERM_NEUTRON_COUNT_RAW)), 'neutron_cph': (str(EPI_NEUTRON_COUNT_CPH), str(EPI_NEUTRON_COUNT_FINAL), str(THERM_NEUTRON_COUNT_CPH)), 'corrected_neutrons': (str(CORRECTED_EPI_NEUTRON_COUNT_FINAL), str(CORRECTED_EPI_NEUTRON_COUNT), str(CORRECTED_THERM_NEUTRON_COUNT_FINAL), str(CORRECTED_THERM_NEUTRON_COUNT)), 'muons': ('muons', 'muon', 'muons_cph', 'muon_cph', 'muons_raw', 'muon_raw'), 'soil_moisture': (str(SOIL_MOISTURE_VOL_FINAL),), 'air_pressure': (str(AIR_PRESSURE),), 'air_temperature': (str(AIR_TEMPERATURE),), 'relative_humidity': (str(AIR_RELATIVE_HUMIDITY),), 'incoming_neutron_intensity': (str(INCOMING_NEUTRON_INTENSITY),), 'precipitation': (str(PRECIPITATION), str(PRECIPITATION_DAILY)), 'solar_radiation': ('solar_radiation_w_m2_for_comparison', str(SOLAR_RADIATION))}
PLOT_KEY_ALIASES
module-attribute
¶
PLOT_KEY_ALIASES = {'epithermal_neutron_cph': 'neutron_cph', 'epithermal_neutrons_cph': 'neutron_cph', 'epithermal_neutron_counts': 'neutron_cph', 'epithermal_counts': 'neutron_cph', 'neutron_counts_cph': 'neutron_cph', 'incoming_neutrons': 'incoming_neutron_intensity', 'incoming_neutron_counts': 'incoming_neutron_intensity', 'incoming_neutron': 'incoming_neutron_intensity', 'nmdb': 'incoming_neutron_intensity'}
PLOT_TITLES
module-attribute
¶
PLOT_TITLES = {'raw_neutrons': 'Raw neutron counts', 'neutron_cph': 'Epithermal neutron counts', 'corrected_neutrons': 'Corrected neutron counts', 'muons': 'Muon counts', 'soil_moisture': 'Volumetric soil moisture', 'air_pressure': 'Air pressure', 'air_temperature': 'Air temperature', 'relative_humidity': 'Relative humidity', 'incoming_neutron_intensity': 'Incoming neutron intensity / NMDB', 'precipitation': 'Precipitation', 'solar_radiation': 'Global solar radiation'}
PLOT_YLABELS
module-attribute
¶
PLOT_YLABELS = {'raw_neutrons': 'cph', 'neutron_cph': 'cph', 'corrected_neutrons': 'corrected cph', 'muons': 'cph', 'soil_moisture': 'm3 m-3', 'air_pressure': 'hPa', 'air_temperature': 'deg C', 'relative_humidity': '%', 'incoming_neutron_intensity': 'counts', 'precipitation': 'mm', 'solar_radiation': 'W m-2'}
__all__
module-attribute
¶
__all__ = ['MultiSensorBatchResult', 'MultiSensorConfig', 'MultiSensorConfigError', 'MultiSensorEntry', 'MultiSensorHarmonizationConfig', 'MultiSensorMeteoHarmonizationConfig', 'MultiSensorNeutronHarmonizationConfig', 'MultiSensorProcessor', 'MultiSensorReferenceDataHarmonizationConfig', 'MultiSensorReportConfig', 'MultiSensorReportingHarmonizationConfig', 'MultiSensorRunPlan', 'MultiSensorRunResult', 'MultiSensorRunner', 'MultiSensorSharedConfig', 'MultiSensorTimeHarmonizationConfig', 'MultiSensorTimePlan', 'load_multi_sensor_config', 'run_multi_sensor_process', 'ATMOSPHERIC_PLOT_KEYS', 'PLOT_COLUMN_CANDIDATES', 'PLOT_KEY_ALIASES', 'PLOT_TITLES', 'PLOT_YLABELS', 'RECOVERABLE_SENSOR_RUN_ERRORS', '_import_report_plotting_stack', '_read_yaml', '_resolve_path']
MultiSensorBatchResult
dataclass
¶
MultiSensorBatchResult(results, output_dir, combined_data_path=None, combined_report_path=None, metadata_path=None)
Combined result object returned by :class:MultiSensorRunner.
MultiSensorConfig ¶
Bases: BaseModel
Top-level multi-sensor processing configuration.
process_config
class-attribute
instance-attribute
¶
process_config = Field(default=None, description='Optional shorthand for shared.process_config. Useful when all sensors should use one existing process YAML.')
harmonization
class-attribute
instance-attribute
¶
harmonization = Field(default_factory=MultiSensorHarmonizationConfig, description='Operational alignment for combined CSV/report outputs. Individual sensor runs keep their own single-sensor processing settings.')
MultiSensorConfigError
dataclass
¶
Bases: NeptoonConfigurationError, ValueError
Raised when a multi-sensor configuration cannot be interpreted.
MultiSensorEntry ¶
Bases: BaseModel
One sensor entry in a multi-sensor workflow.
MultiSensorHarmonizationConfig ¶
Bases: BaseModel
Operational harmonisation for combined multi-sensor outputs.
Harmonisation affects the combined CSV/report only. Individual sensor runs remain controlled by their original single-sensor YAML files.
mode
class-attribute
instance-attribute
¶
mode = Field(default='auto', description="Combined-output harmonisation mode. 'auto' applies safe defaults: UTC, union time window, coarsest sensor resolution, and raw-neutron cph comparison.")
time
class-attribute
instance-attribute
¶
time = Field(default_factory=MultiSensorTimeHarmonizationConfig)
neutron_counts
class-attribute
instance-attribute
¶
neutron_counts = Field(default_factory=MultiSensorNeutronHarmonizationConfig)
reference_data
class-attribute
instance-attribute
¶
reference_data = Field(default_factory=MultiSensorReferenceDataHarmonizationConfig)
meteo
class-attribute
instance-attribute
¶
meteo = Field(default_factory=MultiSensorMeteoHarmonizationConfig)
reporting
class-attribute
instance-attribute
¶
reporting = Field(default_factory=MultiSensorReportingHarmonizationConfig)
MultiSensorMeteoHarmonizationConfig ¶
Bases: BaseModel
Meteorological display-unit harmonisation for combined outputs.
These settings affect only the combined CSV/report. They are intentionally heuristic and conservative so that existing single-sensor outputs stay untouched while common unit mismatches are easier to compare.
normalize_pressure_to_hpa
class-attribute
instance-attribute
¶
normalize_pressure_to_hpa = Field(default=True, description='Convert air_pressure values that look like Pa to hPa in combined outputs.')
normalize_temperature_to_celsius
class-attribute
instance-attribute
¶
normalize_temperature_to_celsius = Field(default=True, description='Convert air_temperature values that look like Kelvin to degrees Celsius in combined outputs.')
normalize_relative_humidity_to_percent
class-attribute
instance-attribute
¶
normalize_relative_humidity_to_percent = Field(default=True, description='Convert air_relative_humidity fractions in [0, 1] to percent in combined outputs.')
solar_radiation_comparison_unit
class-attribute
instance-attribute
¶
solar_radiation_comparison_unit = Field(default='w_m2', description="Unit used for the combined solar-radiation comparison plot. 'w_m2' keeps flux values as W/m2 and converts daily energy-like MJ/m2 values to mean flux when this can be inferred from the time step.")
convert_daily_solar_energy_to_mean_flux
class-attribute
instance-attribute
¶
convert_daily_solar_energy_to_mean_flux = Field(default=True, description='When solar_radiation is daily and looks like MJ/m2 per day, add a derived W/m2 comparison column for combined outputs.')
solar_radiation_comparison_column_name
class-attribute
instance-attribute
¶
solar_radiation_comparison_column_name = Field(default='solar_radiation_w_m2_for_comparison', description='Derived column name used for harmonized solar-radiation comparison output.')
MultiSensorNeutronHarmonizationConfig ¶
Bases: BaseModel
Neutron-count comparison settings for combined outputs.
comparison_unit
class-attribute
instance-attribute
¶
comparison_unit = Field(default='cph', description='Unit for shared raw-neutron comparison plots. The default converts raw counts/cps to counts per hour for display when the time basis is known.')
allow_unit_conversion
class-attribute
instance-attribute
¶
allow_unit_conversion = Field(default=True, description='Allow safe raw-neutron unit conversion for combined comparison outputs.')
comparison_column_name
class-attribute
instance-attribute
¶
comparison_column_name = Field(default='raw_neutrons_cph_for_comparison', description='Derived column name added to combined CSVs when raw counts are converted to cph.')
MultiSensorReferenceDataHarmonizationConfig ¶
Bases: BaseModel
Behaviour for reference data used by different sensors.
keep_distinct_sources
class-attribute
instance-attribute
¶
keep_distinct_sources = Field(default=True, description='Keep different NMDB/DWD/external meteo sources as separate sensor-prefixed series instead of forcing one common reference source.')
deduplicate_identical_sources
class-attribute
instance-attribute
¶
deduplicate_identical_sources = Field(default=True, description='Reserved for report labelling: identical reference sources may be described once, while different sources remain separate.')
MultiSensorReportConfig ¶
Bases: BaseModel
Combined report and combined data export settings.
create_combined_report
class-attribute
instance-attribute
¶
save_combined_csv
class-attribute
instance-attribute
¶
save_combined_csv = Field(default=True, description='Write the combined multi-sensor table as CSV when save_combined_data is true.')
save_combined_json
class-attribute
instance-attribute
¶
save_combined_json = Field(default=False, description='Additionally write the combined multi-sensor table as JSON.')
json_orient
class-attribute
instance-attribute
¶
json_orient = Field(default='records', description='Pandas JSON orient for the combined multi-sensor table.')
json_date_format
class-attribute
instance-attribute
¶
output_name
class-attribute
instance-attribute
¶
output_name = Field(default='combined', description='Base name used for combined CSV/PDF/metadata files.')
plots
class-attribute
instance-attribute
¶
plots = Field(default_factory=lambda: {'raw_neutrons': True, 'neutron_cph': True, 'corrected_neutrons': True, 'muons': True, 'soil_moisture': True, 'air_pressure': True, 'air_temperature': True, 'relative_humidity': True, 'incoming_neutron_intensity': True, 'precipitation': True, 'solar_radiation': True}, description='Enable/disable combined comparison plots by logical name.')
max_points_per_series
class-attribute
instance-attribute
¶
max_points_per_series = Field(default=5000, ge=100, description='Maximum points drawn per sensor series in the combined report.')
MultiSensorReportingHarmonizationConfig ¶
Bases: BaseModel
User-facing combined-report behaviour.
missing_series_policy
class-attribute
instance-attribute
¶
missing_series_policy = Field(default='note', description="How combined plots report missing series. 'note' lists skipped sensors on the plot page; 'strict' fails report creation.")
include_harmonization_summary
class-attribute
instance-attribute
¶
include_harmonization_summary = Field(default=True, description='Include the derived time-resolution/window summary on the report title page.')
MultiSensorRunPlan
dataclass
¶
MultiSensorRunResult
dataclass
¶
MultiSensorRunResult(name, sensor_config_path, process_config_path, completed_successfully, exit_reason, data_frame, output_folder, processor, error_type=None, error_message=None)
Result of one single-sensor run inside a multi-sensor workflow.
MultiSensorSharedConfig ¶
Bases: BaseModel
Settings shared by all sensor runs in a multi-sensor workflow.
process_config
class-attribute
instance-attribute
¶
process_config = Field(default=None, description='Default processing YAML used by sensors without an override.')
save_location
class-attribute
instance-attribute
¶
save_location = Field(default=None, description='Root output folder for individual sensor folders and combined outputs.')
isolate_individual_outputs
class-attribute
instance-attribute
¶
isolate_individual_outputs = Field(default=True, description='Write each individual sensor run below <save_location>/individual/<sensor-name> to avoid folder collisions. Set false to keep legacy behaviour and let every run write directly below save_location.')
cache_dir
class-attribute
instance-attribute
¶
cache_dir = Field(default=None, description='Shared cache directory for DWD and external_meteo providers. NMDB already uses a shared cache by default.')
create_individual_reports
class-attribute
instance-attribute
¶
create_individual_reports = Field(default=None, description='Override data_storage.create_report for each sensor. Leave null to respect the sensor YAML.')
append_timestamp_to_folder_name
class-attribute
instance-attribute
¶
append_timestamp_to_folder_name = Field(default=None, description='Override per-sensor timestamped output folders. Leave null to respect the sensor YAML.')
continue_on_error
class-attribute
instance-attribute
¶
continue_on_error = Field(default=True, description='Continue processing remaining sensors if one sensor run raises an exception or finishes unsuccessfully. Set false for fail-fast/CI runs.')
retain_processors
class-attribute
instance-attribute
¶
retain_processors = Field(default=False, description='Keep each completed ProcessWithConfig object in the batch result. The default false is more memory efficient because processors retain their full data hub; set true for debugging or advanced API use.')
parallel
class-attribute
instance-attribute
¶
parallel = Field(default=False, description='Run enabled sensors concurrently with a process pool. Parallel mode uses the standard ConfigurationManager and ProcessWithConfig classes, keeps results in configuration order, and is best suited for independent sensor batches.')
max_workers
class-attribute
instance-attribute
¶
max_workers = Field(default=None, ge=1, description='Maximum number of worker processes used when shared.parallel is true. The default uses one fewer process than available CPU cores, capped by the number of enabled sensors.')
parallel_backend
class-attribute
instance-attribute
¶
parallel_backend = Field(default='process', description="Execution backend for shared.parallel. Currently only 'process' is supported.")
enable_output
class-attribute
instance-attribute
¶
enable_output = Field(default=None, description='Override verbose console output for individual sensor workflows.')
MultiSensorTimeHarmonizationConfig ¶
Bases: BaseModel
Time-axis harmonisation for combined multi-sensor outputs.
timezone
class-attribute
instance-attribute
¶
timezone = Field(default='UTC', description='Timezone used for combined comparison outputs. Naive timestamps are interpreted in this timezone; timezone-aware timestamps are converted.')
report_window
class-attribute
instance-attribute
¶
report_window = Field(default='union', description="Time span for combined comparison outputs. The default 'union' keeps the full time span covered by any sensor and leaves missing ranges as gaps. Use 'explicit' with manual_start/manual_end to force a window.")
report_resolution
class-attribute
instance-attribute
¶
report_resolution = Field(default='coarsest', description="Combined-output resolution. 'coarsest' and 'auto' use the coarsest native resolution found among the sensor result time series. A pandas duration such as '30min', '1h' or '1d' overrides that behaviour.")
manual_resolution
class-attribute
instance-attribute
¶
manual_resolution = Field(default=None, description="Optional explicit combined-output resolution, e.g. '30min', '1h' or '1d'.")
manual_start
class-attribute
instance-attribute
¶
manual_start = Field(default=None, description='Optional explicit start timestamp for report_window: explicit.')
manual_end
class-attribute
instance-attribute
¶
manual_end = Field(default=None, description='Optional explicit end timestamp for report_window: explicit.')
align_to_clock
class-attribute
instance-attribute
¶
align_to_clock = Field(default=True, description='Align resampling bins to a common clock in the configured timezone. Leave true for comparable multi-sensor plots.')
MultiSensorTimePlan
dataclass
¶
Derived time-axis plan for combined outputs.
MultiSensorRunner ¶
MultiSensorRunner(config, process_class=None, configuration_manager_class=ConfigurationManager, parallel_executor_class=None, status_callback=None)
Bases: MultiSensorHarmonizationMixin, MultiSensorReportingMixin
Run configured station workflows and assemble batch-level outputs.
Each enabled sensor entry is executed with the normal
:class:~neptoon.workflow.process_with_config.ProcessWithConfig pipeline,
sequentially or through a process pool. Per-station results are retained in
input order. When continue_on_error is enabled, supported station failures
become failed result records instead of aborting the batch. Successful frames
may then be time/unit harmonized into combined data and report products.
process_class, configuration_manager_class and
parallel_executor_class are injection points for tests and embedding
applications; ordinary callers should leave them at their defaults.
configuration_manager_class
instance-attribute
¶
from_config_file
classmethod
¶
Load a multi-sensor YAML file and construct a runner.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config_path
|
str | Path
|
Path to the multi-sensor configuration file. |
required |
**kwargs
|
Optional dependency-injection arguments accepted by the constructor. |
{}
|
run ¶
Run enabled sensors and write configured combined artifacts.
The status callback reports batch-local progress by completed sensor.
Sequential sensor runs also scope their normal Neptoon status events
beneath the corresponding multi_sensor.sensor operation.
Returns:
| Type | Description |
|---|---|
MultiSensorBatchResult
|
Ordered per-sensor results plus paths to any combined data, report, and batch metadata files that were created. |
Raises:
| Type | Description |
|---|---|
MultiSensorConfigError
|
If a non-recoverable configuration problem occurs or a station fails
while |
load_multi_sensor_config ¶
Load a config: multi_sensor YAML file and resolve relative paths.
run_multi_sensor_process ¶
Process a multi-sensor YAML file with optional user-status events.
neptoon.workflow.multi_sensor_config¶
multi_sensor_config ¶
Configuration models and result records for multi-sensor workflows.
RECOVERABLE_SENSOR_RUN_ERRORS
module-attribute
¶
RECOVERABLE_SENSOR_RUN_ERRORS = (MultiSensorConfigError, NeptoonUserFacingError, OSError, RuntimeError, ValueError, YAMLError)
__all__
module-attribute
¶
__all__ = ['MultiSensorConfigError', 'MultiSensorSharedConfig', 'MultiSensorEntry', 'MultiSensorReportConfig', 'MultiSensorTimeHarmonizationConfig', 'MultiSensorNeutronHarmonizationConfig', 'MultiSensorMeteoHarmonizationConfig', 'MultiSensorReferenceDataHarmonizationConfig', 'MultiSensorReportingHarmonizationConfig', 'MultiSensorHarmonizationConfig', 'MultiSensorTimePlan', 'MultiSensorConfig', 'MultiSensorRunResult', 'MultiSensorRunPlan', 'MultiSensorBatchResult', 'RECOVERABLE_SENSOR_RUN_ERRORS', 'load_multi_sensor_config']
MultiSensorConfigError
dataclass
¶
Bases: NeptoonConfigurationError, ValueError
Raised when a multi-sensor configuration cannot be interpreted.
MultiSensorSharedConfig ¶
Bases: BaseModel
Settings shared by all sensor runs in a multi-sensor workflow.
process_config
class-attribute
instance-attribute
¶
process_config = Field(default=None, description='Default processing YAML used by sensors without an override.')
save_location
class-attribute
instance-attribute
¶
save_location = Field(default=None, description='Root output folder for individual sensor folders and combined outputs.')
isolate_individual_outputs
class-attribute
instance-attribute
¶
isolate_individual_outputs = Field(default=True, description='Write each individual sensor run below <save_location>/individual/<sensor-name> to avoid folder collisions. Set false to keep legacy behaviour and let every run write directly below save_location.')
cache_dir
class-attribute
instance-attribute
¶
cache_dir = Field(default=None, description='Shared cache directory for DWD and external_meteo providers. NMDB already uses a shared cache by default.')
create_individual_reports
class-attribute
instance-attribute
¶
create_individual_reports = Field(default=None, description='Override data_storage.create_report for each sensor. Leave null to respect the sensor YAML.')
append_timestamp_to_folder_name
class-attribute
instance-attribute
¶
append_timestamp_to_folder_name = Field(default=None, description='Override per-sensor timestamped output folders. Leave null to respect the sensor YAML.')
continue_on_error
class-attribute
instance-attribute
¶
continue_on_error = Field(default=True, description='Continue processing remaining sensors if one sensor run raises an exception or finishes unsuccessfully. Set false for fail-fast/CI runs.')
retain_processors
class-attribute
instance-attribute
¶
retain_processors = Field(default=False, description='Keep each completed ProcessWithConfig object in the batch result. The default false is more memory efficient because processors retain their full data hub; set true for debugging or advanced API use.')
parallel
class-attribute
instance-attribute
¶
parallel = Field(default=False, description='Run enabled sensors concurrently with a process pool. Parallel mode uses the standard ConfigurationManager and ProcessWithConfig classes, keeps results in configuration order, and is best suited for independent sensor batches.')
max_workers
class-attribute
instance-attribute
¶
max_workers = Field(default=None, ge=1, description='Maximum number of worker processes used when shared.parallel is true. The default uses one fewer process than available CPU cores, capped by the number of enabled sensors.')
parallel_backend
class-attribute
instance-attribute
¶
parallel_backend = Field(default='process', description="Execution backend for shared.parallel. Currently only 'process' is supported.")
enable_output
class-attribute
instance-attribute
¶
enable_output = Field(default=None, description='Override verbose console output for individual sensor workflows.')
MultiSensorEntry ¶
Bases: BaseModel
One sensor entry in a multi-sensor workflow.
MultiSensorReportConfig ¶
Bases: BaseModel
Combined report and combined data export settings.
create_combined_report
class-attribute
instance-attribute
¶
save_combined_csv
class-attribute
instance-attribute
¶
save_combined_csv = Field(default=True, description='Write the combined multi-sensor table as CSV when save_combined_data is true.')
save_combined_json
class-attribute
instance-attribute
¶
save_combined_json = Field(default=False, description='Additionally write the combined multi-sensor table as JSON.')
json_orient
class-attribute
instance-attribute
¶
json_orient = Field(default='records', description='Pandas JSON orient for the combined multi-sensor table.')
json_date_format
class-attribute
instance-attribute
¶
output_name
class-attribute
instance-attribute
¶
output_name = Field(default='combined', description='Base name used for combined CSV/PDF/metadata files.')
plots
class-attribute
instance-attribute
¶
plots = Field(default_factory=lambda: {'raw_neutrons': True, 'neutron_cph': True, 'corrected_neutrons': True, 'muons': True, 'soil_moisture': True, 'air_pressure': True, 'air_temperature': True, 'relative_humidity': True, 'incoming_neutron_intensity': True, 'precipitation': True, 'solar_radiation': True}, description='Enable/disable combined comparison plots by logical name.')
max_points_per_series
class-attribute
instance-attribute
¶
max_points_per_series = Field(default=5000, ge=100, description='Maximum points drawn per sensor series in the combined report.')
MultiSensorTimeHarmonizationConfig ¶
Bases: BaseModel
Time-axis harmonisation for combined multi-sensor outputs.
timezone
class-attribute
instance-attribute
¶
timezone = Field(default='UTC', description='Timezone used for combined comparison outputs. Naive timestamps are interpreted in this timezone; timezone-aware timestamps are converted.')
report_window
class-attribute
instance-attribute
¶
report_window = Field(default='union', description="Time span for combined comparison outputs. The default 'union' keeps the full time span covered by any sensor and leaves missing ranges as gaps. Use 'explicit' with manual_start/manual_end to force a window.")
report_resolution
class-attribute
instance-attribute
¶
report_resolution = Field(default='coarsest', description="Combined-output resolution. 'coarsest' and 'auto' use the coarsest native resolution found among the sensor result time series. A pandas duration such as '30min', '1h' or '1d' overrides that behaviour.")
manual_resolution
class-attribute
instance-attribute
¶
manual_resolution = Field(default=None, description="Optional explicit combined-output resolution, e.g. '30min', '1h' or '1d'.")
manual_start
class-attribute
instance-attribute
¶
manual_start = Field(default=None, description='Optional explicit start timestamp for report_window: explicit.')
manual_end
class-attribute
instance-attribute
¶
manual_end = Field(default=None, description='Optional explicit end timestamp for report_window: explicit.')
align_to_clock
class-attribute
instance-attribute
¶
align_to_clock = Field(default=True, description='Align resampling bins to a common clock in the configured timezone. Leave true for comparable multi-sensor plots.')
MultiSensorNeutronHarmonizationConfig ¶
Bases: BaseModel
Neutron-count comparison settings for combined outputs.
comparison_unit
class-attribute
instance-attribute
¶
comparison_unit = Field(default='cph', description='Unit for shared raw-neutron comparison plots. The default converts raw counts/cps to counts per hour for display when the time basis is known.')
allow_unit_conversion
class-attribute
instance-attribute
¶
allow_unit_conversion = Field(default=True, description='Allow safe raw-neutron unit conversion for combined comparison outputs.')
comparison_column_name
class-attribute
instance-attribute
¶
comparison_column_name = Field(default='raw_neutrons_cph_for_comparison', description='Derived column name added to combined CSVs when raw counts are converted to cph.')
MultiSensorMeteoHarmonizationConfig ¶
Bases: BaseModel
Meteorological display-unit harmonisation for combined outputs.
These settings affect only the combined CSV/report. They are intentionally heuristic and conservative so that existing single-sensor outputs stay untouched while common unit mismatches are easier to compare.
normalize_pressure_to_hpa
class-attribute
instance-attribute
¶
normalize_pressure_to_hpa = Field(default=True, description='Convert air_pressure values that look like Pa to hPa in combined outputs.')
normalize_temperature_to_celsius
class-attribute
instance-attribute
¶
normalize_temperature_to_celsius = Field(default=True, description='Convert air_temperature values that look like Kelvin to degrees Celsius in combined outputs.')
normalize_relative_humidity_to_percent
class-attribute
instance-attribute
¶
normalize_relative_humidity_to_percent = Field(default=True, description='Convert air_relative_humidity fractions in [0, 1] to percent in combined outputs.')
solar_radiation_comparison_unit
class-attribute
instance-attribute
¶
solar_radiation_comparison_unit = Field(default='w_m2', description="Unit used for the combined solar-radiation comparison plot. 'w_m2' keeps flux values as W/m2 and converts daily energy-like MJ/m2 values to mean flux when this can be inferred from the time step.")
convert_daily_solar_energy_to_mean_flux
class-attribute
instance-attribute
¶
convert_daily_solar_energy_to_mean_flux = Field(default=True, description='When solar_radiation is daily and looks like MJ/m2 per day, add a derived W/m2 comparison column for combined outputs.')
solar_radiation_comparison_column_name
class-attribute
instance-attribute
¶
solar_radiation_comparison_column_name = Field(default='solar_radiation_w_m2_for_comparison', description='Derived column name used for harmonized solar-radiation comparison output.')
MultiSensorReferenceDataHarmonizationConfig ¶
Bases: BaseModel
Behaviour for reference data used by different sensors.
keep_distinct_sources
class-attribute
instance-attribute
¶
keep_distinct_sources = Field(default=True, description='Keep different NMDB/DWD/external meteo sources as separate sensor-prefixed series instead of forcing one common reference source.')
deduplicate_identical_sources
class-attribute
instance-attribute
¶
deduplicate_identical_sources = Field(default=True, description='Reserved for report labelling: identical reference sources may be described once, while different sources remain separate.')
MultiSensorReportingHarmonizationConfig ¶
Bases: BaseModel
User-facing combined-report behaviour.
missing_series_policy
class-attribute
instance-attribute
¶
missing_series_policy = Field(default='note', description="How combined plots report missing series. 'note' lists skipped sensors on the plot page; 'strict' fails report creation.")
include_harmonization_summary
class-attribute
instance-attribute
¶
include_harmonization_summary = Field(default=True, description='Include the derived time-resolution/window summary on the report title page.')
MultiSensorHarmonizationConfig ¶
Bases: BaseModel
Operational harmonisation for combined multi-sensor outputs.
Harmonisation affects the combined CSV/report only. Individual sensor runs remain controlled by their original single-sensor YAML files.
mode
class-attribute
instance-attribute
¶
mode = Field(default='auto', description="Combined-output harmonisation mode. 'auto' applies safe defaults: UTC, union time window, coarsest sensor resolution, and raw-neutron cph comparison.")
time
class-attribute
instance-attribute
¶
time = Field(default_factory=MultiSensorTimeHarmonizationConfig)
neutron_counts
class-attribute
instance-attribute
¶
neutron_counts = Field(default_factory=MultiSensorNeutronHarmonizationConfig)
reference_data
class-attribute
instance-attribute
¶
reference_data = Field(default_factory=MultiSensorReferenceDataHarmonizationConfig)
meteo
class-attribute
instance-attribute
¶
meteo = Field(default_factory=MultiSensorMeteoHarmonizationConfig)
reporting
class-attribute
instance-attribute
¶
reporting = Field(default_factory=MultiSensorReportingHarmonizationConfig)
MultiSensorTimePlan
dataclass
¶
Derived time-axis plan for combined outputs.
MultiSensorConfig ¶
Bases: BaseModel
Top-level multi-sensor processing configuration.
process_config
class-attribute
instance-attribute
¶
process_config = Field(default=None, description='Optional shorthand for shared.process_config. Useful when all sensors should use one existing process YAML.')
harmonization
class-attribute
instance-attribute
¶
harmonization = Field(default_factory=MultiSensorHarmonizationConfig, description='Operational alignment for combined CSV/report outputs. Individual sensor runs keep their own single-sensor processing settings.')
MultiSensorRunResult
dataclass
¶
MultiSensorRunResult(name, sensor_config_path, process_config_path, completed_successfully, exit_reason, data_frame, output_folder, processor, error_type=None, error_message=None)
Result of one single-sensor run inside a multi-sensor workflow.
MultiSensorRunPlan
dataclass
¶
MultiSensorBatchResult
dataclass
¶
MultiSensorBatchResult(results, output_dir, combined_data_path=None, combined_report_path=None, metadata_path=None)
Combined result object returned by :class:MultiSensorRunner.
load_multi_sensor_config ¶
Load a config: multi_sensor YAML file and resolve relative paths.
neptoon.workflow.multi_sensor_harmonization¶
multi_sensor_harmonization ¶
Combined-table construction and time/unit harmonisation for multi-sensor runs.
MultiSensorHarmonizationMixin ¶
build_combined_data_frame ¶
Return one dataframe with sensor-prefixed, report-harmonised columns.
The combined data export is a comparison product, not a replacement for the individual sensor outputs. By default it keeps the union of all sensor time ranges, converts timestamps to one clock (UTC unless configured otherwise), and resamples every sensor to the coarsest native resolution found in the successful runs. Missing ranges remain NaN so users can see where sensors did not overlap.
neptoon.workflow.multi_sensor_reporting¶
multi_sensor_reporting ¶
Combined PDF, metadata, and plot selection for multi-sensor workflows.
PLOT_COLUMN_CANDIDATES
module-attribute
¶
PLOT_COLUMN_CANDIDATES = {'raw_neutrons': (str(EPI_NEUTRON_COUNT_CPH), str(EPI_NEUTRON_COUNT_FINAL), str(EPI_NEUTRON_COUNT_RAW), str(THERM_NEUTRON_COUNT_CPH), str(THERM_NEUTRON_COUNT_RAW)), 'neutron_cph': (str(EPI_NEUTRON_COUNT_CPH), str(EPI_NEUTRON_COUNT_FINAL), str(THERM_NEUTRON_COUNT_CPH)), 'corrected_neutrons': (str(CORRECTED_EPI_NEUTRON_COUNT_FINAL), str(CORRECTED_EPI_NEUTRON_COUNT), str(CORRECTED_THERM_NEUTRON_COUNT_FINAL), str(CORRECTED_THERM_NEUTRON_COUNT)), 'muons': ('muons', 'muon', 'muons_cph', 'muon_cph', 'muons_raw', 'muon_raw'), 'soil_moisture': (str(SOIL_MOISTURE_VOL_FINAL),), 'air_pressure': (str(AIR_PRESSURE),), 'air_temperature': (str(AIR_TEMPERATURE),), 'relative_humidity': (str(AIR_RELATIVE_HUMIDITY),), 'incoming_neutron_intensity': (str(INCOMING_NEUTRON_INTENSITY),), 'precipitation': (str(PRECIPITATION), str(PRECIPITATION_DAILY)), 'solar_radiation': ('solar_radiation_w_m2_for_comparison', str(SOLAR_RADIATION))}
PLOT_TITLES
module-attribute
¶
PLOT_TITLES = {'raw_neutrons': 'Raw neutron counts', 'neutron_cph': 'Epithermal neutron counts', 'corrected_neutrons': 'Corrected neutron counts', 'muons': 'Muon counts', 'soil_moisture': 'Volumetric soil moisture', 'air_pressure': 'Air pressure', 'air_temperature': 'Air temperature', 'relative_humidity': 'Relative humidity', 'incoming_neutron_intensity': 'Incoming neutron intensity / NMDB', 'precipitation': 'Precipitation', 'solar_radiation': 'Global solar radiation'}
PLOT_YLABELS
module-attribute
¶
PLOT_YLABELS = {'raw_neutrons': 'cph', 'neutron_cph': 'cph', 'corrected_neutrons': 'corrected cph', 'muons': 'cph', 'soil_moisture': 'm3 m-3', 'air_pressure': 'hPa', 'air_temperature': 'deg C', 'relative_humidity': '%', 'incoming_neutron_intensity': 'counts', 'precipitation': 'mm', 'solar_radiation': 'W m-2'}
PLOT_KEY_ALIASES
module-attribute
¶
PLOT_KEY_ALIASES = {'epithermal_neutron_cph': 'neutron_cph', 'epithermal_neutrons_cph': 'neutron_cph', 'epithermal_neutron_counts': 'neutron_cph', 'epithermal_counts': 'neutron_cph', 'neutron_counts_cph': 'neutron_cph', 'incoming_neutrons': 'incoming_neutron_intensity', 'incoming_neutron_counts': 'incoming_neutron_intensity', 'incoming_neutron': 'incoming_neutron_intensity', 'nmdb': 'incoming_neutron_intensity'}
ATMOSPHERIC_PLOT_KEYS
module-attribute
¶
MultiSensorReportingMixin ¶
neptoon.workflow.interactive¶
The application-layer session is the stable boundary for GUIs, notebooks, and
service adapters that need stepwise configured processing without calling private
ProcessWithConfig or CRNSDataHub methods.
interactive ¶
Application-layer API for GUIs, notebooks, and other interactive clients.
The existing Streamlit GUI is intentionally not coupled to this module yet. New
clients can use :class:ConfiguredProcessingSession immediately, while the legacy
GUI can migrate later without changing the scientific workflow implementation.
__all__
module-attribute
¶
__all__ = ['ConfiguredProcessingSession', 'InputSourceKind', 'get_capabilities', 'get_configuration_schema', 'get_method_requirements']
ConfiguredProcessingSession ¶
ConfiguredProcessingSession(processor, *, checkpoint_storage='off', checkpoint_directory=None, checkpoint_memory_limit_bytes=DEFAULT_CHECKPOINT_MEMORY_LIMIT_BYTES, status_callback=None)
Stable stateful application boundary over ProcessWithConfig.
The session is intended for GUIs, notebooks, REST adapters, and other
interactive clients. Configuration changes are always revalidated. Changes
after processing invalidate affected stages instead of mutating live
scientific state silently. Replay checkpoints are disabled by default. When
callers opt in, session-local checkpoints are retained at each scientific stage
boundary so replay can restart immediately before the affected workflow item.
auto keeps small/medium sessions in RAM and rebalances older checkpoints to
atomic Parquet once the estimated in-process scientific-data footprint would
exceed 3 GB; callers can also force memory or disk storage explicitly. Cache
failures are performance misses only and never invalidate successful science.
Correction-factor products are cached only while checkpointing is enabled and
reused only when their own configuration and upstream state remain valid.
When no suitable checkpoint exists Neptoon falls back to a conservative
replay from the pipeline beginning.
sensor_config
property
¶
Return an independent copy of the validated sensor configuration.
process_config
property
¶
Return an independent copy of the validated process configuration.
available_checkpoints
property
¶
Return retained replay checkpoints in scientific stage order.
checkpoint_directory
property
¶
Return the session-local disk cache directory, when enabled.
checkpoint_statistics
property
¶
Return lightweight RAM/disk telemetry for the replay cache.
from_files
classmethod
¶
from_files(*, sensor_config, process_config=None, checkpoint_storage='off', checkpoint_directory=None, checkpoint_memory_limit_bytes=DEFAULT_CHECKPOINT_MEMORY_LIMIT_BYTES, status_callback=None)
from_processor
classmethod
¶
from_processor(processor, *, checkpoint_storage='off', checkpoint_directory=None, checkpoint_memory_limit_bytes=DEFAULT_CHECKPOINT_MEMORY_LIMIT_BYTES, status_callback=None)
clear_diagnostics ¶
Clear accumulated structured warnings without changing processing state.
clear_checkpoints ¶
Discard retained stage checkpoints and compact correction caches.
preview_config_change ¶
Validate a proposed patch and report which stages it would invalidate.
preview_configuration_change ¶
Long-form alias for :meth:preview_config_change.
update_configuration ¶
Apply a validated patch and invalidate dependent processed results.
set_input_source ¶
Set a local raw/prepared input path through validated config overrides.
set_calibration_source ¶
Set calibration input without mutating nested Pydantic objects in place.
get_stage_readiness ¶
Return direct-run readiness and blockers for one or all stages.
reset_from ¶
Invalidate one stage and downstream results without discarding the snapshot.
run_stage ¶
Run exactly one stage after all prerequisites have completed.
run_until ¶
Run all required stages through stage, replaying safely if stale.
run_all ¶
Run the full configured workflow, replaying safely if state is stale.
reprocess_from ¶
reprocess_from(stage, *, until=None, progress_callback=None, status_callback=None, cancellation_token=None)
Invalidate from stage and replay safely through until.
The newest valid upstream checkpoint is restored when available. If a configuration update already invalidated an earlier stage, that earlier scientific boundary wins and its fine-grained correction reuse plan is preserved. Without a suitable checkpoint execution falls back to the pipeline beginning.
preview_input ¶
Return a bounded formatted input sample without changing session state.
Status callbacks use operation-local progress. Raw multi-file previews can
therefore publish fine-grained input.read_files updates from the
reader while this method brackets the preview as one short operation.
inspect_input ¶
Return formatted column/time metadata without changing session state.
write_calibration_sample_map ¶
Write the current calibration sample locations as standalone HTML.
This is the stable application boundary for GUIs and notebooks. It
deliberately delegates to the calibrated hub instead of exposing
CRNSDataHub or CalibrationStation internals to clients.
The method requires a prepared calibration with geographic sample
coordinates and the optional geo dependencies. Clients can use
session.capabilities.calibration_sample_map_available to decide
whether rendering support is installed and
session.result.calibration.sample_map_available to decide whether
the current calibration has located profiles.
get_configuration_schema ¶
Return supported validated configuration schemas for UI generation.
get_capabilities ¶
Return dynamic and model-declared capabilities for interactive clients.
get_method_requirements ¶
Return declared requirements for a selectable interactive feature/method.
neptoon.workflow.contracts¶
contracts ¶
Stable application-layer contracts for interactive Neptoon clients.
These types deliberately contain no Streamlit, CLI, or server-runtime code. They
form the public boundary that a GUI, notebook, REST adapter, or other interactive
client can consume without reaching into ProcessWithConfig or CRNSDataHub
internals.
__all__
module-attribute
¶
__all__ = ['CalibrationResult', 'CancellationToken', 'ConfigurationImpact', 'ConfigurationSchema', 'DiagnosticMessage', 'ErrorInfo', 'InputInspection', 'InputPreview', 'InteractiveSessionState', 'MessageSeverity', 'MethodRequirements', 'NeptoonCapabilities', 'OutputArtifact', 'OutputManifest', 'PIPELINE_STAGE_ORDER', 'PipelineStage', 'ProcessingProvenance', 'ProgressCallback', 'ProgressEvent', 'SerializableContract', 'SeriesDescriptor', 'StageReadiness', 'StageState', 'StageStatus', 'ValidationIssue', 'ValidationResult', 'VisualizationDescriptor', 'WorkflowResult', 'WorkflowSnapshot']
SerializableContract ¶
PipelineStage ¶
StageState ¶
Bases: StrEnum
Execution state reported for one pipeline stage.
MessageSeverity ¶
ErrorInfo
dataclass
¶
DiagnosticMessage
dataclass
¶
Bases: SerializableContract
Structured warning/info/error emitted by an application-layer operation.
ValidationIssue
dataclass
¶
Bases: SerializableContract
One actionable preflight validation issue.
ValidationResult
dataclass
¶
StageReadiness
dataclass
¶
Bases: SerializableContract
Whether one stage can run directly and why it may be blocked.
ConfigurationImpact
dataclass
¶
ConfigurationImpact(changed_paths, earliest_stage, affected_stages, requires_reprocessing, invalidates_existing_results)
Bases: SerializableContract
Stages invalidated by a proposed validated configuration change.
StageStatus
dataclass
¶
StageStatus(stage, state=NOT_STARTED, message=None, error=None)
ProgressEvent
dataclass
¶
Bases: SerializableContract
Best-effort progress notification emitted by the configured workflow.
CancellationToken ¶
SeriesDescriptor
dataclass
¶
VisualizationDescriptor
dataclass
¶
VisualizationDescriptor(id, label, series, preferred_kind='line', x='date_time', y_unit=None, category=None)
OutputArtifact
dataclass
¶
Bases: SerializableContract
One persisted output produced by the configured workflow.
OutputManifest
dataclass
¶
Bases: SerializableContract
Manifest of filesystem outputs known to the application layer.
CalibrationResult
dataclass
¶
CalibrationResult(parameter_name, parameter_value, parameter_unit, parameter_uncertainty, method, profile_id=None, fit_diagnostics=dict(), metadata=dict(), daily_results=None, sample_map_available=False, located_profile_count=0)
InputPreview
dataclass
¶
Bases: SerializableContract
Bounded input sample produced without mutating the active workflow state.
InputInspection
dataclass
¶
InputInspection(available_columns, dtypes, suggested_mapping, unresolved, row_count, time_range=None, inferred_resolution=None)
MethodRequirements
dataclass
¶
MethodRequirements(method, stage, required_series=(), optional_series=(), required_config=(), external_dependencies=())
ProcessingProvenance
dataclass
¶
ProcessingProvenance(session_id, sensor_id, configuration_revision, processing_revision, config_hash, input_fingerprint, started_at, finished_at, package_version, completed_stages=())
Bases: SerializableContract
Reproducibility metadata for the current interactive processing revision.
InteractiveSessionState
dataclass
¶
InteractiveSessionState(session_id, sensor_id, configuration_revision, processing_revision, result_revision, dirty, results_stale, completed, invalidated_from, stages)
CheckpointCacheStatistics
dataclass
¶
CheckpointCacheStatistics(checkpoint_count, memory_checkpoint_count, disk_checkpoint_count, memory_data_bytes, disk_bytes, memory_limit_bytes)
WorkflowSnapshot
dataclass
¶
WorkflowResult
dataclass
¶
WorkflowResult(completed_successfully, workflow_completed, requested_stage, current_stage, stage_statuses, error, snapshot, calibration, outputs, diagnostics=(), session_state=None, provenance=None, sensor_id=None)
Bases: SerializableContract
Stable result object for full or partial configured processing.
to_summary_dict ¶
Return a compact JSON-safe result without tabular payloads.
Service adapters often need workflow state, diagnostics, provenance, and
persisted-output metadata without embedding potentially large pandas
objects in an HTTP response. The summary therefore replaces DataFrame
values with row/column counts while keeping the normal to_dict()
representation unchanged for callers that explicitly need full data.
ConfigurationSchema
dataclass
¶
Bases: SerializableContract
JSON-schema descriptions of supported sensor and process configuration.
NeptoonCapabilities
dataclass
¶
NeptoonCapabilities(pipeline_stages, nmdb_monitors, external_meteo_providers, air_pressure_correction_methods, air_humidity_correction_methods, thermal_neutron_correction_methods, incoming_radiation_correction_methods, biomass_correction_methods, soil_moisture_methods, smoothing_algorithms, quality_assessment_methods=(), quality_assessment_targets=(), input_source_kinds=('auto', 'raw', 'prepared'), figure_identifiers=(), figure_generation_available=False, calibration_sample_map_available=False, tabular_output_formats=('csv', 'json'), method_requirement_ids=(), interactive_features=('validation', 'stage_readiness', 'config_impact', 'reprocessing', 'input_preview', 'column_inspection', 'structured_diagnostics', 'cancellation', 'provenance', 'calibration_sample_map'))
Bases: SerializableContract
Dynamic and enum-like capabilities useful for building interactive UIs.
input_source_kinds
class-attribute
instance-attribute
¶
figure_generation_available
class-attribute
instance-attribute
¶
calibration_sample_map_available
class-attribute
instance-attribute
¶
tabular_output_formats
class-attribute
instance-attribute
¶
interactive_features
class-attribute
instance-attribute
¶
interactive_features = ('validation', 'stage_readiness', 'config_impact', 'reprocessing', 'input_preview', 'column_inspection', 'structured_diagnostics', 'cancellation', 'provenance', 'calibration_sample_map')
neptoon.workflow.timeio¶
The application-layer adapter keeps unattended TimeIO synchronization independent of configured-workflow internals while preserving the stable processing contract.
timeio ¶
Workflow adapter used by unattended TimeIO synchronization.
The TimeIO I/O layer deliberately has no dependency on workflow stage enums or
ProcessWithConfig. This module is the application boundary that connects the
stable configured workflow to the small processor protocol consumed by TimeIO.
neptoon.workflow.process_with_config¶
process_with_config ¶
__all__
module-attribute
¶
__all__ = ['CalibrationStation', 'CorrectionSelectorFromConfig', 'ProcessWithConfig', 'QualityAssessmentFromConfig', 'build_calibration_metadata', 'resolve_calibration_window_alignment', 'resolve_calibration_window_hours']
CalibrationStation ¶
Abstract which does the complete claibration steps. Can be used on its own, but is mainly designed to facilitate CRNSDataHub calibration. Simply include the calibration data, the time series data and the config object and run find_n0_value(), to return the optimum N0.
find_n0_value ¶
Runs the full process to obtain an N0 estimate.
Returns:
| Type | Description |
|---|---|
float
|
N0 estimate after calibration. |
return_calibration_results_data_frame ¶
Returns the daily results as a data frame. When multiple days calibration is undertaken on each day. The outputs of this are saved and this method returns them for viewing.
Returns:
| Type | Description |
|---|---|
DataFrame
|
data frame with the results in it. |
return_weighting_dataframe ¶
Returns the information about the weighting procedure
Returns:
| Type | Description |
|---|---|
DataFrame
|
dataframe with weights |
write_sample_location_map ¶
write_sample_location_map(output_path='calibration_sample_locations.html', *, station_latitude=None, station_longitude=None)
Write an interactive HTML map of geographically located samples.
Latitude/longitude are optional calibration metadata and do not affect
the N0/I_D fit. If the calibration has not yet been run, sample
profiles are prepared from calibration_data solely for this map.
The feature requires the geo optional dependency extra.
CorrectionSelectorFromConfig ¶
Selects and configures neutron count corrections based on configuration.
This class translates correction configuration parameters into the appropriate correction objects to be applied to neutron count data. It supports various correction types including pressure, humidity, incoming intensity, and above-ground biomass.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data_hub
|
CRNSDataHub
|
The data hub instance to which corrections will be applied |
required |
process_config
|
BaseConfig
|
Process configuration containing correction specifications |
required |
sensor_config
|
BaseConfig
|
Sensor configuration containing site-specific parameters |
required |
Attributes
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data_hub
|
CRNSDataHub
|
A CRNSDataHub hub instance |
required |
process_config
|
The process YAML as an object. |
required | |
sensor_config
|
The station information YAML as an object |
required |
resolve_biomass_column ¶
Return the column the biomass correction should read.
Resolution order, and the reason for it: an explicit configuration field
wins, because a user who names a column has answered the question. Then
the derived site_biomass column, which exists only when a
remote-sensing entry served above-ground biomass, because a satellite
series is the only continuous footprint-representative source and is what
the correction was built for. Then an above_ground_biomass column in
the data, which is both the historical name and where a static
sensor_info.above_ground_biomass lands when static values are written
to the frame. Finally the correction's own default, so an unresolvable
case still produces the existing missing-column diagnostic naming that
column rather than a new one.
select_corrections ¶
Applies all configured corrections to the data hub.
This method processes all correction specifications in the process configuration and applies them to the data hub in the appropriate sequence.
Returns:
| Type | Description |
|---|---|
CRNSDataHub
|
The data hub with all corrections selected and ready to be applied |
QualityAssessmentFromConfig ¶
Builds quality assessment checks from configuration data.
This class translates quality assessment configuration parameters into executable QualityCheck objects that can be applied to CRNS data. It supports various check types and automatically handles parameter inheritance from sensor information (e.g., N0 values).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
partial_config
|
BaseConfig
|
Configuration section containing quality assessment parameters |
required |
sensor_config
|
BaseConfig
|
Sensor configuration containing reference parameters |
required |
name_of_target
|
Literal['raw_neutrons', 'corrected_neutrons']
|
Specific target to process; if None, processes all targets in config |
None
|
Attributes
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
partial_config
|
ConfigurationObject
|
A selection from the ConfigurationObject which stores QA selections |
required |
sensor_config
|
ConfigurationObject
|
The config object describing station variables |
required |
name_of_target
|
str
|
The name of the target for QA. If None it will loop through any provided in partial config. |
None
|
Notes
The name_of_section should match the final part of the supplied partial_config. For example:
partial_config = ( config.process_config.neutron_quality_assessment.flag_raw_neutrons )
Therefore:
name_of_section = 'flag_raw_neutrons'
create_checks ¶
Creates quality check objects based on the provided configuration.
This method processes the configuration and converts it into a list of QualityCheck objects ready to be applied to data.
Returns:
| Type | Description |
|---|---|
List[QualityCheck]
|
List of configured quality check objects |
return_a_check ¶
Process checks for a specific target.
ProcessWithConfig ¶
ProcessWithConfig(path_to_sensor_config=None, path_to_process_config=None, configuration_object=None, status_callback=None)
Bases: ConfiguredPipelineStagesMixin, ConfiguredErrorHandlingMixin, ConfiguredCalibrationMixin, ConfiguredRemoteSensingCorrectionMixin, ConfiguredSnowMaskMixin, ConfiguredValidationMixin
Processes CRNS data according to configuration files.
This class implements the complete CRNS data processing pipeline using configurations for both sensor parameters and processing steps. It handles all stages from raw data import to final data output and visualization.
Users can provide configurations either as file paths or as a pre-configured ConfigurationManager object.
Examples:
>>> # Using string paths to configuration files
>>> sensor_config_path = "/path/to/configuration_files/A101_station.yaml"
>>> processing_config_path = "/path/to/configuration_files/v1_processing_method.yaml"
>>>
>>> # Initialize the processor with paths to config files
>>> config_processor = ProcessWithConfig(
... path_to_sensor_config=sensor_config_path,
... path_to_process_config=processing_config_path
... )
>>>
>>> # Run the full processing pipeline
>>> config_processor.run_full_process()
>>>
>>> # Alternatively, with a pre-configured ConfigurationManager:
>>> config_manager = ConfigurationManager()
>>> config_manager.load_configuration(file_path=sensor_config_path)
>>> config_manager.load_configuration(file_path=processing_config_path)
>>>
>>> # Initialize using the configuration manager
>>> config_processor = ProcessWithConfig(configuration_object=config_manager)
>>> config_processor.run_full_process()
Initialise configuration state and the optional user-status channel.
Exactly one configuration source is resolved by the existing configuration
setup logic. When status_callback is provided, construction emits short
frontend-neutral session status events while keeping developer logging
separate. Callback failures are isolated by the status emitter and must not
change processing behavior.
enable_output
instance-attribute
¶
processing_context
instance-attribute
¶
processing_context = ProcessingContext(column_labels=create_registry(), output_enabled=enable_output, report_enabled=bool(getattr(data_storage, 'create_report', False)), status_callback=status_callback)
run_stage ¶
Run exactly one configured pipeline stage.
Earlier stages must already be completed or explicitly skipped. Use
:meth:run_until when an interactive client wants Neptoon to execute
prerequisites automatically.
run_until ¶
Run all remaining configured stages through and including stage.
run_full_process ¶
Run the full workflow inside runtime and server-monitor contexts.
User-facing errors return self with completed_successfully set
to False and exit_reason populated. Unexpected implementation
errors cross the public boundary as NeptoonInternalError while
preserving the original exception as the chained cause. When
sensor.server_runtime is enabled,
an atomic status file and exclusive run lock make the result observable
to an external supervisor.
run_or_raise ¶
Run the workflow and raise when a controlled failure was recorded.
This convenience API is intended for schedulers and service wrappers
that should never forget to inspect completed_successfully. Progress
and cooperative cancellation use the same callbacks/tokens as
:meth:run_full_process.
build_calibration_metadata ¶
build_calibration_metadata(sensor_config, process_config, hours_around_calib, window_alignment, match_smoothing_window, calibration_comparison, fit_diagnostics=None, design_diagnostics=None, footprint_assumption='homogeneous_radial_schroen_2017')
Assemble the CalibrationMetadata from in-scope calibration state.
binding_context and free_context record the FULL calibration-
relevant config blocks (Markus-scope reproducibility) plus the compact
summary keys the human-readable views read.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
sensor_config
|
The validated configs (post-calibration; sensor_info already carries the freshly written N0 and site values). |
required | |
process_config
|
The validated configs (post-calibration; sensor_info already carries the freshly written N0 and site values). |
required | |
hours_around_calib
|
float
|
The already-resolved calibration window settings (reused from
|
required |
window_alignment
|
float
|
The already-resolved calibration window settings (reused from
|
required |
match_smoothing_window
|
float
|
The already-resolved calibration window settings (reused from
|
required |
calibration_comparison
|
List[CalibrationComparisonDay]
|
Per-day comparison from :func: |
required |
Returns:
| Type | Description |
|---|---|
CalibrationMetadata
|
The strict, emitted artifact (not written here). |
resolve_calibration_window_alignment ¶
resolve_calibration_window_alignment(match_smoothing_window, smooth_corrected_neutrons, explicit_alignment=None, smoothing_alignment='centered')
Resolve how the calibration neutron-averaging window is placed relative to the calibration timestamp.
Centred on the timestamp by default. Only when the user opts in to follow the smoothing window and corrected-neutron smoothing is enabled does it use the configured smoothing alignment, so its mean matches the rolling-mean value at the timestamp. An explicit value always wins.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
match_smoothing_window
|
bool
|
User opt-in to make the calibration window follow the smoothing window. |
required |
smooth_corrected_neutrons
|
bool
|
Whether corrected-neutron smoothing is enabled in the pipeline. |
required |
explicit_alignment
|
str | None
|
An explicitly configured alignment ("centered" or "trailing"), by default None. |
None
|
smoothing_alignment
|
str
|
Alignment configured for data smoothing, by default |
'centered'
|
Returns:
| Type | Description |
|---|---|
str
|
"trailing" or "centered". |
resolve_calibration_window_hours ¶
resolve_calibration_window_hours(match_smoothing_window, smooth_corrected_neutrons, smoothing_window, explicit_hours=None, default_hours=6.0)
Resolve the total span (in hours) of the neutron-averaging window used during calibration.
The calibration window controls a systematic calibration quantity and
is decoupled from the analysis smoothing window by default: it must not
silently change just because a downstream post-processing setting
changes. Coupling to the smoothing window is an explicit opt-in
(match_smoothing_window).
Resolution order: 1. An explicit value (e.g. from the sensor config) always wins. 2. Otherwise, if the user opted in and corrected-neutron smoothing is enabled, follow the smoothing window (parsed to hours, robust to sub-hour windows). 3. Otherwise, fall back to the default span.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
match_smoothing_window
|
bool
|
User opt-in to make the calibration window follow the smoothing window. |
required |
smooth_corrected_neutrons
|
bool
|
Whether corrected-neutron smoothing is enabled in the pipeline. |
required |
smoothing_window
|
str
|
The smoothing window string (e.g. "12h", "30m", "1d"). |
required |
explicit_hours
|
float | None
|
An explicitly configured window span in hours, by default None. |
None
|
default_hours
|
float
|
Default span when not coupling, by default 6.0. |
6.0
|
Returns:
| Type | Description |
|---|---|
float
|
The total calibration window span in hours. |
neptoon.workflow.configured_pipeline¶
configured_pipeline ¶
Explicit stages for the configured single-sensor processing pipeline.
The public :class:~neptoon.workflow.process_with_config.ProcessWithConfig
facade owns configuration and all domain extension points. This mixin keeps
its scientific call order visible while grouping the former monolithic
_run_full_process_impl body into cohesive, independently testable stages.
ConfiguredPipelineStagesMixin ¶
Pipeline-stage implementation used by ProcessWithConfig.
Stage methods return True when processing should continue and False
after a controlled user/data failure has already been recorded on the
workflow object.
get_stage_statuses ¶
Return pipeline stage state in scientific execution order.
neptoon.workflow.configured_errors¶
configured_errors ¶
ConfiguredErrorHandlingMixin ¶
Focused implementation details extracted from ProcessWithConfig.
neptoon.workflow.configured_calibration¶
configured_calibration ¶
ConfiguredCalibrationMixin ¶
Focused implementation details extracted from ProcessWithConfig.
resolve_calibration_window_hours ¶
resolve_calibration_window_hours(match_smoothing_window, smooth_corrected_neutrons, smoothing_window, explicit_hours=None, default_hours=6.0)
Resolve the total span (in hours) of the neutron-averaging window used during calibration.
The calibration window controls a systematic calibration quantity and
is decoupled from the analysis smoothing window by default: it must not
silently change just because a downstream post-processing setting
changes. Coupling to the smoothing window is an explicit opt-in
(match_smoothing_window).
Resolution order: 1. An explicit value (e.g. from the sensor config) always wins. 2. Otherwise, if the user opted in and corrected-neutron smoothing is enabled, follow the smoothing window (parsed to hours, robust to sub-hour windows). 3. Otherwise, fall back to the default span.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
match_smoothing_window
|
bool
|
User opt-in to make the calibration window follow the smoothing window. |
required |
smooth_corrected_neutrons
|
bool
|
Whether corrected-neutron smoothing is enabled in the pipeline. |
required |
smoothing_window
|
str
|
The smoothing window string (e.g. "12h", "30m", "1d"). |
required |
explicit_hours
|
float | None
|
An explicitly configured window span in hours, by default None. |
None
|
default_hours
|
float
|
Default span when not coupling, by default 6.0. |
6.0
|
Returns:
| Type | Description |
|---|---|
float
|
The total calibration window span in hours. |
resolve_calibration_window_alignment ¶
resolve_calibration_window_alignment(match_smoothing_window, smooth_corrected_neutrons, explicit_alignment=None, smoothing_alignment='centered')
Resolve how the calibration neutron-averaging window is placed relative to the calibration timestamp.
Centred on the timestamp by default. Only when the user opts in to follow the smoothing window and corrected-neutron smoothing is enabled does it use the configured smoothing alignment, so its mean matches the rolling-mean value at the timestamp. An explicit value always wins.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
match_smoothing_window
|
bool
|
User opt-in to make the calibration window follow the smoothing window. |
required |
smooth_corrected_neutrons
|
bool
|
Whether corrected-neutron smoothing is enabled in the pipeline. |
required |
explicit_alignment
|
str | None
|
An explicitly configured alignment ("centered" or "trailing"), by default None. |
None
|
smoothing_alignment
|
str
|
Alignment configured for data smoothing, by default |
'centered'
|
Returns:
| Type | Description |
|---|---|
str
|
"trailing" or "centered". |
neptoon.workflow.configured_remote_sensing¶
configured_remote_sensing ¶
Remote sensing as an input to a neutron correction, inside the workflow.
Neptoon attaches remote-sensing series at the end of the pipeline, which is the right place for a reference series: the CRNS time grid is final by then and the series is joined onto it. A correction input cannot wait that long, because the correction it feeds is stage 6 and calibration is stage 7.
No scientific stage moves. That is the whole reason this is a second attachment point rather than a reordering: correction is already before calibration, so N0 is already refitted after the correction. Only the attachment comes earlier, at the end of the alignment stage, after timestamp alignment, so the index is the one the corrections use, and before the correction stage, so the column is there when the correction reads it. Reference series keep the stage they have always had, which is also the path every earlier release tested.
What this mixin owns, in the order it happens:
- deciding which variables are correction inputs in this run, a property of the processing configuration, not of the remote-sensing block;
- attaching them early and deriving the column the correction reads;
- the N0 guard, which is the load-bearing part: a published N0 fitted without a biomass correction already contains the vegetation, so applying the correction on top subtracts it twice. Measured at one forest station: mean volumetric soil moisture falls from 0.205 to 0.051 m3/m3. That is not a result and must not be presentable as one. A user who knows the correction context of their own N0 can say so in the configuration and keep it; the guard then steps aside and states the fact in the log, the processing output and the PDF report instead;
- propagating the input's provenance markers onto the soil moisture derived from it, and recording what the correction did in the report and the audit.
ConfiguredRemoteSensingCorrectionMixin ¶
Attach, derive, guard and record a remote-sensing correction input.
neptoon.workflow.configured_snow_mask¶
configured_snow_mask ¶
Snow flagging inside the configured workflow.
The scientific decision lives in
:mod:neptoon.external.remote_sensing.masking; this module only decides when
it happens and what the user is told.
When. The snow condition is captured before temporal aggregation and written to the output once the soil-moisture product is final. Capturing it early is the point of the two steps: mean-aggregating a snow depth to a daily row can push a short but strongly contaminated period below the configured threshold, so the condition is evaluated at source resolution and carried through aggregation as a worst-case flag.
Nothing is removed. The flag is a column beside the product, and the product is exactly what it would have been without it. Dropping those hours is a decision the user takes on their own data, and the user output says so in the same breath as the count, because a flag nobody notices is a flag nobody acts on.
Why not a quality check. Neptoon's quality machinery is SaQC on a column's own values, and it deliberately excludes gridded provider columns from the flag tables: an outlier test on a 9 km reanalysis cell over an hourly CRNS index would find its own interpolation, and a stale gridded epoch must never be able to invalidate a CRNS result on its own terms. A snow flag is the opposite relationship -- a cross-column condition asserted by the reanalysis about the CRNS product -- so it is an explicit step with its own column, not a flag row for the snow column. Whether it should also reach the flagging DataFrame is an open question from the review, and it is deliberately not answered here.
What the user is told. A missing snow column when flagging is enabled is a configuration error and not a skipped step, because the failure it would otherwise cause is silent: no snow value means no flag, which is indistinguishable in the output from a snow-free record.
neptoon.workflow.configured_validation¶
configured_validation ¶
ConfiguredValidationMixin ¶
Focused implementation details extracted from ProcessWithConfig.
neptoon.workflow.quality_from_config¶
quality_from_config ¶
QualityAssessmentFromConfig ¶
Builds quality assessment checks from configuration data.
This class translates quality assessment configuration parameters into executable QualityCheck objects that can be applied to CRNS data. It supports various check types and automatically handles parameter inheritance from sensor information (e.g., N0 values).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
partial_config
|
BaseConfig
|
Configuration section containing quality assessment parameters |
required |
sensor_config
|
BaseConfig
|
Sensor configuration containing reference parameters |
required |
name_of_target
|
Literal['raw_neutrons', 'corrected_neutrons']
|
Specific target to process; if None, processes all targets in config |
None
|
Attributes
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
partial_config
|
ConfigurationObject
|
A selection from the ConfigurationObject which stores QA selections |
required |
sensor_config
|
ConfigurationObject
|
The config object describing station variables |
required |
name_of_target
|
str
|
The name of the target for QA. If None it will loop through any provided in partial config. |
None
|
Notes
The name_of_section should match the final part of the supplied partial_config. For example:
partial_config = ( config.process_config.neutron_quality_assessment.flag_raw_neutrons )
Therefore:
name_of_section = 'flag_raw_neutrons'
create_checks ¶
Creates quality check objects based on the provided configuration.
This method processes the configuration and converts it into a list of QualityCheck objects ready to be applied to data.
Returns:
| Type | Description |
|---|---|
List[QualityCheck]
|
List of configured quality check objects |
return_a_check ¶
Process checks for a specific target.
neptoon.workflow.corrections_from_config¶
corrections_from_config ¶
CorrectionSelectorFromConfig ¶
Selects and configures neutron count corrections based on configuration.
This class translates correction configuration parameters into the appropriate correction objects to be applied to neutron count data. It supports various correction types including pressure, humidity, incoming intensity, and above-ground biomass.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data_hub
|
CRNSDataHub
|
The data hub instance to which corrections will be applied |
required |
process_config
|
BaseConfig
|
Process configuration containing correction specifications |
required |
sensor_config
|
BaseConfig
|
Sensor configuration containing site-specific parameters |
required |
Attributes
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data_hub
|
CRNSDataHub
|
A CRNSDataHub hub instance |
required |
process_config
|
The process YAML as an object. |
required | |
sensor_config
|
The station information YAML as an object |
required |
resolve_biomass_column ¶
Return the column the biomass correction should read.
Resolution order, and the reason for it: an explicit configuration field
wins, because a user who names a column has answered the question. Then
the derived site_biomass column, which exists only when a
remote-sensing entry served above-ground biomass, because a satellite
series is the only continuous footprint-representative source and is what
the correction was built for. Then an above_ground_biomass column in
the data, which is both the historical name and where a static
sensor_info.above_ground_biomass lands when static values are written
to the frame. Finally the correction's own default, so an unresolvable
case still produces the existing missing-column diagnostic naming that
column rather than a new one.
select_corrections ¶
Applies all configured corrections to the data hub.
This method processes all correction specifications in the process configuration and applies them to the data hub in the appropriate sequence.
Returns:
| Type | Description |
|---|---|
CRNSDataHub
|
The data hub with all corrections selected and ready to be applied |
resolve_reference_pressure_from_sensor_info ¶
Return configured mean pressure or a standard-atmosphere estimate from elevation.
The one-row service accepts inline JSON or exactly one JSON/GeoJSON/CSV record.
File-not-found, encoding, JSON syntax, CSV parser, duplicate-column, and row-count
problems raise NeptoonInputDataError. The CLI already treats that class as a
recoverable operator error, while direct Python callers can catch the same
stable exception type.
neptoon.workflow.single_row¶
single_row ¶
Single-row CRNS conversion workflow used by the CLI.
The normal :class:~neptoon.workflow.process_with_config.ProcessWithConfig
workflow is intentionally time-series oriented: it can complete, align,
interpolate, aggregate, quality-control, smooth and report whole datasets. For
API-style use cases it is useful to run exactly one already-supplied row through
only the deterministic conversion steps. This module implements that narrow
path while reusing the existing configuration models, input-column formatting,
correction selection and soil-moisture conversion code.
SingleRowConfigurationError
dataclass
¶
SingleRowConversionResult
dataclass
¶
Result returned by :class:SingleRowConversionProcessor.
build_output_frame ¶
Return a one-row frame with the datetime index exposed as a column.
SingleRowFormatDataForCRNSDataHub ¶
Bases: FormatDataForCRNSDataHub
Formatter variant that accepts exactly one timestamped input row.
The standard formatter requires at least two timestamps so it can infer a time step from the index. A one-row command cannot do that reliably. This subclass keeps all normal parsing, unit conversion and column-mapping logic but injects a configured/CLI-provided timestep in seconds.
SingleRowConversionProcessor ¶
Run one row through correction and soil-moisture conversion only.
workflow
instance-attribute
¶
workflow = ProcessWithConfig(configuration_object=configuration_object)
run ¶
Convert exactly one row and return the processed one-row frame.
This path deliberately does not run time-series completion, timestamp alignment, missing-value interpolation, QA, temporal aggregation, smoothing, figure creation or file/report output.
read_single_row_input ¶
Read one input row from JSON text or a one-row JSON/CSV file.
Expected filesystem, decoding and parser failures are translated into an
actionable :class:~neptoon.exceptions.NeptoonInputDataError. This keeps
the public helper safe for direct service use as well as for the CLI.
infer_single_row_timestep_seconds ¶
Infer the integration timestep used by the one-row formatter.
prepare_single_row_time_series ¶
Format one raw/prepared input row using the existing sensor config.