Sensor configuration¶

Overview¶

The sensor configuration file tells neptoon about the sensor and data source being processed. The main sections are config, sensor_info, raw_data_parse_options, time_series_data, time_series_completion, missing_value_interpolation, dwd, external_meteo, external_soil_moisture, input_data_qa, soil_moisture_qa, calibration, data_storage, and figures. Below is an example file which you can use as a starting point for your own sensor, followed by a quick reference and a detailed reference.

Some of the inputs can be left blank initially, and neptoon will calculate the values during processing (e.g., N0 when calibrating). The output folder when processing is complete will include a file called sensor_config.yaml. This is the config file used as input with any additional calculated information included.

Top-level behaviour controls¶

Parameter	Default	Description
`enable_output`	`false`	Emit optional user-facing progress messages from core processing methods. Logging remains active regardless of this setting.
`qa_missing_target_policy`	`skip`	`skip` optional QA checks whose target column is unavailable, or `raise` a configuration/runtime error.
`processing_missing_policy`	`keep_nan`	Controls missing or invalid values produced during processing: preserve them as `NaN`, `raise`, or `drop` affected rows. This is separate from raw-row parsing policy.
`working_directory`	deprecated	Legacy path-basis key. Relative paths now resolve against the YAML file and this key should not be used in new configurations.

File Structure¶

config: sensor

sensor_info:
  name: Example_station
  country: DEU
  identifier: EXAMPLE001
  install_date: 2024-01-01
  latitude: 50.0
  longitude: 8.0
  elevation: 100.0
  time_zone: 0
  site_cutoff_rigidity: 4.0
  avg_lattice_water: 0.004
  avg_soil_organic_carbon: 0.018
  avg_dry_soil_bulk_density: 1.43
  N0: 1100

raw_data_parse_options:
  parse_raw_data: false

time_series_data:
  path_to_data: ./data/site.csv
  key_column_info:
    epithermal_neutron_columns:
      - N_epithermal
    neutron_count_units: counts_per_hour
    pressure_columns:
      - P_hPa
    pressure_units: hectopascals
    temperature_columns:
      - T_degC
    temperature_units: celsius
    relative_humidity_columns:
      - RH_percent
    relative_humidity_units: percent
    date_time_columns:
      - timestamp
    date_time_format: "%Y-%m-%d %H:%M:%S"
    initial_time_zone: utc
    convert_time_zone_to: utc

data_storage:
  save_location: ./outputs
  save_csv: true
  create_report: false

Configuration quick reference guide¶

Sensor Information¶

This config section describes the sensor being processed. Some values, such as elevation and latitude, are used in calculations. Others, such as the station name, are used in output paths and reports.

It is always better to fill this in as best you can.

The beta_coefficient can be automatically calculated if left blank using elevation and latitude

Same for the N0 however this requires the calibration section to be correctly filled (otherwise you'll have to guess). Generally speaking when calibrating it's also possible to automatically generate the values for avg_lattice_water and avg_soil_organic_carbon, as long as they are available in your calibration dataset.

Parameter	Required	Type	Example	Description
name	Yes	string	`Cunnesdorf_test_site`	Site identifier used for file naming and metadata
country	No	string	`DEU`	Country where sensor is located
identifier	No	string	`A102`	Unique sensor identifier code
install_date	No	string/null	`2016-10-21`	Date sensor was installed (YYYY-MM-DD); omit or use `null` when unknown
latitude	Yes	float	`51.369597`	Site latitude in decimal degrees
longitude	Yes	float	`12.557120`	Site longitude in decimal degrees
elevation	Yes	float	`113`	Site elevation in meters
time_zone	No	integer	`0`	Legacy station UTC-offset metadata; defaults to `0`. Raw timestamp localisation uses `key_column_info.initial_time_zone`.
site_cutoff_rigidity	No	float	`2.94`	Geomagnetic cutoff rigidity in GV
avg_lattice_water	No	float	`0.0043`	Average lattice water content as decimal (e.g., 0.0043 = 0.43%)
avg_soil_organic_carbon	No	float	`0.0184`	Soil organic carbon content as decimal (e.g., 0.0184 = 1.84%)
avg_dry_soil_bulk_density	No	float	`1.6`	Dry soil bulk density in g/cm³
N0	No	float	`1100`	Calibration parameter for neutron-to-soil moisture conversion
beta_coefficient	No	float	-	Site-specific coefficient for pressure correction
l_coefficient	No	float	-	Legacy alias retained for old station files; use `beta_coefficient` in new configurations
mean_pressure	No	float	-	Reference atmospheric pressure for corrections. When omitted, Neptoon estimates it from site elevation where a reference pressure is needed.
avg_precipitation	No	float	-	Legacy metadata field; accepted for compatibility but excluded from rewritten configuration files
avg_soil_moisture	No	float	-	Legacy metadata field; accepted for compatibility but excluded from rewritten configuration files
avg_biomass	No	float	-	Legacy metadata field; accepted for compatibility but excluded from rewritten configuration files

Raw Data Parse Options¶

Raw Data Parsing refers to the required step to take your raw data files (e.g., found on the SD card in the logger) and converting them to a single csv file. No data manipulation is done at this stage. In its simplest form it will take a list of .txt files and order them by date. When things get more complicated (e.g., only files with a certain prefix contain CRNS data), other settings are required.

Is this step needed?

You can skip this step if your data is already available as a single csv file. Simply set parse_raw_data to False and move on to time series data below.

Requirements

For this section the Required column will change if you select True for parse_raw_data.

Parameter	Required	Type	Example	Description
parse_raw_data	Yes	boolean	`True`	Toggle for raw data parsing functionality. When False, this entire section is ignored
data_location	No	Path or List[Path]	`"data/raw"` or `["data/2025-01", "data/2025-02"]`	One raw-data file/directory/archive or an ordered list of local inputs
recursive	No	boolean	`True`	Search local `data_location` directories recursively. Independent of `sftp.recursive`
column_names	No	List[str]	`["date", "time", "counts"]`	Expected column names in order. If not provided, will attempt auto-detection
prefix	No	string	`"CRNS_"`	Filter raw files by filename prefix
suffix	No	string	`".dat"` or `.txt`	Filter raw files by filename suffix
filename_include_regex	No	string	`"^2024/.+\\.dat$"`	Include raw files by regular expression after prefix/suffix filtering
filename_exclude_regex	No	string	`"(^\|/)archive/"`	Exclude raw files by regular expression after include filtering
encoding	No	string	`"cp850"`	File encoding format. Common alternatives: utf-8, ascii
skip_lines	No	integer	`2`	Fixed number of header/metadata lines to skip before data
data_start_regex	No	string/null	`^\s\d+\s,`	Per-file pattern for the first real data row; overrides fixed `skip_lines` for data parsing
separator	No	string	`","`	Column delimiter character (e.g., comma, tab, semicolon)
decimal	No	string	`"."`	Decimal point character.
skip_initial_space	No	boolean	`True`	Remove leading whitespace in data fields
parser_kw	No	object	-	Advanced parser configuration
├─ strip_left	No	boolean	`True`	Remove leading whitespace in fields
├─ digit_first	No	boolean	`True`	Expect numeric data at start of line
starts_with	No	string	`"#"`	Required prefix for header lines
multi_header	No	boolean	`False`	Support for multi-line header formats
strip_names	No	boolean	`True`	Remove whitespace from column names
remove_prefix	No	string	`"//"`	Remove lines that start with this
parallel_file_parsing	No	boolean	`False`	Parse multiple folder-based raw files concurrently
file_parse_workers	No	integer/null	`null`	Worker thread limit for parallel raw-file parsing
regexp_in	No	string/null	`^#`	Legacy line-level regular expression pattern applied before row parsing
regexp_out	No	string/null	``	Replacement string used with `regexp_in`
invalid_data_policy	No	string	`"drop_required"`	How to handle raw rows with too few fields or empty fields. Allowed values: `"drop_required"`, `"drop"`, `"raise"`, `"keep"`
validate_column_count	No	boolean	`true`	Compare raw row widths with the configured or detected schema before parsing.
extra_columns_policy	No	string	`"trim"`	Handle rows wider than `column_names`: `trim`, `drop`, or `raise`.
schema_mismatch_policy	No	string	`"warn"`	Handle a dominant row width that differs from `column_names`: `warn`, `raise`, or `auto`.
schema_sample_size	No	integer	`10000`	Maximum rows sampled for row-shape/schema diagnostics.
line_error_report_limit	No	integer	`20`	Maximum malformed-row examples logged before only aggregate counts are reported.
missing_value_tokens	No	list	`["*"]`	Non-empty raw tokens treated as missing before numeric conversion.
parser_mode	No	string	`"auto"`	Parser strategy: `auto`, `direct`, `fast`, or `robust`.
diagnostics_level	No	string	`"summary"`	Raw row-shape diagnostics: `off`, `summary`, or `full`.
sftp.enabled	No	boolean	`false`	Download raw files from an SFTP server instead of `data_location` before parsing
sftp.host	Required when SFTP enabled	string	`sftp.example.org`	SFTP host name
sftp.remote_path	Required when SFTP enabled	string	`/data/crns`	Remote file or directory to download
sftp.file_glob	No	string	`*.csv`	Remote file pattern used after traversal
archive_safety	No	object	defaults below	Limits member count, expanded sizes, compression ratio, nesting, and required free disk space before ZIP/TAR extraction

Additional Information

Paths in data_location can be absolute or relative to the configuration file
When column_names is not provided, the parser attempts to detect headers from the first file
For compressed data, both .zip and .tar formats are automatically detected and extracted

`parallel_file_parsing`¶

Description Optional switch for parsing multiple raw logger files concurrently when data_location points to a normal folder or an archive that can be safely extracted before parsing. This is intended for inputs containing many independent raw files; single-file inputs remain sequential.

Specification

Type: boolean
Required: No
Default: false
Example: true

`file_parse_workers`¶

Description Maximum number of worker threads used when parallel_file_parsing is enabled. Leave as null to let neptoon choose a conservative bounded value. The CLI option --workers N can override this at runtime without editing the YAML.

Specification

Type: integer or null
Required: No
Default: null
Example: 4

Runtime worker override

For existing sensor YAML files, you can enable parallel raw-file parsing without editing the configuration:

neptoon -p process.yaml -s sensor.yaml --workers 4

Use --no-parallel to disable runtime parallelism for a run, even when the YAML enables it.

Legacy raw-parser keys

Older configs may contain regexp_in and regexp_out. These keys remain accepted. When both are set, neptoon applies re.sub(regexp_in, regexp_out, line) before normal raw-row parsing. Empty values mean no replacement.

`invalid_data_policy`¶

Description
Controls how raw rows with structural problems or missing values are handled. The default defers value-level decisions until the configured timestamp and neutron inputs are known, so empty optional or unused columns do not remove otherwise usable measurements.

Specification

Type: string
Required: No
Allowed values: "drop_required", "drop", "raise", "keep"
Default: "drop_required"
Example:

raw_data_parse_options:
  parse_raw_data: true
  invalid_data_policy: "drop_required"

Policy behaviour:

drop_required drops structurally malformed rows, but keeps expected-width rows with empty fields during raw parsing. It later drops only rows without a usable timestamp and core neutron measurement. For absolute counts with configured elapsed-time columns, at least one complete count/live-time pair is required. Empty optional, external-source, or unused raw columns remain NaN.
drop preserves the legacy strict behaviour and drops a row when any raw field is empty or invalid.
raise stops at the first malformed row, empty raw field, or invalid configured value.
keep preserves parseable rows and missing values without the later required-input row filter.

Raw files with empty optional columns

drop_required is designed for logger exports that contain unused empty columns. Use legacy drop only when every raw field is mandatory for every row.

Schema diagnostics and parser strategy¶

The parser can validate the observed row shape before the full import. These options are useful for long logger archives where a firmware change, partial transmission, or extra delimiter creates a second schema.

raw_data_parse_options:
  validate_column_count: true
  schema_mismatch_policy: warn
  schema_sample_size: 10000
  extra_columns_policy: trim
  line_error_report_limit: 20
  missing_value_tokens: ["*", "NA"]
  parser_mode: auto
  diagnostics_level: summary

schema_mismatch_policy: raise stops before processing when the dominant sampled row width does not match column_names.
warn logs the mismatch but keeps the configured schema.
auto truncates or extends the configured names with generated extra_column_* labels. Use it only when downstream mappings have been reviewed.
parser_mode: auto uses a one-pass pandas reader for clean tables whose configuration needs no row filtering or repair, otherwise it uses the fast split path for simple delimiters and the robust CSV parser when necessary. direct requires the one-pass path and fails clearly when incompatible row-repair options are enabled; fast forces split-based normalization; robust always uses CSV row parsing.
diagnostics_level: full includes example malformed rows and can be noisy for large archives. summary is the recommended default.
Empty strings are already handled by the CSV parser. Blank values in missing_value_tokens are ignored deliberately.

SFTP raw-data source¶

When raw logger files are stored on an SFTP server, configure raw_data_parse_options.sftp and leave data_location blank. Neptoon downloads matching files to a local temporary or cache directory and then uses the normal raw parser. Install the optional SFTP dependency with pip install "neptoon[sftp]" or pipx install "neptoon[sftp]".

raw_data_parse_options:
  parse_raw_data: true
  data_location: null
  sftp:
    enabled: true
    host: sftp.example.org
    username: crns-reader
    remote_path: /incoming/station_A101
    file_glob: "*.csv"
    recursive: true
    password_env: NEPTOON_SFTP_PASSWORD
    host_key_policy: reject

Prefer password_env, private_key_path, SSH agent authentication, or system keys over plaintext passwords in YAML files. Use host_key_policy: reject for production unless you deliberately accept new host keys in a controlled setup.

Time Series Data¶

The time series data section is interested in how we prepare your CRNS time series data for processing. It imagines that your data is at least in a datetime ordered csv format. You state where that data is with the path_to_data setting and it will read it in and begin preparations. If you needed to run the above "Raw Data Parse Options" stage, the path to the data is not really needed, but the remaining settings are!

Parameter	Required	Type	Example	Description
path_to_data	No	string	-	Path to pre-processed data (leave blank if parsing raw data)
time_step_resolution	No	string	`1h`	Legacy sampling-resolution hint accepted for old YAML files; configure current temporal behaviour in the process YAML
key_column_info.cosmos_network	No	string	-	Legacy compatibility metadata accepted in old YAML files and excluded from rewritten configuration output

Time Series Completion and Missing-Value Interpolation¶

Time-series completion creates a regular timestamp grid before optional gap filling. Missing-value interpolation is a separate step and only touches explicitly configured numeric columns. This separation is important for aggregation: low-coverage aggregation bins may become NaN, and interpolation decides later whether such gaps may be filled.

time_series_completion:
  enabled: true
  target_frequency: auto
  create_missing_timestamps: true
  duplicate_timestamp_policy: first

missing_value_interpolation:
  enabled: false
  columns:
    - air_temperature
    - air_relative_humidity
    - air_pressure
  method: time
  max_interpolation_hours: 12
  missing_column_policy: skip
  non_numeric_policy: skip

Parameter	Required	Type	Example	Description
time_series_completion.enabled	No	boolean	`true`	Insert missing timestamps on a regular target grid
time_series_completion.target_frequency	No	string	`auto` or `1h`	Target frequency for the completed grid
time_series_completion.create_missing_timestamps	No	boolean	`true`	Keep inserted rows with `NaN` values
time_series_completion.duplicate_timestamp_policy	No	string	`first`, `last`, or `raise`	How duplicate timestamps are handled before reindexing
missing_value_interpolation.enabled	No	boolean	`false`	Interpolate configured columns after time-series completion
missing_value_interpolation.columns	No	list	`[air_temperature]`	Columns eligible for interpolation
missing_value_interpolation.method	No	string	`time` or `linear`	Pandas interpolation method
missing_value_interpolation.max_interpolation_hours	No	float/null	`12`	Maximum consecutive gap length to fill; longer gaps remain missing

Key Column Information (Time Series Data)¶

Here we define settings to prepare the data for processing. For example in neptoon we standardise all neutron counts to counts per hour (cph). So if your data is in another format, state it here and neptoon will take care of the conversion.

Other things include if you have multiple columns of certain data readings (e.g., multiple pressure sensors). State the names in a list under the specific column section and state how you wish to merge them into a single column. priority means it will use the first value in the list and gap fill with the next if missing. average means it will take the mean.

Column names can also carry compact metadata using column[min,max]:dtype:aggregation. This is useful for preserving integer flags, marking cumulative variables such as precipitation, or documenting that a column is already an interval mean such as neutron counts per hour. For a full explanation, see Columns in neptoon.

time_series_data:
  key_column_info:
    epithermal_neutron_columns:
      - N_cph[0,inf]:float:mean
    pressure_columns:
      - iPress[900,1100]:float:mean
    temperature_columns:
      - Tair[-40,60]:float:mean
    relative_humidity_columns:
      - RH[0,100]:float:mean

Parameter	Required	Type	Example	Description
epithermal_neutron_columns	Yes	list	`[N2Cts]`	Columns containing epithermal neutron counts
thermal_neutron_columns	No	list	`[N1Cts]`	Columns containing thermal neutron counts
neutron_count_units	Yes	string	`absolute_count` or `counts_per_hour` or `counts_per_second`	Units for neutron measurements
pressure_columns	No	list or null	`[P4_mb, P3_mb, P1_mb]` or `null`	Local pressure columns in priority order. Use `null` when pressure is supplied later by DWD, external meteo, fallback, or an inactive correction.
pressure_units	Conditional	string	`hectopascals`	Required only when local `pressure_columns` are configured. Units for pressure measurements.
pressure_merge_method	Yes	string	`priority`	How to handle multiple pressure columns
temperature_columns	No	list or null	`[T1_C, T2_C]` or `null`	Local temperature columns. Use `null` when temperature is supplied later by DWD, external meteo, fallback, or an inactive correction.
temperature_units	Conditional	string	`celcius`	Required only when local `temperature_columns` are configured. Units for temperature measurements.
temperature_merge_method	Yes	string	`priority`	How to handle multiple temperature columns
relative_humidity_columns	No	list or null	`[RH1]` or `null`	Local relative humidity columns. Use `null` when relative humidity is supplied later by DWD, external meteo, fallback, or an inactive correction.
relative_humidity_units	Conditional	string	`percent`	Required only when local `relative_humidity_columns` are configured. Units for humidity measurements.
relative_humidity_merge_method	No	string	`priority`	Merge multiple humidity columns using `priority`, `priority_fill`, `average`, or `mean`
date_time_columns	Yes	list	`[Date Time(UTC)]`	Columns containing date/time data
date_time_format	Conditional	string	`"%Y/%m/%d %H:%M:%S"`	`strptime` format for text timestamps; ignored when `is_timestamp: true`
is_timestamp	No	boolean	`false`	Treat the single datetime column as Unix epoch values instead of formatted text
timestamp_unit	No	string	`s`	Unix epoch unit: `s`, `ms`, `us`, or `ns`
datetime_parse_error_policy	No	string	`warn`	Handle invalid parsed timestamps with `warn`, `raise`, or `ignore`
datetime_parse_error_fraction	No	float	`0.05`	Invalid timestamp fraction at which `warn`/`raise` escalates
datetime_parse_error_report_limit	No	integer	`5`	Maximum invalid timestamp examples included in diagnostics
timezone_ambiguous	No	string	`NaT`	DST fall-back handling: `raise`, `infer`, or `NaT`
timezone_nonexistent	No	string	`shift_forward`	DST spring-forward handling: `raise`, `NaT`, `shift_forward`, or `shift_backward`

Time Formats

DateTime format strings must be enclosed in quotes (e.g., "%Y/%m/%d %H:%M:%S") to comply with YAML syntax.

Column metadata fields¶

column_value_ranges, column_dtypes, and column_aggregations are populated automatically when you use inline metadata such as rain_mm[0,inf]:float:sum. They can also be provided explicitly when inline notation would be unclear.

time_series_data:
  key_column_info:
    column_value_ranges:
      rain_mm: [0, .inf]
    column_dtypes:
      rain_mm: float
      qc_flag: int
    column_aggregations:
      rain_mm: sum
      qc_flag: first

These hints are used during numeric conversion, missing-value interpolation, and temporal aggregation. Discrete dtypes such as int, bool, string, and category are not interpolated.

Legacy elapsed-time key

neutron_count_elapsed_time_columns is the current name for elapsed-time columns used with absolute neutron counts. Older configs that use nsecs_columns are still accepted and mapped to neutron_count_elapsed_time_columns.

DWD and external meteorological data¶

Meteorological values can come from local input columns, the German DWD provider, or ordered external_meteo fallback providers. Generic providers run in YAML order; the dedicated dwd block is a separate post-alignment path. By default providers fill only missing values unless prefer_dwd_values or prefer_external_values is set. For full examples, see External data.

The special marker DWD can be used in pressure, temperature, or humidity column lists to request DWD filling from the sensor config:

time_series_data:
  key_column_info:
    pressure_columns: [DWD]
    temperature_columns: [DWD]
    relative_humidity_columns: [DWD]

dwd:
  enabled: true
  station: nearest
  fill_pressure: true
  fill_temperature: true
  fill_relative_humidity: true

For generic fallback ordering, list external_meteo blocks in order; provider: dwd is also supported there.

Quality Assessment Settings¶

We include some simple options for quality assessment in neptoon using SaQC as the back-end. More information about how to do this is provided further below on this page (and examples are shown in the example config above).

We do not plan to expand this further to avoid scope creep. Neptoon is designed to process CRNS data. We provide some options to QA data used directly in this process. To QA any additional co-located sensors, we would recommend using a system designed for QA specifically (e.g., SaQC).

Arbitrary columns that are already present in the working data frame can nevertheless use the same built-in checks through input_data_qa.custom_columns. The mapping key is the exact column name and the value is a normal QA block:

input_data_qa:
  custom_columns:
    battery_voltage:
      flag_range:
        min: 10.5
        max: 15.0

This does not create or rename a column; it only registers QA checks for an existing one.

Calibration¶

Calibration finds your N0 term. For this we need sample data acquired from the site. When available the following section tells neptoon where the data is and what the format is.

Parameter	Required	Type	Example	Description
calibrate	Yes	boolean	`True`	Toggle for whether calibration will be done
location	No	string	`home_dir/example_data/FSCD001_calibration.csv`	Location of the calibration data
date_time_format	No	string	`"%d.%m.%Y %H:%M"`	DateTime format of the calibration data
horizontal_weighting_method	No	string	`schroen_etal_2017`	Horizontal sample-weighting method
vertical_weighting_method	No	string	`schroen_etal_2017`	Vertical sample-weighting method
hours_of_data_around_calib	No	float/null	`6`	Total neutron averaging span around each calibration timestamp
calib_window_alignment	No	string/null	`centered`	Place the calibration window `centered` on or `trailing` behind the sample timestamp; `centred` is accepted as an input alias
match_smoothing_window	No	boolean	`false`	Match calibration span/alignment to corrected-neutron smoothing when enabled
metadata_reference	No	path/null	-	Previously written calibration metadata sidecar used for compatibility checks
biomass_correction_acknowledged	No	boolean	`false`	Declare that a reused `N0` may be used with an active above-ground-biomass correction
calibration_comparison_info_threshold	No	float	`0.05`	Relative difference above which a calibration comparison is logged at INFO level

Key Column Names (Calibration)¶

These values are required if calibrate is set to true in the above section.

Parameter	Required	Type	Example	Description
date_time	No	string	`"DateTime_utc"`	Name of the column with DateTime
profile_id	No	string	`"Profile_ID"`	Name of the column with profile ID
sample_depth	No	string	`"Profile_Depth_cm"`	Name of the column with sample depth values
radial_distance_from_sensor	No	string	`"Distance_to_CRNS_m"`	Name of the column with distance of the sample from the sensor (m)
bulk_density_of_sample	No	string	`"DryBulkDensity_g_cm3"`	Name of the column with bulk density of the samples
gravimetric_soil_moisture	No	string	`"SoilMoisture_g_g"`	Name of the column with gravimetric soil moisture values
soil_organic_carbon	No	string	`"SoilOrganicCarbon_g_g"`	Name of the column with soil organic carbon values
lattice_water	No	string	`"LatticeWater_g_g"`	Name of the column with lattice water values

External soil-moisture files¶

Use external_soil_moisture when reference sensors are logged in CSV files separate from the CRNS stream. The sensor YAML maps each file to a stable sensor_id, timestamp column, time zone, measurement depth and optional input unit. Omitted units default to m3/m3. Files with the same layout can share file_template and use a compact files list. See External soil-moisture time series for complete examples and processing semantics.

Parameter	Required	Type	Description
`enabled`	No	boolean	Enable external soil-moisture file input.
`sources`	One style	list	Fully explicit source records.
`file_template`	One style	mapping	Shared timestamp, measurement and CSV parsing layout.
`files`	With template	list	Explicit file-to-`sensor_id` mappings.
`timestamp.column`	Yes	string	Timestamp column in the CSV.
`timestamp.format`	No	string/null	Optional strptime format.
`timestamp.initial_time_zone`	Yes	string	IANA zone for naive timestamps.
`measurements[].column`	Yes	string	Soil-moisture value column.
`measurements[].depth_cm`	Yes	number	Measurement depth in centimetres.
`measurements[].unit`	No	string	Defaults to `m3/m3`; accepts `fraction`/`m3/m3` or `vol_percent`/`vol%`.
`sources[].distance_from_crns_m` / `files[].distance_from_crns_m`	No	number	Horizontal distance to the CRNS detector. Supply it for every profile or for none when the weighted mean is enabled. `distance_m` is accepted as an alias.

Data Storage¶

Parameter	Required	Type	Example	Description
save_location	No	string	`./outputs`, `~/outputs`, or `/data/outputs`	Directory for saving outputs. The legacy key `save_folder` is accepted as an alias. Relative YAML paths resolve against the config-file directory. Relative CLI `--save-location` overrides resolve against the current working directory. If blank, Neptoon next checks `NEPTOON_OUTPUT_DIR`, then uses the current working directory only when it is writable.
save_csv	No	boolean	`true`	Write the standard tabular station outputs as CSV files. At least one of `save_csv` or `save_json` must be true.
save_json	No	boolean	`false`	Additionally write tabular station outputs as JSON files.
json_orient	No	string	`records`	Pandas JSON orient used for table exports. Common choices are `records` for APIs and `table` for schema-aware round trips.
json_date_format	No	`iso` or `epoch`	`iso`	Date format used for JSON table exports.
append_timestamp_to_folder_name	No	boolean	`True`	Whether to append a timestamp to the output folder name. Useful when experimenting to avoid overwriting data.
append_audit_log_hash_to_folder_name	No	boolean	`False`	Whether to append a short audit-log hash to the output folder name for traceability. The legacy key `append_yaml_hash_to_folder_name` is accepted.
create_report	No	boolean	`false`	Whether to create a PDF processing report in the output folder. The report contains station metadata, selected figures, external-source/weather summaries and calibration content when available.
report_plot_downsampling	No	boolean	`true`	Whether report-only plots may be downsampled for readability. Processed CSV/JSON outputs are not downsampled.
output_mode	No	`full`, `selected`, or `both`	`full`	Controls whether Neptoon writes the standard processed-data/flags tables, only an explicitly selected column subset under the standard filenames, or both standard and `_selected` sidecar tables.
output_columns	No	list	`[date_time, smv, crns_measurement_depth_cm]`	Ordered columns used by `output_mode: selected` or `both`. `date_time` and `utc_timestamp` may be selected explicitly. `selected_output_columns` is accepted as an alias.
static_pivot_enabled	No	boolean	`false`	Additionally write a compact export where fully constant columns are moved to a one-row static-values sidecar table.
static_pivot_mode	No	`sidecar` or `replace`	`sidecar`	`sidecar` keeps the normal processed-data table and adds compact sidecars; `replace` writes the compact table as the main processed-data table.
static_pivot_reference_column	No	string	`static_profile_id`	Column inserted into the compact time-series table to refer to the one-row static-values table.
static_pivot_profile_id	No	string/null	`null`	Optional profile id for the compact export. If omitted, Neptoon derives one from the sensor name or output folder.
static_pivot_min_rows	No	integer	`2`	Minimum row count before constant columns may be moved into the static-values table.
static_pivot_exclude_columns	No	list	`[]`	Columns that must never be moved to the static-values table.
data_source	No	string/null	`null`	Legacy output metadata field retained for backwards compatibility. New configs normally leave this unset.

Figures¶

Figures are tightly coupled to the create_report feature above. Neptoon creates diagnostic figures for quick visual checks after processing. Figures are saved in the output folder and included in the PDF report when report creation is enabled.

Parameter	Required	Type	Example	Description
create_figures	No	boolean	`True`	Generate visualization figures. If false, report figures are not generated even when `create_report` is true.
make_all_figures	No	boolean	`True`	Generate all built-in figure types that have their required columns.
precipitation_resolution	No	`auto`, `hourly`, `daily`	`auto`	Controls precipitation bars in report soil-moisture plots. `auto` switches from hourly to daily for long reports; `hourly` forces hourly bars when an hourly column is available and falls back to daily otherwise; `daily` always uses daily totals.
custom_list	No	list	`[nmdb_incoming_radiation]`	List of specific built-in figures to generate when `make_all_figures` is false.
soil_moisture_sensor_comparison	No	mapping/list	`{columns: [smt1SoilMoist], include_external_crns_weighted_mean: true, external_crns_weighted_mean_label: CRNS-weighted in-situ mean, display_unit: fraction}`	Optional CRNS/in-situ soil-moisture comparison plot. Per-column `units` describe input values (`fraction`, `vol_percent`, or `auto`); `display_unit` selects the shared plot scale. The weighted external reference can be included automatically with a configurable legend label.
additional_plots	No	mapping	`{meteo_check: {columns: [air_pressure, air_temperature], title: Meteo}}`	User-defined report plots for arbitrary existing DataFrame columns. Each plot needs at least one column and may set `title` and `y_label`.

See PDF output for the report layout, built-in figure ids and detailed report plotting options.

Detailed configuration reference¶

Below is more details on some of the features of sensor config file.

Sensor Information (`sensor_info`)¶

This section contains metadata for the CRNS station and site. The values are used in soil-moisture estimation and output organisation.

`name`¶

Description A unique identifier for the monitoring station that will be used in file naming and outputs.

Specification

Type: string
Required: Yes
Example: "Cunnesdorf_test_site"

Technical Details

Should be URL-safe (avoid special characters)
Used as default folder name for outputs
No spaces recommended (use underscores)

`identifier`¶

Description The unique hardware identifier for the CRNS unit. This can be an additional was to identify the site.

Specification

Type: string
Required: No
Example: "A102"

`install_date`¶

Description The date when the CRNS was installed at the monitoring site.

Specification

Type: Date string
Format: YYYY-MM-DD
Required: No
Default: null
Example: "2016-10-21"

Technical Details

When provided, it is used as the cutoff for data processing
When omitted or null, no installation-date cutoff is applied
Single-digit months/days require leading zeros
Must be in format YYYY-MM-DD to be registered as a date

`latitude` and `longitude`¶

Description Geographic coordinates of the CRNS installation location.

Specification

Type: float
Required: Yes
Range: -90 to 90 (latitude), -180 to 180 (longitude)
Example: 51.369597, 12.557120

Technical Details

Decimal degrees format

`elevation`¶

Description

Height above sea level of the CRNS installation site.

Specification

Type: float
Required: Yes
Units: meters above sea level
Example: 113

Technical Details

Used in atmospheric pressure corrections
Important for neutron flux calculations

`site_cutoff_rigidity`¶

Description The geomagnetic cutoff rigidity at the installation site, which affects cosmic ray flux.

Specification

Type: float
Required: No
Units: GV (gigavolts)
Example: 2.94

Technical Details

Affects incoming neutron corrections
Location-dependent parameter
www.crnslab.org provides methods to calculate this with latitude and longitude values
If not supplied in config it will use a lookup table to find the value with lat and lon

`avg_lattice_water`¶

Description The average lattice water content in soil minerals at the monitoring site.

Specification

Type: float
Required: No
Units: g/g (decimal percentage)
Example: 0.0043

Technical Details

Represented as decimal (0.0043 = 0.43%)
Site-specific constant
Used in soil moisture conversion
If not supplied defaults to 0
Can be automatically calculated if calibration sample data is available and lattice water content is a provided data.

`avg_soil_organic_carbon`¶

Description The average soil organic carbon content at the monitoring site.

Specification

Type: float
Required: No
Units: g/g (decimal percentage)
Example: 0.0184

Technical Details

Represented as decimal (0.0184 = 1.84%)
If not supplied defaults to 0
Can be automatically calculated if calibration sample data is provided with this in it (a site average is used)
Used in soil moisture conversion equations

`avg_dry_soil_bulk_density`¶

Description The average dry soil bulk density across the CRNS footprint. This value is used to convert gravimetric to volumetric soil moisture content.

Specification

Type: float
Required: No
Units: g/cm³
Example: 1.6

Technical Details

Important for use in converting neutrons to soil moisture (particularly converting gravimetric to volumetric soil moisture)
Influences effective measurement depth
Can be automatically calculated if calibration sample data is provided with this data in it (a site average is used)

`N0`¶

Description Site-specific calibration parameter used to convert corrected neutron counts to soil moisture.

Specification

Type: float
Required: No
Example: 1100

Technical Details

Determined through field calibration
Can be calibrated with soil sampling data if this option turned on
If no calibration data is availble, you will have to guess it. Although this will mean there is a bias problem in your data.

`beta_coefficient`¶

Description Site-specific coefficient used in the atmospheric pressure correction of neutron count rates.

Specification

Type: float
Required: No
Units: hPa⁻¹
Example: 0.0076

Technical Details

Used in pressure correction equations
Location and elevation dependent
Affects neutron count normalization
Will be automatically calculated in neptoon if not provided using supplied elevation and latitude data

`mean_pressure`¶

Description The long-term average atmospheric pressure at the monitoring site. Used as a reference pressure for neutron count corrections.

Specification

Type: float
Required: No
Units: hPa (hectopascals)
Example: 1013.25

Technical Details

Used for pressure corrections
Elevation dependent
When omitted, configured workflows estimate the reference from sensor_info.elevation using Neptoon's standard-atmosphere helper before falling back to downstream defaults.

`time_zone`¶

Description The time zone offset from UTC for the monitoring site. Essential for proper temporal alignment of data.

Specification

Type: string
Required: Yes
Format: ±H
Example: "+1" or "-10"

Raw Data Parse Options (`raw_data_parse_options`)¶

This section configures how neptoon reads raw CRNS data files. These settings handle differences between logger formats and sensor manufacturers.

`parse_raw_data`¶

Description Primary toggle that determines whether neptoon should process raw data files or expect pre-processed data.

Specification

Type: boolean
Required: Yes
Example: True

Technical Details

Controls entire raw data processing pipeline
Determines workflow path

`data_location`¶

Description Path or ordered list of paths to raw data files, directories, or compressed archives.

Specification

Type: string (path) or list[string]
Required: Yes (if parse_raw_data is True)
Example: "data/CRNS-station_data.zip", "../raw_data/", or:

data_location:
  - ../raw_data/2025-01
  - ../raw_data/2025-02
  - ../raw_data/2025-03

Technical Details

Supports absolute or relative paths
Handles zip and tar archives automatically
Multiple entries are searched in the configured order
Files inside each directory are sorted deterministically
Overlapping entries are deduplicated by their resolved file path
Directory recursion is controlled by raw_data_parse_options.recursive
Path resolution relative to config file
Supported archive formats: .zip, .tar

`recursive`¶

Description Controls whether local directories listed in data_location are searched below their immediate level.

Specification

Type: boolean
Required: No
Default: true
Example: false

Technical Details

true preserves the historical recursive-search behaviour
false considers only files directly inside each configured directory
Explicit file paths are always considered
This option is separate from raw_data_parse_options.sftp.recursive, which controls traversal on the remote server

Example for targeted monthly folders without descending into nested archive directories:

raw_data_parse_options:
  parse_raw_data: true
  data_location:
    - ./raw/2025-01
    - ./raw/2025-02
    - ./raw/2025-03
  recursive: false
  prefix: ENV
  suffix: .TXT

`sftp`¶

Description Optional SFTP source for raw data files. When sftp.enabled is true, downloaded files are used as the raw parser input instead of data_location.

Specification

Type: object
Required: No
Requires extra: neptoon[sftp]

Example:

raw_data_parse_options:
  parse_raw_data: true
  data_location: null
  sftp:
    enabled: true
    host: sftp.example.org
    port: 22
    username: crns-reader
    remote_path: /incoming/station_A101
    file_glob: "*.csv"
    recursive: true
    local_cache_dir: .neptoon_cache/sftp/station_A101
    password_env: NEPTOON_SFTP_PASSWORD
    host_key_policy: reject

Technical Details

host and remote_path are required when SFTP is enabled.
Authentication can use password_env, private_key_path, SSH agent auth, or system/user keys.
Avoid storing plaintext password values in shared YAML files.
file_glob filters remote files after traversal.
recursive: true traverses remote directories.
local_cache_dir keeps downloaded files in a predictable local folder; otherwise a temporary directory is used.
host_key_policy accepts reject, warning, or auto_add; use reject for production where host keys are known.
Server-provided file names are validated before download. Absolute paths, traversal components, path separators inside directory entries, control characters, drive-qualified names, and paths redirected outside local_cache_dir by local symlinks are rejected.

`archive_safety`¶

Description Resource limits applied before ZIP or TAR raw-data archives are extracted. The defaults protect against path-safe but resource-exhausting archives.

Specification

Type: object
Required: No
Defaults:

raw_data_parse_options:
  archive_safety:
    max_members: 10000
    max_total_uncompressed_bytes: 5368709120
    max_single_file_bytes: 1073741824
    max_compression_ratio: 200.0
    allow_nested_archives: false
    minimum_free_space_bytes: 104857600

Technical Details

All member paths and TAR link targets must remain inside the temporary extraction directory.
Device files, FIFOs, ZIP symlinks, duplicate destination paths, and other special entries are rejected.
Member count, largest file, total expanded size, and compression ratio are checked before extraction begins.
Extraction also requires enough free disk space for the declared expanded size plus minimum_free_space_bytes.
Files whose names indicate another archive are rejected unless allow_nested_archives is explicitly enabled. Enabling it permits storage only; Neptoon does not recursively extract nested archives.
Increase limits only for archives from a trusted source. Metadata checks substantially reduce archive-bomb risk but cannot make untrusted compressed input risk-free.

`column_names`¶

Description Explicit list of column names in the order they appear in the raw data files. Provides direct control over column identification and naming.

Specification

Type: list[string]
Required: No

Example:

column_names:
  - date_time
  - neutron_counts
  - pressure_hpa
  - temperature_c
  - humidity_percent

Technical Details

Overrides automatic header detection
Case-sensitive matching
Maintains column order

Inline metadata in column_names

Raw parser column names can use the same compact metadata syntax as key column lists:

raw_data_parse_options:
  column_names:
    - DateTime:datetime
    - rain_mm[0,inf]:float:sum
    - qc_flag[0,3]:int:first
    - N_cph[0,inf]:float:mean

This populates value ranges, dtype hints and aggregation hints automatically. Use :sum for interval totals such as precipitation, and :first, :last or :mode for flags and labels. See Columns in neptoon for details.

`prefix`¶

Description

String pattern used to filter raw data files by their filename prefix.

Specification

Type: string
Required: No
Example: "CRNS_"

Technical Details

Case-sensitive matching
Only applied on actual file name, i.e. not preceeding names of subdirectories, if any
Used in file selection phase

suffix

Description String pattern used to filter raw data files by their filename suffix. Specification

Type: string
Required: No
Example: ".dat" or ".txt"

Technical Details

Case-sensitive matching
Include the dot for file extensions
Applied after prefix filtering

filename_include_regex

Description Regular expression used to include raw files after the prefix and suffix filters have already been applied.

Specification

Type: string
Required: No
Example: "^2024/.+\\.TXT$"

Technical Details

Applied after prefix and suffix filtering
Evaluated against the file path relative to each data_location entry, not the absolute path
Relative paths use POSIX-style / separators on all platforms, including Windows
Empty strings and null disable the filter

filename_exclude_regex

Description Regular expression used to exclude raw files after prefix, suffix, and filename_include_regex filters have already been applied.

Specification

Type: string
Required: No
Example: "(^|/)archive/"

Technical Details

Applied after filename_include_regex filtering
Evaluated against the file path relative to each data_location entry, not the absolute path
Relative paths use POSIX-style / separators on all platforms, including Windows
Empty strings and null disable the filter

`skip_lines`¶

Description Number of lines to skip at the beginning of each data file.

Specification

Type: integer
Required: No
Default: 0
Example: 3

Technical Details

Affects all files in batch when data_start_regex is not configured
Prefer data_start_regex when files in one collection have different preamble lengths

`data_start_regex`¶

Description Regular expression that identifies the first real data row independently in every raw file. All preceding lines are ignored. This avoids relying on one shared header length when logger metadata changes between files.

Specification

Type: string or null
Required: No
Default: null
Example: "^\\s*\\d+\\s*,"

Technical Details

The matching line itself is included as the first data row
When set, this per-file boundary takes precedence over skip_lines during data parsing
The configuration wizard emits ^\s*\d+\s*, for Hydroinnova/Quaesta files
Invalid regular expressions are rejected during configuration validation

`encoding`¶

Description Specifies the character encoding used in the raw data files. Critical for correct text interpretation, especially with international characters.

Specification

Type: string
Required: No
Default: "utf-8"
Example: "cp850"

Technical Details

Common options:
- "utf-8": Universal encoding
- "cp850": Windows Western European
- "ascii": 7-bit ASCII
- "latin1": ISO-8859-1

`separator`¶

Description Character used to separate columns in the raw data files. Must be explicitly defined to ensure correct data parsing.

Specification

Type: string
Required: Yes
Example: ","

Technical Details - Common separators: - ",": CSV files - "\t": Tab-separated - ";": European CSV - "|": Pipe-separated - Must be in quotes

`decimal`¶

Description Character used as decimal separator in numeric values.

Specification

Type: string
Required: No
Default: "."
Example: ","

Technical Details

Must be in quotes

`skip_initial_space`¶

Description Controls whether leading whitespace in data fields should be removed during parsing.

Specification

Type: boolean
Required: No
Default: True
Example: True

`parser_kw`¶

Description AAdditional parser key words

Specification

Type: object
Required: No
Properties:
- strip_left: boolean - Remove leading whitespace
- digit_first: boolean - Expect numeric data at start

Technical Details

Specialized parsing behavior
Applied during data import

`starts_with`¶

Description String pattern that identifies header lines in the data files.

Specification

Type: string
Required: No
Default: ""
Example: "#"

Technical Details

Used in header detection
Case-sensitive matching

`multi_header`¶

Description Indicates whether data files contain multiple header lines that need special processing.

Specification

Type: boolean
Required: No
Default: False
Example: False

`strip_names`¶

Description Controls whether whitespace should be removed from column names during parsing.

Specification

Type: boolean
Required: No
Default: True
Example: True

Technical Details

Applied to column headers
Affects column name matching

`remove_prefix`¶

Description String pattern to be removed from the beginning of column names.

Specification

Type: string
Required: No
Example: "//"

Technical Details

Must be in quotes
Used for cleanup of raw headers

Time Series Data (`time_series_data`)¶

This section defines how data is formatted ready for use in neptoon. It presumes that the format has already been compiled into a .csv format.

`path_to_data`¶

Description The path to the .csv containing time series data

Specification

Type: string
Required: Yes (if no parsing done)
Example: /path/to/data.csv

Time Series Completion (`time_series_completion`)¶

This section controls whether neptoon creates a regular timestamp grid before external-data attachment, aggregation, and optional interpolation. It does not invent measurement values; inserted timestamps contain NaN unless later steps fill them.

time_series_completion:
  enabled: true
  target_frequency: auto
  create_missing_timestamps: true
  duplicate_timestamp_policy: first

Parameter	Type	Default	Description
enabled	boolean	`true`	Enable regular time-grid completion.
target_frequency	string	`auto`	Infer the dominant raw-data step or use a pandas frequency such as `15min` or `1h`.
create_missing_timestamps	boolean	`true`	Keep newly inserted rows as missing values.
duplicate_timestamp_policy	string	`first`	How to handle duplicate timestamps before reindexing: `first`, `last`, or `raise`.

Missing-Value Interpolation (`missing_value_interpolation`)¶

This optional step fills gaps only in explicitly listed numeric columns. It is separate from aggregation and should be used deliberately, especially for neutron data.

missing_value_interpolation:
  enabled: true
  columns:
    - air_temperature
    - air_relative_humidity
    - air_pressure
  method: time
  max_interpolation_hours: 12
  missing_column_policy: skip
  non_numeric_policy: skip

Parameter	Type	Default	Description
enabled	boolean	`false`	Enable interpolation for configured columns.
columns	list[string]	`[]`	Columns that may be interpolated. No other columns are touched.
method	string	`time`	Pandas interpolation method, currently `time` or `linear`.
max_interpolation_hours	float/null	`12`	Maximum consecutive gap duration to fill; longer gaps remain `NaN`. Set `null` to disable this limit.
missing_column_policy	string	`skip`	`skip` missing configured columns or `raise` an error.
non_numeric_policy	string	`skip`	`skip` non-numeric columns or `raise` an error.

Aggregation and interpolation are separate

If temporal aggregation marks a low-coverage absolute-count neutron bin as NaN, interpolation is the later, explicit decision point. The aggregation step does not silently interpolate or back-fill incomplete bins.

Temporal Configuration (`time_series_data.temporal`)¶

`input_resolution`¶

Description Specifies the time step of the input data, critical for proper temporal processing and aggregation.

Specification

Type: string
Required: Yes
Format: <number><unit>
Example: "15mins" or "1hour"

Technical Details

Valid units:
- Minutes: "min", "minute", "minutes"
- Hours: "hour", "hours", "hr", "hrs"
- Days: "day", "days"
Number must be positive integer

`output_resolution`¶

Description Desired time step for processed data output. Determines the temporal resolution of final results.

Specification

Type: string
Required: Yes
Format: <number><unit> or "None"
Example: "1hour"

Technical Details

Must be greater than or equal to input_resolution
Use "None" to maintain input resolution
When different from input aggregation will occur

`align_timestamps`¶

Description Controls whether timestamps should be aligned to regular intervals.

Specification

Type: boolean
Required: Yes
Example: true

Technical Details

Ensures consistent temporal spacing
Affects data aggregation methods
If aggregation occurs this is ignored (already aligned)

`alignment_method`¶

Description Specifies how timestamps should be aligned when processing data.

Specification

Type: string
Required: Yes
Example: "time"

Technical Details

Alignment places values on a regular grid using pandas interpolation.
"time": interpolate linearly in time onto the grid. This is the recommended default.
"linear": interpolate assuming equally spaced samples.
The former SaQC snap methods "nshift", "fshift" and "bshift" are no longer supported and raise a configuration error. Use "time" or "linear" instead.

Key Column Configuration¶

`epithermal_neutron_columns`¶

Description Specifies which columns contain epithermal neutron count data, the primary measurement for soil moisture estimation.

Specification

Type: list[string]
Required: Yes

Example:

epithermal_neutron_columns:
  - N2Cts
  - ModNeutrons

Technical Details

Must match column names exactly

`thermal_neutron_columns`¶

Description

Identifies columns containing thermal neutron count data, used for advanced corrections and quality control.

Specification

Type: list[string]
Required: No
Example:
```
thermal_neutron_columns:
  - N1Cts
```

Technical Details

Optional, recommended if available

`neutron_count_units`¶

Description Specifies the units of the neutron count measurements.

Specification

Type: string
Required: Yes
Options:
- "absolute_count"
- "counts_per_hour"
- "counts_per_second"
Example: "absolute_count"

Technical Details

Affects count rate calculations
Critical for cross-site comparisons
Must match sensor configuration
Internally counts are converted into absolute counts (raw) and counts_per_hour (corrected)

`pressure_columns`¶

Description List of local columns containing atmospheric pressure measurements, in order of priority. Set this to null when no local pressure column is configured and pressure is supplied later by DWD, external meteo, fallback values, or an inactive pressure correction.

Specification

Type: list[string] or null
Required: No

Example:

pressure_columns:
  - P4_mb  # Primary sensor
  - P3_mb  # Backup sensor
  - P1_mb  # Tertiary sensor

pressure_columns: null  # no local pressure column configured

Technical Details

Order determines priority. Use priority to select only the first configured column, or priority_fill to fill missing values in the first column from later fallback columns row by row.
If configured, all local pressure columns must use compatible units unless pressure_column_units specifies per-column units.
If omitted or set to null, no local pressure column is imported; a later processing step must provide pressure if it is required.

`pressure_units`¶

Description Units of the pressure measurements in the specified columns.

Specification

Type: string
Required: Only when local pressure_columns are configured
Options: aliases for Pa, hPa/mbar, or kPa; values are normalized internally to pascals, hectopascals, or kilopascals
Example: "hectopascals"

Technical Details

Required only when local pressure_columns are configured.
Must be consistent across all local pressure columns unless pressure_column_units specifies per-column units.
Neptoon converts configured Pa and kPa inputs to its canonical hPa working unit

`pressure_merge_method`¶

Description Method used to combine multiple pressure measurements when available.

Specification

Type: string
Required: Yes
Options: "priority", "mean"
Example: "priority"

Technical Details

"priority": Uses only the highest-priority configured column
"mean": Averages all available values
Handles missing data automatically

`temperature_columns`¶

Description List of local columns containing air temperature measurements, in order of priority. Set this to null when no local temperature column is configured and temperature is supplied later by DWD, external meteo, fallback values, or an inactive humidity correction.

Specification

Type: list[string] or null
Required: No

Example:

temperature_columns:
  - T1_C
  - T2_C

temperature_columns: null  # no local temperature column configured

Technical Details

Order determines priority.
If configured, all local temperature columns must use compatible units.
If omitted or set to null, no local temperature column is imported; a later processing step must provide temperature if it is required for absolute humidity.

`temperature_units`¶

Description Units of the temperature measurements.

Specification

Type: string
Required: Only when local temperature_columns are configured
Options: Celsius aliases (including legacy celcius), kelvin, or fahrenheit
Example: "celcius"

Technical Details

Required only when local temperature_columns are configured.
Must be consistent across all local temperature columns
Kelvin and Fahrenheit inputs are converted to Celsius before meteorological merging

`temperature_merge_method`¶

Description Method used to combine multiple temperature measurements when available.

Specification

Type: string
Required: Yes
Options: "priority", "mean"
Example: "priority"

Technical Details

Follows same logic as pressure_merge_method

`relative_humidity_columns`¶

Description List of local columns containing relative humidity measurements. Set this to null when no local relative humidity column is configured and relative humidity is supplied later by DWD, external meteo, fallback values, or an inactive humidity correction.

Specification

Type: list[string] or null
Required: No

Example:

relative_humidity_columns:
  - RH1

relative_humidity_columns: null  # no local relative humidity column configured

Technical Details

If configured, local relative humidity columns must use compatible units.
If omitted or set to null, no local relative humidity column is imported; a later processing step must provide relative humidity if it is required for absolute humidity.

`relative_humidity_units`¶

Description Units of the relative humidity measurements.

Specification

Type: string
Required: Only when local relative_humidity_columns are configured
Options: "percent", "percentage", "fraction"
Example: "percent"

Technical Details

Required only when local relative_humidity_columns are configured.
Percent is the canonical local relative humidity unit; fraction inputs are converted to percent before meteorological merging.

`date_time_columns`¶

Description Columns containing temporal information for measurements.

Specification

Type: list[string]
Required: Yes
Example: ["Date Time(UTC)"]

Technical Details

Must contain valid datetime information
Used for all temporal alignment
Critical for data processing
Multiple columns can be merged e.g., ['Date', 'Time']

`date_time_format`¶

Description Format string specifying how datetime information is encoded.

Specification

Type: string
Required: Yes
Format: Python datetime format string
Example: "%Y/%m/%d %H:%M:%S"

Technical Details

Must be in quotes
Follows Python strftime format

`initial_time_zone`¶

Description Timezone of data. Most CRNS data is given in UTC, but if it's not we can handle that here.

Specification

Type: string
Required: Yes
Example: utc or Europe/Berlin

`convert_time_zone_to`¶

Description Timezone to convert data to STRONG recommendation to leave this as utc.

Specification

Type: string
Required: Yes
Example: utc

Technical Details

Epoch timestamps, parse failures, and daylight-saving transitions¶

For Unix epoch input, configure exactly one date_time_columns entry and enable is_timestamp. date_time_format is then ignored.

time_series_data:
  key_column_info:
    date_time_columns: [timestamp_ms]
    is_timestamp: true
    timestamp_unit: ms
    initial_time_zone: utc
    convert_time_zone_to: utc

Text timestamp parsing can be made stricter or quieter with the following fields:

Parameter	Default	Description
`datetime_parse_error_policy`	`warn`	`warn`, `raise`, or `ignore` invalid timestamps after parsing/localisation.
`datetime_parse_error_fraction`	`0.05`	Fraction of invalid timestamps above which warnings or errors escalate.
`datetime_parse_error_report_limit`	`5`	Maximum invalid examples included in diagnostics.
`timezone_ambiguous`	`NaT`	Handling of duplicated local times during DST fall-back: `raise`, `infer`, or `NaT`.
`timezone_nonexistent`	`shift_forward`	Handling of skipped local times during DST spring-forward: `raise`, `NaT`, `shift_forward`, or `shift_backward`.

Prefer logger data in UTC whenever possible. These controls are mainly for legacy archives recorded in local civil time.

Quality Assessment Configuration¶

The Quality Assessment (QA) system validates meteorological, neutron, soil-moisture, and explicitly named custom columns. Config-driven checks include range, persistence, UniLOF, rolling z-score, offset-spike, and N0-relative checks; availability depends on the selected target and processing stage.

Supported Variables¶

Quality assessment can be performed on the following meteorological variables:

air_relative_humidity
air_pressure
air_temperature

For QA on soil moisture data the style is like:

soil_moisture_qa:
  soil_moisture:
    flag_range:
      min: 0
      max: 1

Assessment Methods¶

1. Range Check (`flag_range`)¶

The range check method flags values that fall outside specified minimum and maximum thresholds.

Required Parameters¶

min: Minimum acceptable value (in data units)
max: Maximum acceptable value (in data units)

Example Configuration¶

input_data_qa:
  air_pressure:
    flag_range:
      min: 850  # hPa
      max: 1050 # hPa

  air_relative_humidity:
    flag_range:
      min: 0    # %
      max: 100  # %

  air_temperature:
    flag_range:
      min: -30  # °C
      max: 50   # °C

2. Univariate Local Outlier Factor (`spike_uni_lof`)¶

This method uses the Local Outlier Factor algorithm to detect anomalies in univariate time series data. More information on this here

Optional Parameters¶

periods_in_calculation: Number of time steps included in LOF calculation
- Default: 20
- Units: time steps
threshold: Threshold for flagging outliers
- Default: 1.5
- Units: decimal
algorithm: Algorithm for calculating nearest neighbors
- Default: "ball_tree"
- Options: ["ball_tree", "kd_tree", "brute", "auto"]

Example Configuration¶

input_data_qa:
  air_temperature:
    spike_uni_lof:
      periods_in_calculation: 24  # Use 24 time steps
      threshold: 2.0             # More permissive threshold
      algorithm: "ball_tree"     # Default algorithm

SaQC parameter names are also accepted. The equivalent configuration is:

input_data_qa:
  air_temperature:
    spike_uni_lof:
      n: 24
      thresh: 2.0
      algorithm: "ball_tree"

Neptoon and SaQC parameter names may be mixed, but the two names for the same parameter must not both be supplied. Neptoon normalizes SaQC aliases back to its descriptive internal names before constructing the runtime quality check.

3. Z-score spike check (`spike_zscore`)¶

This method flags observations whose rolling z-score exceeds a configured threshold. It is useful when a transparent statistical threshold is preferred over the neighbourhood-based spike_uni_lof check.

input_data_qa:
  air_pressure:
    spike_zscore:
      periods_in_calculation: 24
      threshold: 4.0
      centered: false

The exact accepted fields and defaults are also shown in the SpikeZScore configuration API.

4. Relative offset spike check (`spike_offset`)¶

spike_offset detects a sudden relative jump followed by a return toward the previous level. Supply the positive and negative limits in that order and a pandas-compatible duration for the comparison window.

Every configured QA method also accepts the common policy fields severity (error or warning) and action (keep, mask, or raise). Omitting both preserves the historical error + mask behavior; setting severity: warning without an action defaults to keep.

input_data_qa:
  custom_columns:
    battery_voltage:
      spike_offset:
        threshold_relative: [0.2, -0.2]
        window: 12h
        severity: warning
        action: keep

5. Persistence check (`persistence_check`)¶

The persistence check flags repeated or nearly repeated values inside a rolling window. This is useful for stuck humidity, pressure, temperature, or neutron channels. The legacy misspelling persistance_check is still accepted for backwards compatibility, but new configs should use persistence_check.

Required Parameters¶

window: Rolling window length in samples

Optional Parameters¶

threshold: Maximum total change inside the window before values are treated as persistent; defaults to 0
min_periods: Minimum valid observations required inside the window

Example Configuration¶

input_data_qa:
  air_relative_humidity:
    persistence_check:
      window: 12
      min_periods: 6
      threshold: 0

Complete Example¶

Here's a complete example showing how to combine several methods:

input_data_qa:
  air_pressure:
    flag_range:
      min: 850
      max: 1050
    spike_uni_lof:
      periods_in_calculation: 12
      threshold: 1.8

  air_relative_humidity:
    flag_range:
      min: 0
      max: 100
    spike_uni_lof:
      periods_in_calculation: 6
      threshold: 1.3
    persistence_check:
      window: 12
      min_periods: 6
      threshold: 0

  air_temperature:
    flag_range:
      min: -30
      max: 50

Best Practices¶

Range Selection
- Choose ranges based on physically possible values for your location
- Consider seasonal variations when setting thresholds
LOF Parameters
- periods_in_calculation: Choose based on your data's temporal resolution
- Hourly data: 24 periods = 1 day window
- 15-min data: 96 periods = 1 day window
- threshold: Start conservative (1.5) and adjust based on results
- algorithm: Use default unless you have specific performance requirements

Notes¶

QA configuration is optional but recommended
Methods can be applied individually or in combination
Configuration is applied during data processing via the CRNSDataHub
Flagged data will be excluded from subsequent processing steps

Calibration (`calibration`)¶

`calibrate`¶

Description Toggle for whether calibration will be done.

Specification

Type: boolean
Required: Yes
Example: True

`data_format`¶

Description Legacy calibration-data format selector retained for backwards compatibility. New configurations can usually omit this field and rely on explicit calibration column names.

Specification

Type: string
Required: No
Example: custom

`location`¶

Description Location of the calibration data.

Specification

Type: string
Required: No
Example: home_dir/example_data/FSCD001_calibration.csv

`date_time_format`¶

Description DateTime format of the calibration data.

Specification

Type: string
Required: No
Example: "%d.%m.%Y %H:%M"

Weighting and calibration-window controls¶

horizontal_weighting_method and vertical_weighting_method select the sample-weighting theories used by the calibration routine. The current default for both is schroen_etal_2017.

The neutron averaging window is controlled independently from normal output smoothing:

hours_of_data_around_calib sets the total span around a calibration time;
calib_window_alignment chooses centered or trailing placement (centred is accepted as an input alias);
match_smoothing_window: true opts into using the corrected-neutron smoothing window and trailing alignment;
metadata_reference points to a prior calibration metadata JSON sidecar for compatibility warnings, not as an N0 source;
biomass_correction_acknowledged waives the double-count guard, see below;
calibration_comparison_info_threshold controls when relative calibration differences are surfaced at INFO level. Values are always retained in the metadata output.

Reusing an `N0` with an active biomass correction¶

With calibrate: false and an above-ground-biomass correction fed from remote sensing, neptoon stops the run. An N0 fitted without that correction already contains the vegetation, so correcting again subtracts it twice: measured at one forest station, mean volumetric soil moisture fell from 0.205 to 0.051 m³/m³, which is an artefact rather than a measurement.

There are three ways forward, and the first two need no declaration:

set calibrate: true and point location at sample data, so N0 is refitted with the correction active;
reuse an N0 from a neptoon run whose calibration metadata records an active biomass correction, by linking it with metadata_reference. This is verified rather than promised, which is why it needs nothing else;
set biomass_correction_acknowledged: true if you know your N0 was itself determined with a biomass correction in force, for example a published value from a study that applied one.

The third path is deliberately loud. The run continues, and it repeats what was assumed in the log, in the processing output and in the PDF report, every time, so that a reader who never saw the configuration file still learns that the number rests on your declaration.

Key Column Names (`calibration.key_column_names`)¶

`date_time`¶

Description Name of the column with DateTime.

Specification

Type: string
Required: No
Example: "DateTime_utc"

`profile_id`¶

Description Name of the column with profile ID.

Specification

Type: string
Required: No
Example: "Profile_ID"

`sample_depth`¶

Description Name of the column with sample depth values.

Specification

Type: string
Required: No
Example: "Profile_Depth_cm"

`radial_distance_from_sensor`¶

Description Name of the column with distance of the sample from the sensor (m).

Specification

Type: string
Required: No
Example: "Distance_to_CRNS_m"

`bulk_density_of_sample`¶

Description Name of the column with bulk density of the samples.

Specification

Type: string
Required: No
Example: "DryBulkDensity_g_cm3"

`gravimetric_soil_moisture`¶

Description Name of the column with gravimetric soil moisture values.

Specification

Type: string
Required: No
Example: "SoilMoisture_g_g"

`soil_organic_carbon`¶

Description Name of the column with soil organic carbon values.

Specification

Type: string
Required: No
Example: "SoilOrganicCarbon_g_g"

`lattice_water`¶

Description Name of the column with lattice water values.

Specification

Type: string
Required: No
Example: "LatticeWater_g_g"

Data Storage Options (`data_storage`)¶

`save_location`¶

Description Directory for saving outputs (data, flags, figures, report).

Specification

Type: string
Required: No
Example: ./outputs, ~/neptoon-outputs/station_A101 or /data/outputs

Technical Details

Accepts absolute paths (/home/user/outputs), home-relative paths (~/outputs), and relative paths (./outputs or ../outputs). The ~ is expanded to the user's home directory.
Relative paths written in YAML files are resolved relative to the directory containing that YAML config file. This keeps config files portable when the config and output folders are moved together.
A CLI override with --save-location takes precedence over the YAML value. Relative CLI override paths are resolved relative to the current working directory.
Missing parent directories are created automatically.
If left blank, Neptoon checks NEPTOON_OUTPUT_DIR first and otherwise uses the current working directory only after verifying that it is writable. Persistent outputs never silently fall back to a temporary directory.
Used by save_data() as the default output directory for both the ProcessWithConfig and DataHubFromConfig (step-by-step / notebook) workflows. Passing save_folder_location to save_data() explicitly still takes precedence.

`append_audit_log_hash_to_folder_name`¶

Description Append a short audit-log hash to the output folder name. This can help distinguish runs that use the same sensor name and timestamp settings but produce different processing metadata.

Specification

Type: boolean
Required: No
Example: False

Technical Details

Defaults to False.
Use together with append_timestamp_to_folder_name for human-readable and traceable output folders.
append_yaml_to_folder_name is a legacy compatibility field; prefer append_audit_log_hash_to_folder_name for new configs.

`create_report`¶

Description Whether to create the PDF processing report during the processing run. When enabled, Neptoon writes Report-<sensor-name>.pdf into the output folder together with the processed data, flags, figures and audit information. The report is intended for quality control and communication; use the CSV/JSON tables for reproducible downstream analysis.

Specification

Type: boolean
Required: No
Default: False
Example: True

Technical Details

Report content depends on which processing stages were run and which columns exist. Missing figure inputs cause the relevant figure to be skipped rather than inventing data.
Calibration pages appear only when calibration was run.
External-source/weather pages are populated when DWD or external_meteo columns/provenance exist.
Detailed report plotting options are described in PDF output.

---¶

Figures Options (`figures`)¶

`create_figures`¶

Description Generate visualization figures and automatically store them when saved.

Specification

Type: boolean
Required: Yes
Example: True

Technical Details

Figures are saved in the folder when saved

`make_all_figures`¶

Description Generate all available figure types in figure registry.

Specification

Type: boolean
Required: Yes
Example: True

`custom_list`¶

Description List of specific figures to generate if not doing all

Specification

Type: list
Required: No
Example: [nmdb_incoming_radiation]

Technical Details

If make_all_figures is true this is ignored.

Merge method semantics¶

priority selects the first/highest-priority configured column and does not gap-fill from later columns. Use priority_fill when the primary column should be filled row-wise from lower-priority fallback columns. average and mean compute a row-wise mean across configured columns.

Horizontal footprint model¶

horizontal_footprint selects the horizontal CRNS footprint model used consistently by calibration, dynamic footprint-radius products, and distance-aware external soil-moisture weighting. Existing configurations default to schroen_etal_2017.

horizontal_footprint:
  model: schroen_etal_2017_2d
  map:
    location: footprint_soil_moisture.csv
    x_column: x_m
    y_column: y_m
    soil_moisture_column: soil_moisture
    detector_x_m: 0.0
    detector_y_m: 0.0
    radial_bin_width_m: 1.0
    max_radius_m: 300.0
    angular_samples: 720

The 2-D model requires a regular sensor-relative soil-moisture map that completely covers the configured calculation circle. Its native result is a normalised 2-D kernel; existing distance-only consumers use the kernel's radial projection. Optional porosity_column and soil_type_column values are retained as map metadata but do not alter horizontal weights. See Horizontal footprint models for the scientific boundary, map format, 1-D projection, and Python API.

Calibration uncertainty and footprint settings¶

The following optional fields make the scientific assumptions and uncertainty inputs explicit:

Field	Meaning
`N0_uncertainty`	Systematic one-standard-deviation uncertainty of the active calibration parameter. For Köhli 2025 this is the uncertainty of the detector-specific `I_D` used directly in `I_N = (I / I_D) C_h`. The legacy field name is retained for configuration compatibility.
`avg_dry_soil_bulk_density_uncertainty`	One-standard-deviation uncertainty of the site-average dry bulk density.
`avg_lattice_water_uncertainty`	One-standard-deviation uncertainty of the site-average lattice-water mass fraction.
`avg_soil_organic_carbon_uncertainty`	One-standard-deviation uncertainty of the site-average organic-carbon mass fraction.
`systematic_uncertainty`	Optional positive systematic uncertainty values. Only defined values greater than zero are written, propagated and plotted.
`calibration.converge_accuracy`	Relative convergence threshold for iterative footprint weighting.
`calibration.max_iterations`	Hard iteration limit; calibration fails rather than silently returning a non-converged result.
`calibration.vegetation_height_m`	Representative vegetation height used by the Schrön distance rescaling.
`calibration.footprint_assumption`	Spatial-support assumption recorded in calibration metadata. Configured workflows set it from `horizontal_footprint` (`homogeneous_radial_schroen_2017` or `heterogeneous_2d_schroen_2017`).

Systematic uncertainty values¶

Calibration uncertainty is systematic for a processed time series because the same calibrated N0 or I_D is reused for every timestamp. Neutron counting statistics remain a separate statistical uncertainty. The YAML stores only the positive values that are actually available:

sensor_info:
  N0: 1800.0  # I_D for a Köhli-2025 profile, despite the legacy field name
  systematic_uncertainty:
    calibration_parameter:
      value: 35.0
      unit: counts_per_hour
    soil_moisture:
      value: 0.012
      unit: volumetric_soil_moisture
    dry_soil_bulk_density:
      value: 0.04
      unit: g_per_cm3
    correction_factors:
      value: 0.02
      unit: relative_fraction

Each component may also be written as a scalar. Scalar calibration_parameter values default to counts_per_hour; scalar soil_moisture values default to volumetric_soil_moisture (m3/m3). The remaining parameter components default to their native parameter units, and correction_factors defaults to a relative fraction, so 0.02 means 2 %. A direct soil_moisture value is combined in quadrature with propagated calibration/input contributions. Missing or zero values are omitted from YAML, text output and plots. During calibration, only positive uncertainties that can actually be estimated are written to the sensor configuration and calibration metadata.

The nested uncertainty objects intentionally accept and ignore unknown metadata keys for compatibility with older calibration records and external metadata producers. This is a narrow exception to the otherwise strict configuration policy: ignored keys do not affect calculations and are not written back when the configuration is serialized.

Standard calibration-function set¶

One calibration campaign is also evaluated independently with the standard comparison functions Desilets 2010, UTS 2021 Mar21_mcnp_drf, Köhli 2025 drf_endf7, and Köhli 2025 normresponse_endf8. Each function receives its own footprint-weighting iterations and parameter fit. The selected function is stored as the active profile; the complete four-profile result set is inserted as YAML comments below soil_moisture_calibrations.entries. Commented profiles do not affect configuration loading or automatic profile selection.

Sensor configuration¶

Overview¶

Top-level behaviour controls¶

File Structure¶

Configuration quick reference guide¶

Sensor Information¶

Raw Data Parse Options¶

parallel_file_parsing¶

file_parse_workers¶

invalid_data_policy¶

Schema diagnostics and parser strategy¶

SFTP raw-data source¶

Time Series Data¶

Time Series Completion and Missing-Value Interpolation¶

Key Column Information (Time Series Data)¶

Column metadata fields¶

DWD and external meteorological data¶

Quality Assessment Settings¶

Calibration¶

Key Column Names (Calibration)¶

External soil-moisture files¶

Data Storage¶

Figures¶

Detailed configuration reference¶

Sensor Information (sensor_info)¶

name¶

identifier¶

install_date¶

latitude and longitude¶

elevation¶

site_cutoff_rigidity¶

avg_lattice_water¶

avg_soil_organic_carbon¶

avg_dry_soil_bulk_density¶

N0¶

beta_coefficient¶

mean_pressure¶

time_zone¶

Raw Data Parse Options (raw_data_parse_options)¶

parse_raw_data¶

data_location¶

recursive¶

sftp¶

archive_safety¶

column_names¶

prefix¶

skip_lines¶

data_start_regex¶

encoding¶

separator¶

decimal¶

skip_initial_space¶

parser_kw¶

starts_with¶

multi_header¶

strip_names¶

remove_prefix¶

Time Series Data (time_series_data)¶

path_to_data¶

Time Series Completion (time_series_completion)¶

Missing-Value Interpolation (missing_value_interpolation)¶

Temporal Configuration (time_series_data.temporal)¶

input_resolution¶

output_resolution¶

align_timestamps¶

alignment_method¶

Key Column Configuration¶

epithermal_neutron_columns¶

thermal_neutron_columns¶

neutron_count_units¶

pressure_columns¶

pressure_units¶

pressure_merge_method¶

temperature_columns¶

temperature_units¶

temperature_merge_method¶

relative_humidity_columns¶

relative_humidity_units¶

date_time_columns¶

date_time_format¶

`parallel_file_parsing`¶

`file_parse_workers`¶

`invalid_data_policy`¶

Sensor Information (`sensor_info`)¶

`name`¶

`identifier`¶

`install_date`¶

`latitude` and `longitude`¶

`elevation`¶

`site_cutoff_rigidity`¶

`avg_lattice_water`¶

`avg_soil_organic_carbon`¶

`avg_dry_soil_bulk_density`¶

`N0`¶

`beta_coefficient`¶

`mean_pressure`¶

`time_zone`¶

Raw Data Parse Options (`raw_data_parse_options`)¶

`parse_raw_data`¶

`data_location`¶

`recursive`¶

`sftp`¶

`archive_safety`¶

`column_names`¶

`prefix`¶

`skip_lines`¶

`data_start_regex`¶

`encoding`¶

`separator`¶

`decimal`¶

`skip_initial_space`¶

`parser_kw`¶

`starts_with`¶

`multi_header`¶

`strip_names`¶

`remove_prefix`¶

Time Series Data (`time_series_data`)¶

`path_to_data`¶

Time Series Completion (`time_series_completion`)¶

Missing-Value Interpolation (`missing_value_interpolation`)¶

Temporal Configuration (`time_series_data.temporal`)¶

`input_resolution`¶

`output_resolution`¶

`align_timestamps`¶

`alignment_method`¶

`epithermal_neutron_columns`¶

`thermal_neutron_columns`¶

`neutron_count_units`¶

`pressure_columns`¶

`pressure_units`¶

`pressure_merge_method`¶

`temperature_columns`¶

`temperature_units`¶

`temperature_merge_method`¶

`relative_humidity_columns`¶

`relative_humidity_units`¶

`date_time_columns`¶

`date_time_format`¶

`initial_time_zone`¶

`convert_time_zone_to`¶

1. Range Check (`flag_range`)¶

2. Univariate Local Outlier Factor (`spike_uni_lof`)¶

3. Z-score spike check (`spike_zscore`)¶

4. Relative offset spike check (`spike_offset`)¶

5. Persistence check (`persistence_check`)¶

Calibration (`calibration`)¶

`calibrate`¶

`data_format`¶

`location`¶

`date_time_format`¶

Reusing an `N0` with an active biomass correction¶

Key Column Names (`calibration.key_column_names`)¶

`date_time`¶

`profile_id`¶

`sample_depth`¶

`radial_distance_from_sensor`¶

`bulk_density_of_sample`¶

`gravimetric_soil_moisture`¶

`soil_organic_carbon`¶

`lattice_water`¶