Using configuration files¶

Processing from the command line¶

You can process data from the command line using your config files. First, make sure Neptoon is installed in an environment that exposes the neptoon command. pipx is one supported option. This means you will have access to the CLI commands in your terminal. See the installation guide for more on installation steps.

To run:

neptoon -p /path/to/process.yaml -s /path/to/sensor.yaml

You can also check the installed neptoon version, which is useful for bug reports and reproducible processing notes:

neptoon --version

Dry runs and diagnostics¶

Use --dry-run to load and validate the supplied configuration files without reading raw data and without writing outputs. This is useful before long processing runs or when checking paths in Docker and CI environments.

neptoon -p process.yaml -s sensor.yaml --dry-run

The dry run prints the temporal aggregation summary by default. Add --explain-output-columns when you also want a config-driven explanation of expected output and diagnostic columns:

neptoon -p process.yaml -s sensor.yaml --dry-run --explain-output-columns

For aggregation-specific debugging, use the dedicated command:

neptoon inspect-aggregation -p process.yaml -s sensor.yaml

This shows whether temporal aggregation is enabled, which output resolution is configured, which neutron-count units are used, which elapsed-time columns are available, and whether absolute_count data will be normalised as events divided by measured live time.

To print the same aggregation diagnostics before a normal processing run, use:

neptoon -p process.yaml -s sensor.yaml --show-aggregation

To inspect expected output columns without running the full workflow, use:

neptoon explain-output-columns -p process.yaml -s sensor.yaml

The same explanation can be printed before a normal processing run with:

neptoon -p process.yaml -s sensor.yaml --explain-output-columns

Estimate Kalman filter parameters from processed output¶

After a normal Neptoon run has created a processed output table, use estimate-kalman-params to estimate event-based Kalman soil-moisture filter parameters. The tool is standalone: it reads processed CSV/JSON/Parquet output, writes a YAML fragment and diagnostic tables, and does not modify your main process config automatically.

neptoon estimate-kalman-params \
  --input outputs/processed_data.csv \
  --use-precipitation \
  --soil-probe-column <your_probe_column> \
  --use-soil-probe \
  --model single_store \
  --output-yaml kalman_params.yaml

For normal Neptoon output CSV files the processed soil-moisture, uncertainty, precipitation, and aggregation diagnostic columns are inferred from common Neptoon names. Use --use-precipitation to tell the tool to use a precipitation column without typing its name. Near-surface TDR/TDT/FDR probe columns are not inferred automatically because site naming is too variable; pass them explicitly with --soil-probe-column and optionally --use-soil-probe. Explicit --*-column options always override auto-detection. Use --measurement-noise-mode poisson with *_counts_sum and *_elapsed_time_sum columns to derive observation noise from neutron counting statistics. The generated process noise is a dynamic-Q block with wetting-event, drydown and unexplained-spike multipliers. See Kalman parameter estimation for details.

Output path resolution¶

data_storage.save_location in a YAML config may be absolute, home-relative, or relative. Relative paths in YAML files are resolved relative to the directory containing the config file. This makes a config and its output folder portable when they are moved together.

A command-line override with --save-location takes precedence over the YAML value. Relative paths supplied through the CLI are resolved relative to the current working directory because the shell command, not the YAML file, provides the path context.

neptoon -p process.yaml -s sensor.yaml --save-location ./outputs/test-run

Complete script¶

You can use this code block as a starting point for your own, make sure to change the paths to your configs. It's usually better to use absolute paths if you know where your data is.

Using path stringsUsing pathlib.Path

from pathlib import Path
from neptoon.workflow.process_with_config import (
    ProcessWithConfig,
)
from neptoon.config import ConfigurationManager

# Instantiate a ConfigurationManager to handle config files
config = ConfigurationManager()

# Create paths to, and load, the configuration files.
sensor_config_path = Path("home/path/to/configuration_files/A101_station.yaml")
processing_config_path = Path("home/path/to/configuration_files/v1_processing_method.yaml")

# Load the configs
config.load_configuration(
    file_path=sensor_config_path,
)
config.load_configuration(
    file_path=processing_config_path,
)

# Add config manager to the ProcessWithConfig class ready for processing.
config_processor = ProcessWithConfig(configuration_object=config)

# Run
config_processor.run_full_process()

from pathlib import Path
from neptoon.workflow.process_with_config import (
    ProcessWithConfig,
)
from neptoon.config import ConfigurationManager

# Instantiate a ConfigurationManager to handle config files
config = ConfigurationManager()

# Create paths to, and load, the configuration files.
sensor_config_path = Path.cwd().parent / "configuration_files" / "A101_station.yaml"
processing_config_path = (
    Path.cwd().parent / "configuration_files" / "v1_processing_method.yaml"
)

# Load the configs
config.load_configuration(
    file_path=sensor_config_path,
)
config.load_configuration(
    file_path=processing_config_path,
)

# Add config manager to the ProcessWithConfig class ready for processing.
config_processor = ProcessWithConfig(configuration_object=config)

# Run
config_processor.run_full_process()