Quality Assessment in neptoon

Overview

The quality assessment (QA) system in neptoon identifies and flags suspect values in CRNS time series. It uses the SaQC framework and provides configuration wrappers for common CRNS checks.

For YAML-driven workflows, see the sensor QA configuration and the process-level neutron QA settings. This page focuses on the programmatic API.

Key Components

The quality assessment system in neptoon is built from several interconnected parts:

• QualityCheck: This is the fundamental building block that defines what you want to check (like neutron counts), how you want to check it (like range validation), and the specific thresholds or criteria to use.
• QualityAssessmentFlagBuilder: Collects multiple quality checks into a set
• DataQualityAssessor: Applies quality checks to data and manages the SaQC integration
• QAMethod: This is a set of pre-defined checking methods you can choose from, such as range checks (is the value between X and Y?), spike detection, or constant value detection.
• QATarget: This defines which data columns in your CRNS dataset you want to check, such as raw neutron counts, air pressure, or humidity measurements.

Getting Started

Creating Quality Checks

Quality checks are created using the QualityCheck class. A check defines what to inspect, how to inspect it, and what Neptoon should do when the check flags a value:

1. Target: what data column to check (using QATarget)
2. Method: what type of check to perform (using QAMethod)
3. Parameters: configuration parameters specific to the method
4. Severity: error (default) or warning
5. Action: mask, keep, or raise

For backwards compatibility, checks without an explicit policy keep the historical behavior: severity=error and action=mask. A warning defaults to action=keep, so unusual but plausible observations remain in the processed data while still being recorded as QA events.

from neptoon.quality_control import QualityCheck, QAMethod, QATarget

# Create a range check for neutron counts
neutron_range_check = QualityCheck(
    target=QATarget.RAW_EPI_NEUTRONS,
    method=QAMethod.RANGE_CHECK,
    parameters={
        "min": 500,  # Minimum acceptable value
        "max": 2000  # Maximum acceptable value
    }
)

# Create an offset spike check for air pressure
pressure_spike_check = QualityCheck(
    target=QATarget.AIR_PRESSURE,
    method=QAMethod.SPIKE_OFFSET,
    parameters={
        "window": "24h",
        "threshold_relative": (0.2, -0.2),
    },
)

# Create a persistence check for repeated or stuck values
humidity_persistence_check = QualityCheck(
    target=QATarget.RELATIVE_HUMIDITY,
    method=QAMethod.PERSISTENCE_CHECK,
    parameters={
        "window": 12,
        "min_periods": 6,
        "threshold": 0,
    },
)

Building a Set of Quality Checks

Multiple quality checks can be combined using the QualityAssessmentFlagBuilder:

from neptoon.quality_control import QualityAssessmentFlagBuilder

# Create a flag builder
flag_builder = QualityAssessmentFlagBuilder()

# Add multiple checks
flag_builder.add_check(neutron_range_check)
flag_builder.add_check(pressure_spike_check)

# Alternatively, add multiple checks at once
flag_builder.add_check(neutron_range_check, pressure_spike_check)

Applying Quality Checks to Data

Once you have defined your quality checks, you can apply them to your data using the DataQualityAssessor:

from neptoon.quality_control import DataQualityAssessor
from neptoon.hub import CRNSDataHub

# Assuming you have a CRNSDataHub with data
data_hub = CRNSDataHub(crns_data_frame=your_data_frame)

# Add quality checks
data_hub.add_quality_flags(custom_flags=flag_builder)

# Apply the quality checks
data_hub.apply_quality_flags()

# Masking now follows each check's QA action.
# Flags remain available in data_hub.flags_data_frame.
# Row-level warning/error metadata is in data_hub.quality_events_data_frame.

Quality Check Methods

neptoon provides several quality check methods through the QAMethod enum:

Method	Description	Common Parameters
RANGE_CHECK	Flags values outside a specified range	min, max
SPIKE_OFFSET	Flags abrupt offsets or short plateaus relative to the previous value	threshold_relative, window
SPIKE_UNILOF	Detects spikes using the univariate Local Outlier Factor algorithm	periods_in_calculation, threshold, algorithm
SPIKE_ZSCORE	Detects spikes using a rolling z-score	periods_in_calculation, threshold, min_residual, centered
CONSTANT	Flags periods where values remain constant	-
PERSISTENCE_CHECK	Flags persistent or stuck values inside a rolling window	window, threshold, min_periods
ABOVE_N0	Flags neutron counts above a factor of the N0 calibration value	N0, percent_maximum
BELOW_N0_FACTOR	Flags neutron counts below a factor of the N0 calibration value	N0, percent_minimum

Finding Required Parameters

To discover what parameters are required for a specific quality check method, use the WhatParamsDoINeed utility:

from neptoon.quality_control import WhatParamsDoINeed, QAMethod

# Display parameter information for a method
WhatParamsDoINeed(QAMethod.RANGE_CHECK)

This will print detailed information about required and optional parameters for the specified method.

Neptoon and SaQC parameter names

Quality checks accept both Neptoon's descriptive parameter names and the parameter names used by the underlying SaQC method. Neptoon normalizes both forms to one canonical internal representation, so existing configurations remain unchanged while users familiar with SaQC can use the names from the SaQC documentation directly.

For example, these two checks are equivalent:

QualityCheck(
    target=QATarget.AIR_PRESSURE,
    method=QAMethod.SPIKE_UNILOF,
    parameters={"periods_in_calculation": 24, "threshold": 2.0},
)

QualityCheck(
    target=QATarget.AIR_PRESSURE,
    method=QAMethod.SPIKE_UNILOF,
    parameters={"n": 24, "thresh": 2.0},
)

The aliases may also be mixed per parameter. Supplying both names for the same parameter, for example both threshold and thresh, is rejected rather than choosing one silently. SaQC's field argument is intentionally not an alias: Neptoon selects the checked column through QATarget or column_name.

Check	Neptoon name	SaQC name
SPIKE_UNILOF	periods_in_calculation	n
SPIKE_UNILOF	threshold	thresh
SPIKE_ZSCORE	periods_in_calculation	window
SPIKE_ZSCORE	threshold	thresh
SPIKE_ZSCORE	min_residual	min_residuals
SPIKE_ZSCORE	centered	center
SPIKE_OFFSET	threshold_relative	thresh_relative
PERSISTENCE_CHECK	threshold	thresh

The same aliases are accepted in YAML configuration. Model dumps and runtime QualityCheck.parameters continue to use the Neptoon names.

Quality Check Targets

The following data columns can be targeted for quality checks via the QATarget enum:

Target	Description
RAW_EPI_NEUTRONS	Raw epithermal neutron counts
CORRECTED_EPI_NEUTRONS	Corrected epithermal neutron counts
RELATIVE_HUMIDITY	Air relative humidity
AIR_PRESSURE	Atmospheric pressure
AIR_TEMPERATURE	Air temperature
SOIL_MOISTURE_VOL	Calculated volumetric soil moisture
CUSTOM	User-defined column (requires specifying column_name in parameters)

Advanced Usage

Custom Quality Check Targets

To quality-check a column not covered by the standard QATarget enum:

from neptoon.quality_control import QualityCheck, QAMethod, QATarget

custom_check = QualityCheck(
    target=QATarget.CUSTOM,
    method=QAMethod.RANGE_CHECK,
    parameters={
        "column_name": "your_custom_column_name",  # Specify the column name
        "min": 0,
        "max": 100
    }
)

Changing the SaQC Flagging Scheme

neptoon uses SaQC's "simple" flagging scheme by default, but you can change this:

quality_assessor = DataQualityAssessor(data_frame=your_data_frame)
quality_assessor.change_saqc_scheme("dmp")  # Other options: "float", "positional", "annotated-float"

Direct Access to Flags

You can access the generated flags directly:

# Get the flags dataframe
flags_df = data_hub.flags_data_frame

# Values are "UNFLAGGED" or specific flags (depends on the SaQC scheme)
print(flags_df.head())

Recommendations for CRNS Data

For CRNS data processing, we recommend implementing the following quality checks as a starting point in your workflow:

1. Spike Detection on Raw Neutron Counts

Identify anomalous spikes in your raw neutron count data:

QualityCheck(
    target=QATarget.RAW_EPI_NEUTRONS,
    method=QAMethod.SPIKE_OFFSET,
    parameters={
        "window": "24h",
        "threshold_relative": (0.2, -0.2),
    },
)

2. Meteorological Variable Validation

Ensure meteorological variables are within physically reasonable ranges:

# Air pressure range check 
QualityCheck(
    target=QATarget.AIR_PRESSURE,
    method=QAMethod.RANGE_CHECK,
    parameters={"min": 800, "max": 1100} 
)

# Relative humidity range check
QualityCheck(
    target=QATarget.RELATIVE_HUMIDITY,
    method=QAMethod.RANGE_CHECK,
    parameters={"min": 0, "max": 100}  # Percentage (0-100%)
)

# Temperature checks....

further processing here

At this point you would correct your neutrons and calibrate your sensor to get an N0 value

3. Calibration-Based Checks (Apply After Calibration)

After performing calibration and determining your N0 value, add these checks based on the N0:

qa_flags_2 = QualityAssessmentFlagBuilder()
qa_flags_2.add_check(

    QualityCheck(
        target=QATarget.CORRECTED_EPI_NEUTRONS,
        method=QAMethod.ABOVE_N0,
        parameters={
            "percent_maximum":1.075,
            "N0":data_hub.sensor_info.N0
                }),

    QualityCheck(
        target=QATarget.CORRECTED_EPI_NEUTRONS,
        method=QAMethod.BELOW_N0_FACTOR,
        parameters={
            "N0":data_hub.sensor_info.N0,
            "percent_minimum":0.3
                }),
    )
data_hub.add_quality_flags(custom_flags=qa_flags_2)
data_hub.apply_quality_flags()

4. Persistence checks for stuck sensors

Persistence checks are useful when a sensor or logger repeats the same value for longer than expected. They use SaQC's constant-value check with Neptoon-specific parameter names.

QualityCheck(
    target=QATarget.RELATIVE_HUMIDITY,
    method=QAMethod.PERSISTENCE_CHECK,
    parameters={
        "window": 12,
        "min_periods": 6,
        "threshold": 0,
    },
)

In YAML configuration files the same method is written as persistence_check:

input_data_qa:
  air_relative_humidity:
    persistence_check:
      window: 12
      min_periods: 6
      threshold: 0

The parameter values presented here are general guidelines. You should adjust these values based on your: - Site-specific environmental conditions - Sensor model and characteristics - Elevation and local atmospheric patterns - Expected soil moisture range

These quality checks should be applied at appropriate stages in your processing pipeline: spike detection on raw data, meteorological variable validation before corrections, and N0-based checks after calibration has been performed.

QA severity and actions

Detection, severity, and data handling are separate concepts. severity describes the scientific importance of a QA event; action controls what Neptoon does with the affected value.

Severity	Default action	Intended use
error	mask	Values that should not propagate into processed products
warning	keep	Unusual but potentially real events that should remain visible

The action may be overridden explicitly with keep, mask, or raise. raise stops processing with a user-facing quality-assessment error.

from neptoon.quality_control import QAAction, QAMethod, QASeverity, QATarget, QualityCheck

storm_warning = QualityCheck(
    target=QATarget.RELATIVE_HUMIDITY,
    method=QAMethod.SPIKE_OFFSET,
    parameters={
        "window": "12h",
        "threshold_relative": (0.2, -0.2),
    },
    severity=QASeverity.WARNING,
    action=QAAction.KEEP,
)

After QA runs, CRNSDataHub.quality_events_data_frame contains one row per detected event with its timestamp, target column, method, flag, severity, action, and original value. CRNSDataHub.quality_actions_data_frame contains the resolved per-value action after multiple checks are combined; the most restrictive action wins (raise > mask > keep).

When outputs are saved, non-empty QA events are written as <station>_qa_events.csv and/or JSON using the configured output formats. When figures/reports are enabled, Neptoon can create a quality_assessment_events timeline under the Data Preparation report topic so kept warnings remain visible instead of disappearing into logs.

QA flagging summaries

After configured QA checks run, Neptoon reports a compact warning/error summary. Legacy checks still behave exactly as before: flagged values are errors and are masked as NaN. Warning checks can keep their data while remaining present in the flag table, QA event output, and optional QA figure.