Visualisation, logging, utilities, and audit¶
Figures, report sampling, exceptions, logging, audit helpers, compatibility utilities, secrets, and GUI support helpers.
This committed category page is the canonical mkdocstrings location, so links remain valid in GitLab and on the rendered site. The generated module index mirrors the Python package tree and links back to these canonical API sections without rendering objects twice.
The audit log is optional observability infrastructure and must not destabilize
scientific processing. Creation falls back to a disabled audit when its log file
cannot be opened; cleanup is idempotent; and archive, parsing, serialization,
and output failures use NeptoonOutputError. Malformed audit lines retain their
line number in the error so an operator can diagnose the original file.
neptoon.data_audit.data_audit¶
data_audit ¶
core_logger
module-attribute
¶
core_logger = get_logger()
IDEAS:
The DataAuditLog is currently named by the user and a warning tells the user when multiple logs of same name are shown (it would add logging to old file). Perhaps in the final form it uses a universal name and gets auto deleted when it is parsed into a YAML. That way the DataAuditLog only there for the instance? For now I'll leave in the naming part but perhaps do this.
DataAuditLog ¶
The DataAuditLog in a singleton pattern
__new__ ¶
Singleton pattern to ensure only one DataAuditLog is created at any one time. When instantiated it uses the init_data_audit_log method for creation.
archive_and_delete_log
classmethod
¶
add_step ¶
Adds a record of a processing step to the log
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
function_name
|
str
|
The name of the function being recorded. |
required |
parameters
|
dict
|
The parameters being applied in the function that are set to be recorded by the decorator: @key_steps_log() |
required |
ParseDataAuditLog ¶
Parse the DataAuditLog file into a YAML
parse_log_to_yaml_string
staticmethod
¶
Convert structured JSONL or a legacy audit log into YAML.
New logs contain one versioned JSON object per line. Historical
function: ... parameters: {...} logs remain readable so existing
audit archives and interrupted runs do not become unusable.
hash_yaml_string
staticmethod
¶
Creates a SHA256 hash based on the Yaml string provided.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
yaml_string
|
str
|
String in YAML style |
required |
Returns:
| Type | Description |
|---|---|
SHA256 Hash
|
The SHA256 Hash representing the string |
normalize_audit_value ¶
Return a deterministic JSON-safe representation of an audit value.
Audit records may contain enums, paths, timestamps, Pydantic models, NumPy
scalars, nested mappings and sequences. Converting them before writing means
the audit trail never has to reconstruct Python repr strings.
log_key_step ¶
Decorator which is used to record functions and values used in such functions.
neptoon.exceptions¶
exceptions ¶
Neptoon exception hierarchy and error-boundary helpers.
Public Neptoon boundaries should expose Neptoon exception types rather than raw
Python or third-party exceptions. Expected operational failures use the
NeptoonUserFacingError hierarchy. Unexpected implementation defects use
NeptoonInternalError and preserve the original exception as __cause__
when raised with raise ... from error.
NeptoonError
dataclass
¶
Bases: Exception
Base class for failures that cross a public Neptoon boundary.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
message
|
str
|
Short explanation of what failed. |
required |
stage
|
str
|
Pipeline/domain stage where the failure was identified. |
'workflow'
|
hint
|
str | None
|
Optional diagnostic or recovery hint. |
None
|
code
property
¶
Return the stable diagnostic code used by UI-facing messages.
error_type remains the canonical exception-facing name. code
is an additive alias so GUI and service adapters can consume errors,
diagnostics, and validation issues through the same attribute name.
to_dict ¶
Return a language-neutral representation of this public error.
The payload deliberately keeps both code and error_type.
code aligns exceptions with workflow diagnostics while
error_type preserves the established Python-facing contract.
python_error_type is present for a stable schema and is populated
only by :class:NeptoonInternalError.
ensure_stage ¶
Attach a concrete stage when an older producer used the default.
This lets the nearest public/domain boundary enrich legacy Neptoon exceptions without overwriting a more specific stage chosen at source.
NeptoonUserFacingError
dataclass
¶
NeptoonInternalError
dataclass
¶
Bases: NeptoonError
Unexpected implementation failure crossing a public Neptoon boundary.
This error is intentionally not a NeptoonUserFacingError. It must not
be converted into a clean operational stop because programming defects need
a traceback. python_error_type records the native/third-party exception
type while the chained __cause__ retains the original traceback.
NeptoonConfigurationError
dataclass
¶
NeptoonDependencyError
dataclass
¶
NeptoonInputDataError
dataclass
¶
NeptoonColumnMappingError
dataclass
¶
NeptoonTimestampError
dataclass
¶
NeptoonExternalDataError
dataclass
¶
NeptoonCalibrationError
dataclass
¶
NeptoonNoUsableDataError
dataclass
¶
NeptoonOutputError
dataclass
¶
NeptoonRuntimeError
dataclass
¶
NeptoonRunLockedError
dataclass
¶
NeptoonRunTimeoutError
dataclass
¶
NeptoonStaleDataError
dataclass
¶
Bases: NeptoonInputDataError
Input data is structurally valid but too old or unchanged for server use.
NeptoonProcessingCancelled
dataclass
¶
neptoon.interface.neptoon_gui_utils¶
neptoon_gui_utils ¶
save_uploaded_file ¶
Save a Streamlit upload to a uniquely named temporary file.
render_html_content ¶
Render trusted Neptoon-generated HTML inside the Streamlit GUI.
render_html_file ¶
Render a saved HTML artifact inside the Streamlit GUI.
save_uploaded_files_to_directory ¶
Save Streamlit uploads below one directory using safe relative names.
Relative subdirectories are preserved when a host supplies them. Duplicate names with different content are rejected instead of silently overwriting a previous upload.
neptoon.logging.log_utils¶
log_utils ¶
neptoon.logging.logging¶
logging ¶
Robust Neptoon logging configuration.
Logging is infrastructure, not a reason for a scientific workflow to fail. If the configured log directory is unavailable Neptoon falls back to stderr and continues with an explicit warning. Multiprocessing children never own the shared rotating file because Windows does not permit one process to rename a log file while another process still has it open.
cli_logging_scope ¶
Keep CLI console output single-sourced while preserving log capture.
Neptoon deliberately leaves global_logger.propagate enabled for normal
library use so embedding applications, pytest, systemd, and container log
collectors can capture records through their root handlers. A command-line
workflow is different: configured processing also emits curated user-facing
messages through write_user_output/Rich. If a launcher such as uv
has installed a root handler, those mirrored records otherwise appear a
second time as INFO:global_logger:....
During the CLI workflow only, stop propagation to external root handlers and provide one Neptoon-owned stderr handler. Records explicitly marked as also rendered through the user-output channel are filtered from that handler; all logger-only INFO/WARNING/ERROR records remain visible once. The managed file handler continues to receive every record, including the mirrored ones.
If file logging is unavailable and Neptoon already owns a stderr fallback, the same duplicate filter is attached temporarily to that handler instead of adding a second stderr handler. The original propagation/filter state is restored on exit, including exceptional exits.
configure_worker_logging ¶
Configure process-safe logging for a multiprocessing worker.
Worker processes write to stderr only. In particular they never inherit or
open Neptoon's shared TimedRotatingFileHandler. This prevents Windows
workers from keeping core_log.log open while the parent rotates it at
midnight. The function is idempotent and is also called explicitly by
long-lived soil-profile process workers so POSIX fork children drop an
inherited parent file descriptor.
configure_logger ¶
Return the process logger with a file handler or stderr fallback.
NEPTOON_LOG_DIR can override the normal platform cache directory.
Repeated imports or test reloads do not add duplicate handlers. Child
processes deliberately use stderr-only logging so only the parent owns the
shared rotating log file.
neptoon.logging.output¶
output ¶
Helpers for optional user-facing console output.
Configured workflows read the output setting from the active
:class:~neptoon.runtime.ProcessingContext. set_output_enabled remains as
a backward-compatible fallback for standalone library calls.
set_output_enabled ¶
Set output for the active run or the standalone default.
Configured workflows own an explicit processing context, so changing output inside one run cannot affect another thread or task. Outside a context this function preserves the historical standalone behavior in the current thread or asynchronous task.
user_output_log_extra ¶
Mark a log record only when the same message will reach user output.
emit_user_message ¶
emit_user_message(logger, message, *, output, level=INFO, enabled=None, output_message=None, exc_info=False)
Log a message and mirror it once to the user-facing output channel.
The helper is the single coordination point for messages that intentionally
belong in both Neptoon's operational log and its optional CLI/user output.
It marks the log record so :func:cli_logging_scope can suppress only the
duplicate console rendering while preserving the record in file logs and in
normal library/root-handler capture.
output is supplied by the caller instead of imported internally. This
preserves module-local monkeypatching and embedding hooks. When enabled
is explicit it is forwarded to callbacks such as write_user_output; when
omitted the callback retains its historical one-argument call shape.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
logger
|
Logger
|
Logger that receives the operational record. |
required |
message
|
object
|
Message written to the log. |
required |
output
|
Callable[..., None]
|
Caller-owned user-output callback. |
required |
level
|
int | str
|
Logging level or standard level name, by default |
INFO
|
enabled
|
bool | None
|
Explicit user-output state. If omitted, the active processing context or standalone output setting is used. |
None
|
output_message
|
object | None
|
Alternative text for the user-facing channel. The log message remains unchanged. |
None
|
exc_info
|
bool
|
Include the active exception traceback in the log record. |
False
|
write_user_output ¶
Print a user-facing message only when optional output is enabled.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
message
|
object
|
Message to print. |
required |
enabled
|
bool | None
|
Per-call override. If omitted, the active processing context or the
standalone fallback set by |
None
|
neptoon.utils.docker_utils¶
docker_utils ¶
is_running_in_docker ¶
Checks whether the current kernel is running in a docker container.
Returns:
| Type | Description |
|---|---|
bool
|
Whether in docker container |
return_file_path_with_suffix ¶
This is specifically used internally during docker runs. Input data is appended as /workingdir/inputdata. This could be a .csv .zip .tar or folder. This will check and return name with correct suffix.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
base_path
|
str
|
The path to check |
required |
Returns:
| Type | Description |
|---|---|
str
|
path with suffix |
neptoon.utils.flaggable_columns¶
flaggable_columns ¶
neptoon.utils.general_utils¶
general_utils ¶
TemporalResolutionSummary
dataclass
¶
TemporalResolutionSummary(median_seconds, p10_seconds, p90_seconds, min_seconds, max_seconds, variability_ratio, sample_count)
Diagnostic summary of positive timestamp intervals.
is_highly_variable
property
¶
Return True when the median cadence is not representative.
validate_timestamp_index ¶
Validate that a dataframe has a complete DatetimeIndex.
validate_and_convert_file_path ¶
Ensures that file paths are correctly parsed into pathlib.Path objects.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
file_path
|
str | Path | None
|
The path to the folder or file. |
required |
base
|
str | Path | None
|
Base to add to file (e..g, a custom base directory) |
None
|
Returns:
| Type | Description |
|---|---|
Path | None
|
The file_path as a pathlib.Path object. |
Raises:
| Type | Description |
|---|---|
ValueError
|
Error if string, pathlib.Path, or None not given. |
timedelta_to_freq_str ¶
Convert a positive fixed timedelta to an exact pandas frequency string.
Sub-second durations must not be routed through total_seconds() and
truncated to an integer number of seconds. pandas.Timedelta can carry
nanosecond precision, while datetime.timedelta carries microseconds.
parse_resolution_to_timedelta ¶
Parse a string representation of a time resolution and convert it to a timedelta object.
This method takes a string describing a time resolution (e.g., "30 minutes", "2 hours", "1 day") and converts it into a Python timedelta object. It supports minutes, hours, and days as units.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
resolution_str
|
str
|
A string representing the time resolution. The format should
be " |
required |
Returns:
| Type | Description |
|---|---|
timedelta
|
A timedelta object representing the parsed time resolution. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the resolution string format is invalid or cannot be parsed. |
ValueError
|
If an unsupported time unit is provided. |
validate_data_frame ¶
Validates a data frame against a pandera.pandas DataFrameSchema.
df is accepted as a backward-compatible keyword alias for older
call sites and notebooks. Prefer data_frame in new code.
NOTES: Keep it lazy to give info of all df issues
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data_frame
|
DataFrame
|
DataFrame to validate |
None
|
schema
|
DataFrameSchema
|
Pandera Schema to check against |
None
|
df
|
DataFrame
|
Backward-compatible alias for data_frame. |
None
|
validate_df ¶
Backward-compatible alias for :func:validate_data_frame.
describe_temporal_resolution ¶
Describe the positive timestamp spacing of a time-series frame.
find_temporal_resolution_seconds ¶
Return the median positive temporal resolution in seconds.
The function fails early with actionable messages for empty frames, non-datetime indexes, one-row frames, duplicated-only indexes, or non-positive/NaT temporal differences. When the 10th-to-90th percentile interval ratio is large, it logs a warning because the median cadence is a poor alignment grid for highly variable acquisition schedules.
return_df_with_temporal_resolution_seconds ¶
is_resolution_greater_than ¶
Returns True if resolution_a is greater (coarser) than resolution_b
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
resolution_a
|
str | timedelta
|
First resolution to compare |
required |
resolution_b
|
str | timedelta
|
Second resolution to compare |
required |
Returns:
| Type | Description |
|---|---|
bool
|
True if resolution_a > resolution_b |
Note
If resolution is str then it should be in a form such as "1h", "6hour" or "1day". It will be auto converted internally to the func
recalculate_neutron_uncertainty ¶
Adjust the staistical uncertainty of neutrons value based on the aggregation.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data_frame
|
DataFrame
|
DataFrame with data |
required |
temporal_scaling_factor
|
int | float
|
The scaling factor to adjust from original resolution to revised output resolution |
required |
uncertainty_col_name
|
str | None
|
Name of the col, if None will use the default supplied in ColumnInfo.Name.CORRECTED_EPI_NEUTRON_COUNT_UNCERTAINTY, by default None |
None
|
Returns:
| Type | Description |
|---|---|
_type_
|
description |
neptoon.utils.optional_dependencies¶
optional_dependencies ¶
Helpers for loading genuinely optional third-party dependencies.
Only the requested top-level package is treated as optional. Import failures for transitive dependencies, binary extensions, or package initialization are re-raised so users see the real problem instead of an inaccurate install hint.
OptionalDependencyError
dataclass
¶
Bases: NeptoonDependencyError, ImportError
Raised when a requested optional top-level package is not installed.
import_optional_dependency ¶
Import an optional top-level package without hiding transitive failures.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
module_name
|
str
|
Exact top-level package to import. |
required |
feature
|
str
|
User-facing description of the feature requiring the package. |
required |
extra
|
str
|
Neptoon optional-dependency extra that installs the package. |
None
|
install_hint
|
str
|
Explicit installation command for optional packages that are distributed
outside Neptoon extras. Exactly one of |
None
|
Returns:
| Type | Description |
|---|---|
ModuleType
|
Imported package. |
Raises:
| Type | Description |
|---|---|
OptionalDependencyError
|
If and only if |
ModuleNotFoundError
|
If package initialization fails because another dependency is missing. |
ImportError
|
For any other import-time failure inside the package. |
neptoon.utils.pandas_compat¶
pandas_compat ¶
Compatibility helpers for pandas frequency and timedelta aliases.
Pandas 2.2 deprecated several uppercase aliases (for example H and
T) and pandas 3.x removes some of them. Keep user-facing YAML strings
stable by normalizing legacy aliases before they are handed to pandas.
normalize_pandas_timedelta ¶
Return value with deprecated pandas timedelta aliases normalized.
Examples:
"1H" -> "1h"
"30T" -> "30min"
"500L" -> "500ms"
to_pandas_timedelta ¶
Create a :class:pandas.Timedelta after alias normalization.
normalize_pandas_frequency ¶
Normalize deprecated pandas frequency aliases.
This covers both fixed-duration aliases (H -> h, T ->
min) and anchored resampling aliases (M -> ME, Q ->
QE, Y -> YE). Non-string values are returned unchanged.
is_fixed_duration_frequency ¶
Return True if value can be represented as a fixed Timedelta.
neptoon.utils.secrets¶
secrets ¶
Helpers for passing Pydantic secret values to runtime clients.
reveal_secret ¶
Return the underlying secret value at an explicit runtime boundary.
resolve_secret ¶
Resolve an explicit secret or fall back to an environment variable.
Explicit values always win. The environment-variable name itself is not a secret and can therefore be stored safely in configuration files.
neptoon.utils.soil_moisture_units¶
soil_moisture_units ¶
Utilities for normalising configured soil-moisture units.
SOIL_MOISTURE_UNITS
module-attribute
¶
SOIL_MOISTURE_UNITS = (SOIL_MOISTURE_UNIT_AUTO, SOIL_MOISTURE_UNIT_FRACTION, SOIL_MOISTURE_UNIT_VOL_PERCENT)
normalise_soil_moisture_unit ¶
Return the canonical soil-moisture unit name.
Canonical values are auto, fraction and vol_percent. Common
human-readable spellings are accepted so YAML configs can use familiar
scientific notation such as m3/m3 or vol%.
normalise_soil_moisture_display_unit ¶
Return a canonical display unit for soil-moisture plots.
Display units must be explicit because auto is meaningful only for
interpreting input series. Supported display units are fraction
(m³/m³) and vol_percent (vol%).
calculate_soil_moisture_display_scale ¶
Return the factor that converts an internal fraction to display units.
get_soil_moisture_display_label ¶
Return the conventional y-axis label for a display unit.
neptoon.utils.uncertainty¶
uncertainty ¶
Formatting helpers for values with separated uncertainty classes.
format_value_with_uncertainties ¶
format_value_with_uncertainties(value, *, statistical=None, systematic=None, unit=None, significant_digits=6)
Format value ± statistical ± systematic without merging classes.
Parameters are interpreted as one-standard-deviation uncertainties. Missing classes are omitted rather than displayed as zero.
neptoon.visualisation.figures¶
figures ¶
Here are basic figures for creating plots.
REPORT_PALETTE
module-attribute
¶
REPORT_PALETTE = {'ink': '#1F2937', 'muted': '#6B7280', 'frame': '#000000', 'grid': '#D1D5DB', 'navy': '#1F4E79', 'blue': '#2F80ED', 'teal': '#1B9E77', 'cyan': '#2A9D8F', 'orange': '#D97706', 'red': '#B23A48', 'purple': '#6D597A', 'grey': '#9CA3AF', 'light_grey': '#E5E7EB'}
REPORT_LINE_COLORS
module-attribute
¶
REPORT_LINE_COLORS = [REPORT_PALETTE['navy'], REPORT_PALETTE['teal'], REPORT_PALETTE['orange'], REPORT_PALETTE['purple'], REPORT_PALETTE['blue'], REPORT_PALETTE['red'], REPORT_PALETTE['muted']]
REFERENCE_SERIES_DASH_PATTERNS
module-attribute
¶
REFERENCE_SERIES_COLORS
module-attribute
¶
REFERENCE_SERIES_COLORS = (REPORT_PALETTE['orange'], REPORT_PALETTE['purple'], REPORT_PALETTE['cyan'], REPORT_PALETTE['red'], REPORT_PALETTE['muted'])
validate_columns_present ¶
Utility function to validate column existence in data frame Raises ValueError if columns are missing
make_nmdb_data_figure ¶
make_nmdb_data_figure(data_frame, reference_value, nmdb_station_name=None, incoming_neutron_col_name=str(INCOMING_NEUTRON_INTENSITY), resolution=60, show=False, backend='agg', save_location=None, report_plot_downsampling=True)
Makes the figure
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data_frame
|
DataFrame
|
DataFrame containing data |
required |
nmdb_station_name
|
str
|
NMDB station name |
None
|
reference_value
|
int
|
reference value |
required |
resolution
|
int
|
resolution in minutes, by default 60 |
60
|
show
|
bool
|
show interactively, by default False |
False
|
backend
|
str
|
the backend to use in matplotlib, by default "agg |
'agg'
|
save_location
|
str
|
Path used to save the figure. |
None
|
report_plot_downsampling
|
bool
|
Downsample dense report plots when enabled. |
True
|
Returns:
| Type | Description |
|---|---|
BytesIO
|
Figure object to be used for later display |
uncorr_and_corr_neutrons_figure ¶
uncorr_and_corr_neutrons_figure(data_frame, station_name, raw_neutron_col_name=str(EPI_NEUTRON_COUNT_FINAL), corr_neutron_col_name=str(CORRECTED_EPI_NEUTRON_COUNT_FINAL), corr_unsmoothed_col_name=str(CORRECTED_EPI_NEUTRON_COUNT), show=False, backend='agg', save_location=None, report_plot_downsampling=True)
soil_moisture_figure_uncertainty ¶
soil_moisture_figure_uncertainty(data_frame, station_name, soil_moisture_col=str(SOIL_MOISTURE_VOL_FINAL), statistical_uncertainty_col=str(SOIL_MOISTURE_STATISTICAL_UNCERTAINTY_VOL), systematic_uncertainty_col=str(SOIL_MOISTURE_SYSTEMATIC_UNCERTAINTY_VOL), upper_uncertainty_col=str(SOIL_MOISTURE_UNCERTAINTY_VOL_UPPER), lower_uncertainty_col=str(SOIL_MOISTURE_UNCERTAINTY_VOL_LOWER), precipitation_col=str(PRECIPITATION), daily_precipitation_col=str(PRECIPITATION_DAILY), sm_range=(0, 0.5), show=False, backend='agg', save_location=None, report_plot_downsampling=True, precipitation_resolution='auto')
Create a soil-moisture figure with separated uncertainty classes.
The reconstructed soil-moisture panel includes the available total one-standard-deviation band. A dedicated uncertainty panel then shows the statistical, systematic and quadratically combined 1-sigma magnitudes on their own scale so small errors remain visible. Legacy combined upper/lower bounds are used when separated uncertainty columns are unavailable.
create_soil_moisture_sensor_comparison_figure ¶
create_soil_moisture_sensor_comparison_figure(data_frame, station_name, sensor_columns, soil_moisture_col=str(SOIL_MOISTURE_VOL_FINAL), precipitation_col=str(PRECIPITATION), daily_precipitation_col=str(PRECIPITATION_DAILY), sm_range=None, include_precipitation=True, title=None, crns_label='CRNS soil moisture', y_label=None, show=False, backend='agg', save_location=None, report_plot_downsampling=True, precipitation_resolution='auto', sensor_units=None, sensor_labels=None, display_unit=SOIL_MOISTURE_UNIT_FRACTION)
Plot CRNS soil moisture with additional in-situ soil-moisture sensors.
CRNS soil moisture is stored as a volumetric fraction (m³/m³), while many in-situ SMT/logger exports use volumetric percent. Input series are first normalised to the internal fraction representation and then converted to the configured display unit so all lines share one physical y-axis. When configured and available, precipitation is drawn with the same inverted right-axis convention as the existing soil-moisture report plot.
soil_moisture_coloured_figure ¶
soil_moisture_coloured_figure(data_frame, station_name, sm_column_name=str(SOIL_MOISTURE_VOL_FINAL), lower_bound=0, save_location=None, report_plot_downsampling=True)
Plot soil moisture with an efficient wet-to-dry colour field.
Report-only downsampling is applied before drawing. The colour field is one
raster image clipped below the soil-moisture line rather than dozens of
full-series fill_between polygons. Processed output data are unchanged.
atmospheric_conditions_figure ¶
atmospheric_conditions_figure(data_frame, station_name, pressure_col=str(AIR_PRESSURE), temperature_col=str(AIR_TEMPERATURE), rel_humidity_col=str(AIR_RELATIVE_HUMIDITY), temperature_range=(-20.0, 40.0), show=False, backend='agg', save_location=None, report_plot_downsampling=True)
Creates a three-panel figure showing atmospheric conditions affecting CRNS measurements.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data_frame
|
DataFrame
|
DataFrame containing the atmospheric measurements |
required |
station_name
|
str
|
Name of the CRNS station for the title |
required |
pressure_col
|
str
|
Column name for air pressure data (hPa) |
str(AIR_PRESSURE)
|
temperature_col
|
str
|
Column name for air temperature data (°C) |
str(AIR_TEMPERATURE)
|
rel_humidity_col
|
str
|
Column name for relative humidity data (%) |
str(AIR_RELATIVE_HUMIDITY)
|
temperature_range
|
Tuple[float, float]
|
Temperature axis range (min, max) in °C |
(-20.0, 40.0)
|
show
|
bool
|
Whether to display the figure |
False
|
backend
|
str
|
Matplotlib backend to use |
'agg'
|
save_location
|
str
|
Path to save the figure |
None
|
Notes
The function handles missing data gracefully - panels with no data will be left blank with an appropriate message. This is particularly useful for stations that may not record all atmospheric variables.
create_dwd_weather_figure ¶
create_dwd_weather_figure(data_frame, station_name, precipitation_col=str(PRECIPITATION), daily_precipitation_col=str(PRECIPITATION_DAILY), solar_radiation_col=str(SOLAR_RADIATION), show=False, backend='agg', save_location=None, report_plot_downsampling=True)
Create a report figure for optional DWD/external weather products.
The atmospheric report page intentionally focuses on meteorological variables that affect neutron corrections (pressure, temperature and humidity). Precipitation and solar radiation are optional DWD/public meteo additions and therefore live in this separate figure. The function is intentionally permissive: it plots whichever optional products are present and contain at least one numeric value, and it raises only when none of the supported products is available.
correction_factors_figure ¶
correction_factors_figure(data_frame, station_name, pressure_corr_col=str(PRESSURE_CORRECTION), humidity_corr_col=str(HUMIDITY_CORRECTION), intensity_corr_col=str(INTENSITY_CORRECTION), biomass_corr_col=str(ABOVEGROUND_BIOMASS_CORRECTION), show=False, backend='agg', save_location=None, report_plot_downsampling=True)
Creates a figure showing correction factors applied to CRNS data.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data_frame
|
DataFrame
|
DataFrame containing the correction factors and soil properties |
required |
station_name
|
str
|
Name of the CRNS station for the title |
required |
pressure_corr_col
|
str
|
Column name for pressure correction factor |
str(PRESSURE_CORRECTION)
|
humidity_corr_col
|
str
|
Column name for humidity correction factor |
str(HUMIDITY_CORRECTION)
|
intensity_corr_col
|
str
|
Column name for incoming neutron intensity correction |
str(INTENSITY_CORRECTION)
|
show
|
bool
|
Whether to display the figure |
False
|
backend
|
str
|
Matplotlib backend to use |
'agg'
|
save_location
|
str
|
Path to save the figure |
None
|
create_gridded_reference_comparison_figure ¶
create_gridded_reference_comparison_figure(data_frame, station_name, reference_columns, soil_moisture_col=str(SOIL_MOISTURE_VOL_FINAL), precipitation_col=str(PRECIPITATION), daily_precipitation_col=str(PRECIPITATION_DAILY), reference_labels=None, caption=None, sm_range=None, include_precipitation=True, title=None, crns_label='CRNS soil moisture', y_label=None, show=False, backend='agg', save_location=None, report_plot_downsampling=True, precipitation_resolution='auto', display_unit=SOIL_MOISTURE_UNIT_FRACTION)
Plot CRNS soil moisture against gridded reference series.
The reference series come from satellite products or from land-surface reanalysis. They are drawn so they cannot be mistaken for the CRNS series: the CRNS line is solid and navy, every reference line is dashed, thinner and a different colour, and each legend entry carries its provenance class and native resolution. The caption repeats the provenance and states plainly that these series enter no calculation.
The CRNS series is optional. Comparing reference series among themselves is a legitimate question, and a station whose soil moisture has not been derived yet should still get the plot rather than an exception.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data_frame
|
DataFrame
|
Frame holding the CRNS series and the reference columns. |
required |
station_name
|
str
|
Station name for the default title. |
required |
reference_columns
|
List[str]
|
Gridded reference columns to draw. Columns that are absent or hold no numeric data are skipped, because an optional reference series is routinely unavailable for part of a period. |
required |
reference_labels
|
Mapping[str, str] | None
|
Legend label per reference column, normally including its provenance. Falls back to the column name. |
None
|
caption
|
str | None
|
Provenance caption written under the axes. |
None
|
create_additional_columns_figure ¶
create_additional_columns_figure(data_frame, columns, title, y_label=None, show=False, backend='agg', save_location=None, report_plot_downsampling=True)
Create a report figure with user-selected columns on one axis.
All named columns that exist and contain numeric data are drawn into the same plot. Missing or non-numeric columns are skipped so a report can still be produced when optional logger channels are absent in a particular file.
create_snow_flag_figure ¶
create_snow_flag_figure(data_frame, station_name, snow_column, flag_column, snow_unit='m', threshold=None, soil_moisture_col=str(SOIL_MOISTURE_VOL_FINAL), caption='', show=False, backend='agg', save_location=None, report_plot_downsampling=True)
Show the snow condition as bars, and the product it questions above it.
Bars rather than a line because the flag is a per-timestamp decision, and a bar carries the one thing a reader has to take away: which hours Neptoon considers contaminated. Flagged bars are drawn in the alert colour, bars that were read and stayed below the threshold in grey, so "checked and snow-free" and "flagged" are told apart at a glance. Hours with no snow value at all are absent from both, which is why the caption names their count instead of leaving a gap to interpret.
The soil-moisture panel is drawn when the column exists, with the flagged hours shaded behind it. That is the whole point of a flag that removes nothing: the reader sees the values that stand, and where they would be dropped if they chose to drop them.
create_quality_assessment_events_figure ¶
create_quality_assessment_events_figure(events, station_name, show=False, backend='agg', save_location=None)
Plot row-level QA warning/error events on a categorical timeline.
The y-axis is the affected data column so values with incomparable units do not share an artificial numeric scale. Marker colour communicates severity and marker shape communicates whether Neptoon kept or masked the value.
neptoon.visualisation.figures_handler¶
figures_handler ¶
FIGURE_CREATION_RECOVERABLE_ERRORS
module-attribute
¶
FigureTopic ¶
Bases: Enum
FigureMetadata
dataclass
¶
Key information on each figure created.
topic : FigureTopic The topic the figure is related to description : str A brief description about the figure required_columns: List[str] The columns required for this particular figure. method : callable The method (found in FigureHandler) which will be called to produce this figure.
TempFigure
dataclass
¶
TempFigureHandler ¶
Handles the creation and storage of figures made with the FigureHandler by creating a temp directory, recording paths for each figure, and removing temp files upon program exit.
store_figure ¶
Stores metadata of temporary figure storage
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name
|
str
|
name of the figure being stored |
required |
topic
|
FigureTopic
|
Topic the figure is part of (e.g., NMDB, or Soil) |
required |
Returns:
| Type | Description |
|---|---|
Path
|
Path of the temp figure |
get_figures ¶
Collect the figures that are saved
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
topic
|
FigureTopic
|
Topic the figure is part of (e.g., NMDB, or Soil) |
None
|
Returns:
| Type | Description |
|---|---|
List[TempFigure]
|
A list of TempFigure objects with location of save files. |
FigureHandler ¶
FigureHandler(data_frame, sensor_info=None, create_all=False, ignore_sections=None, selected_figures=None, show_figures=False, backend='Agg', enable_output=False, report_plot_downsampling=True, additional_plots=None, soil_moisture_sensor_comparison=None, precipitation_resolution='auto', processing_context=None, quality_events_data_frame=None)
Class to manage creating figures when data has been processed using the CRNSDataHub.
ignore_sections
instance-attribute
¶
selected_figures
instance-attribute
¶
report_plot_downsampling
instance-attribute
¶
soil_moisture_sensor_comparison
instance-attribute
¶
precipitation_resolution
instance-attribute
¶
quality_events_data_frame
instance-attribute
¶
quality_events_data_frame = copy() if isinstance(quality_events_data_frame, DataFrame) else DataFrame()
neptoon.visualisation.report_sampling¶
report_sampling ¶
Automatic report-only downsampling helpers.
The functions in this module reduce the number of points drawn in PDF/report figures without touching the processed output data. The policy is deliberately small and automatic: line-like time series are reduced to at most roughly 5000 points with Largest-Triangle-Three-Buckets (LTTB), while precipitation is aggregated as sums when the plotted period is long.
SampledBand
dataclass
¶
report_downsampling_enabled ¶
Return the effective report-downsampling setting.
None is treated as the default, which is enabled.
sample_line_for_report ¶
sample_line_for_report(series, *, enabled=True, max_points=MAX_REPORT_POINTS)
Return a visually representative copy of a line series.
The original data are never modified. Downsampling is skipped for short series. Large gaps between valid observations are preserved by inserting a NaN break point; small gaps may disappear when the plot is reduced, which is acceptable for report display.
sample_band_for_report ¶
sample_band_for_report(main, lower, upper, *, enabled=True, max_points=MAX_REPORT_POINTS)
Downsample a main line and align bounds to the same timestamps.
sample_precipitation_for_report ¶
sample_precipitation_for_report(series, *, enabled=True, max_points=MAX_REPORT_POINTS)
Return precipitation as report-friendly summed bars.
Precipitation is never averaged. If the time span is longer than 300 days, values are aggregated to daily sums independent of the generic point limit. If there are still too many bars after that, a coarser summed interval is chosen automatically.
sample_daily_precipitation_for_report ¶
sample_daily_precipitation_for_report(series, *, enabled=True, max_points=MAX_REPORT_POINTS)
Return daily precipitation as exactly one report bar per calendar day.
DWD daily products are already daily totals. After alignment to a sub-daily
CRNS time grid, the same daily value can be repeated on many timestamps.
Summing those repeated values would inflate the precipitation and drawing
each repeated timestamp as its own narrow bar creates vertical line
artefacts in the PDF. Use max per day instead: repeated aligned copies
of the same daily total collapse to the original daily value, while missing
days stay missing.
The enabled and max_points arguments are accepted for API symmetry
with the other report samplers, but daily precipitation deliberately keeps
one bin per calendar day instead of coarsening long records further.