Saving data¶

To save processed data, run:

data_hub.save_data()

That is the quickest way to save data. In this case it will save the outputs (that is the data, data flags, figures, pdf etc.) into a folder. The folder is named after the site (sensor_info.name) with a timestamp appended, so successive runs do not overwrite each other.

For YAML output settings, see the data_storage section in the sensor configuration guide. Report-specific settings are covered in PDF output. Applications consuming files from scheduled runs should also follow the atomic-output and completion guidance; extension authors can use the lower-level writers in the input/output API reference.

Where does it save to?¶

save_data() picks the output directory in this order:

The save_folder_location argument, if you pass one.
data_storage.save_location from the sensor config — used automatically if the hub was built with DataHubFromConfig (or ProcessWithConfig) and the YAML sets it.
NEPTOON_OUTPUT_DIR, when the environment variable is set.
The current working directory, but only when Neptoon can verify that it is writable.

Neptoon does not silently move persistent outputs into a temporary directory. If neither an explicit location nor NEPTOON_OUTPUT_DIR is available and the working directory is read-only, the run fails early with a NeptoonOutputError that explains how to choose a writable persistent location. This behaviour is important for read-only containers, hosted notebooks, and serverless/cloud runtimes where the application directory is commonly not writable.

Container-friendly environment override¶

For deployments where editing the station YAML is inconvenient, set a persistent output root in the environment:

export NEPTOON_OUTPUT_DIR=/data/neptoon/outputs

The variable is only consulted when neither save_folder_location nor data_storage.save_location is set. It is therefore safe to define a deployment-wide default while still allowing individual stations or CLI runs to choose their own output directory.

Passing a path directly¶

from pathlib import Path

data_hub.save_data(
    folder_name="first_test",
    save_folder_location=Path("/path/to/save/location"),
    use_custom_column_names=False,
    custom_column_names_dict=None,
    append_timestamp=True,
)

Missing intermediate directories are created for you.

Custom output column names¶

Pass a mapping from current Neptoon column names to export names and enable custom names when saving:

data_hub.save_data(
    use_custom_column_names=True,
    custom_column_names_dict={
        "air_pressure": "pressure_hpa",
        "air_temperature": "temperature_c",
    },
)

The mapping is applied consistently to processed-data, flag and calibration tables. Unknown source columns, duplicate targets and collisions with existing columns fail before any table is written.

Letting the sensor config decide¶

If you loaded a sensor config, put the path in the YAML and save_data() will use it with no argument:

# sensor_config.yaml
data_storage:
  save_location: ~/neptoon-outputs/station_A101
  create_report: true

from neptoon.io.read import DataHubFromConfig

data_hub = DataHubFromConfig(sensor_config=sensor_config).create_data_hub()
# ... processing steps ...
data_hub.save_data()   # writes to ~/neptoon-outputs/station_A101

Accepted path formats for save_location

Absolute — e.g. /home/user/outputs or C:/Users/me/outputs
Home-relative — e.g. ~/outputs (the ~ is expanded to your home directory)
Relative in YAML — e.g. ./outputs or outputs; these paths are resolved relative to the directory containing the YAML config file.
Relative in CLI overrides — e.g. --save-location ./outputs/test-run; these paths are resolved relative to the current working directory of the shell command.

A command-line --save-location override takes precedence over the YAML value. This difference is intentional: YAML paths travel with the config file, while CLI paths are interpreted in the context of the command you typed.

Optional JSON table exports¶

CSV remains the default output format. To write JSON sidecars for the standard processed-data, flags, calibration, and static-pivot tables, enable data_storage.save_json:

data_storage:
  save_location: ~/neptoon-outputs/station_A101
  save_csv: true
  save_json: true
  json_orient: records
  json_date_format: iso

To write JSON instead of CSV for tabular station outputs:

data_storage:
  save_csv: false
  save_json: true

json_orient: records writes a list of row objects and is convenient for APIs. Use json_orient: table when you want pandas schema information for more robust round-trips. json_date_format: iso keeps timestamps readable.

User-selected processed-data exports¶

Use output_columns when a downstream system needs a stable, explicitly ordered subset of the processed table instead of an automatically compacted table.

data_storage:
  output_mode: both
  output_columns:
    - date_time
    - smv
    - crns_measurement_depth_cm

Option	Default	Description
`output_mode`	`full`	`full` writes the standard processed-data and flags tables; `selected` writes the selected columns under the standard filenames; `both` keeps the standard tables and adds `_processed_data_selected` and `_flags_selected` sidecars.
`output_columns`	`[]`	Ordered list of columns for `selected` or `both`. `date_time` and `utc_timestamp` are export columns and may be selected explicitly. `selected_output_columns` is accepted as an alias.

For output_mode: both, a station named A101 writes both A101_processed_data.csv and A101_processed_data_selected.csv, together with their corresponding flag tables. Columns that do not have quality flags, such as crns_measurement_depth_cm, receive UNFLAGGED entries in the selected flag table so its schema stays aligned with the selected processed table.

Column selection is independent of static pivoting. A constant column can be moved to *_static_values.csv in the compact export and still remain present in the explicitly selected export.

Compact static-column exports¶

For integrations that prefer a compact time-series table, enable the static-pivot output. Fully constant columns are moved to a one-row static-values table and the time-series table receives a reference column.

data_storage:
  static_pivot_enabled: true
  static_pivot_mode: sidecar
  static_pivot_reference_column: static_profile_id
  static_pivot_profile_id: station_A101
  static_pivot_min_rows: 2
  static_pivot_exclude_columns:
    - date_time

Option	Default	Description
`static_pivot_enabled`	`false`	Enable compact static-column output.
`static_pivot_mode`	`sidecar`	`sidecar` keeps the normal processed-data table and adds compact sidecars; `replace` writes the compact table as the main processed-data table.
`static_pivot_reference_column`	`static_profile_id`	Reference column inserted into the compact time-series table.
`static_pivot_profile_id`	`null`	Optional explicit id for the one-row static profile. If omitted, the sensor name or output folder is used.
`static_pivot_min_rows`	`2`	Minimum row count before constant columns may be moved to the static table.
`static_pivot_exclude_columns`	`[]`	Columns that must always remain in the time-series table.

This export is additive by default. It does not change the normal processed CSV unless static_pivot_mode: replace is selected.