External data¶
NMDB, DWD, international meteorological providers, provider registries, and source metadata.
This committed category page is the canonical mkdocstrings location, so links remain valid in GitLab and on the rendered site. The generated module index mirrors the Python package tree and links back to these canonical API sections without rendering objects twice.
neptoon.external.sms_exchange¶
Optional metadata exchange with the Helmholtz Sensor Management System through the
public sms_interface client. Remote writes are opt-in and are not part of normal
scientific processing.
sms_exchange ¶
Sensor Management System (SMS) metadata exchange boundary.
Neptoon delegates authentication and HTTP details to the public
sms_interface package. This module provides a small, stable Neptoon-facing
adapter around JSON:API resources and deliberately keeps remote writes opt-in.
It does not participate automatically in scientific processing.
__all__
module-attribute
¶
__all__ = ['SMSConfigurationError', 'SMSExchangeClient', 'SMSExchangeError', 'SMSMappedMetadata', 'SMSMetadataBundle', 'SMSMutationResult', 'SMSResource', 'SMSWriteDisabledError']
SMSConfigurationError
dataclass
¶
SMSWriteDisabledError
dataclass
¶
SMSExchangeError
dataclass
¶
SMSResource
dataclass
¶
Language-neutral representation of one SMS JSON:API resource.
SMSMutationResult
dataclass
¶
SMSMappedMetadata
dataclass
¶
Pure, local SMS attribute payloads derived from Neptoon sensor metadata.
SMSMetadataBundle
dataclass
¶
Selected SMS metadata associated with one Neptoon sensor configuration.
SMSExchangeClient ¶
SMSExchangeClient(api, *, allow_write=False, device_id=None, configuration_id=None, site_id=None, include_device_properties=True, metadata_mapping=None)
Explicit Neptoon adapter around an initialized sms_interface.SMSAPI.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
api
|
_SMSAPI
|
Initialized SMS API object. Passing it explicitly supports dependency
injection in services and tests without importing |
required |
allow_write
|
bool
|
Mutations are rejected unless this flag is explicitly true. |
False
|
device_id
|
str | None
|
Optional defaults used by :meth: |
None
|
configuration_id
|
str | None
|
Optional defaults used by :meth: |
None
|
site_id
|
str | None
|
Optional defaults used by :meth: |
None
|
include_device_properties
|
bool
|
Default for measured-quantity metadata in bundle reads. |
True
|
include_device_properties
instance-attribute
¶
metadata_mapping
instance-attribute
¶
metadata_mapping = metadata_mapping or SMSMetadataMappingConfig()
from_config
classmethod
¶
Build a client from validated Neptoon SMS configuration.
from_paths
classmethod
¶
from_paths(config_path, *, dotenv_path=None, allow_write=False, device_id=None, configuration_id=None, site_id=None, include_device_properties=True, metadata_mapping=None)
Initialize the official SMS client without making a remote request.
list_resources ¶
List resources through the official SMS JSON:API client.
list_all_resources ¶
List an entire SMS collection with bounded automatic pagination.
Existing filters are preserved while Neptoon owns page[number] and
page[size] for this call. The loop stops on an empty/short page or an
explicit JSON:API links.next value of null. Repeated full pages
and exhausted max_pages fail loudly instead of silently truncating.
list_all_configurations ¶
list_device_properties ¶
List measured-quantity metadata, optionally filtered to one device.
list_all_device_properties ¶
map_sensor_info ¶
Map Neptoon sensor_info into configured SMS attribute payloads.
This method is pure and never performs a remote request. Mapping keys are
SMS target attributes; mapping values name sensor_info fields.
Datetimes and other Pydantic-supported values are converted to JSON-safe
representations before they cross the integration boundary.
map_bundle_to_sensor_info ¶
Extract configured SMS attributes as explicit sensor_info updates.
The returned dictionary is local-only and does not mutate a Pydantic model.
If multiple SMS resources map to the same sensor_info field with
different values, the ambiguity is reported instead of choosing an
implicit resource precedence.
fetch_metadata_bundle ¶
fetch_metadata_bundle(*, device_id=None, configuration_id=None, site_id=None, include_device_properties=None)
Fetch the SMS metadata explicitly selected for a Neptoon sensor.
create_resource ¶
Create one SMS resource when remote writes were explicitly enabled.
update_resource ¶
Patch one complete JSON:API resource when writes are enabled.
update_attributes ¶
Patch selected attributes while preserving the current SMS resource.
delete_resource ¶
Delete one SMS resource when writes are enabled.
neptoon.external.dwd_data_collection¶
dwd_data_collection ¶
DWD_BASE_URL
module-attribute
¶
DWD_STATION_PRODUCTS_FILE
module-attribute
¶
DWD_VARIABLE_SPECS
module-attribute
¶
DWD_VARIABLE_SPECS = {'pressure': DWDVariableSpec(name='pressure', product_code='P0', availability_flag='has_pressure', output_column=str(AIR_PRESSURE), candidate_columns=('P', 'P0'), product_path='hourly/pressure/recent', zip_suffix='akt'), 'temperature': DWDVariableSpec(name='temperature', product_code='TU', availability_flag='has_temperature', output_column=str(AIR_TEMPERATURE), candidate_columns=('TT_TU', 'TT'), product_path='hourly/air_temperature/recent', zip_suffix='akt'), 'relative_humidity': DWDVariableSpec(name='relative_humidity', product_code='TU', availability_flag='has_relative_humidity', output_column=str(AIR_RELATIVE_HUMIDITY), candidate_columns=('RF_TU', 'RF'), product_path='hourly/air_temperature/recent', zip_suffix='akt'), 'precipitation': DWDVariableSpec(name='precipitation', product_code='RR', availability_flag='has_precipitation', output_column=str(PRECIPITATION), candidate_columns=('R1', 'RR_1', 'NIEDERSCHLAGSHOEHE'), product_path='hourly/precipitation/recent', zip_suffix='akt', value_semantics='interval_total', interval_duration='1h', interval_label='end', historical_product_path='hourly/precipitation/historical'), 'solar_radiation': DWDVariableSpec(name='solar_radiation', product_code='ST', availability_flag='has_global_radiation', output_column=str(SOLAR_RADIATION), candidate_columns=('FG_LBERG', 'FG_STRAHL', 'FG_STRAHLUNG', 'GLOBALSTRAHLUNG', 'GLOBAL_RADIATION', 'GS'), product_path='hourly/solar', zip_suffix='row')}
DWD_FALLBACK_VARIABLE_SPECS
module-attribute
¶
DWD_FALLBACK_VARIABLE_SPECS = {'precipitation': (DWDVariableSpec(name='precipitation_daily_more_precip', product_code='RR', availability_flag='has_precipitation', output_column=str(PRECIPITATION_DAILY), candidate_columns=('RSK', 'RS', 'R1', 'RR', 'NIEDERSCHLAGSHOEHE'), product_path='daily/more_precip/recent', zip_suffix='akt', zip_prefix='tageswerte', temporal_resolution='daily', historical_product_path='daily/more_precip/historical'), DWDVariableSpec(name='precipitation_daily_kl', product_code='KL', availability_flag='has_precipitation', output_column=str(PRECIPITATION_DAILY), candidate_columns=('RSK', 'RS', 'R1', 'RR', 'NIEDERSCHLAGSHOEHE'), product_path='daily/kl/recent', zip_suffix='akt', zip_prefix='tageswerte', temporal_resolution='daily', historical_product_path='daily/kl/historical'))}
DWD_PRECIPITATION_PRODUCT_SPECS
module-attribute
¶
DWD_PRECIPITATION_PRODUCT_SPECS = {'hourly': DWD_VARIABLE_SPECS['precipitation'], 'hourly_rr': DWD_VARIABLE_SPECS['precipitation'], 'daily': DWD_FALLBACK_VARIABLE_SPECS['precipitation'][0], 'daily_rr': DWD_FALLBACK_VARIABLE_SPECS['precipitation'][0], 'daily_more_precip': DWD_FALLBACK_VARIABLE_SPECS['precipitation'][0], 'daily_kl': DWD_FALLBACK_VARIABLE_SPECS['precipitation'][1]}
DWD_RECOVERABLE_ERRORS
module-attribute
¶
DWD_RECOVERABLE_ERRORS = (DWDDataError, FileNotFoundError, KeyError, OSError, ValueError, EmptyDataError, ParserError)
DWDVariableSpec
dataclass
¶
DWDVariableSpec(name, product_code, availability_flag, output_column, candidate_columns, product_path, zip_suffix, zip_prefix='stundenwerte', temporal_resolution='hourly', value_semantics='state', interval_duration=None, interval_label='start', historical_product_path=None, historical_zip_suffix='hist')
Mapping between a neptoon variable and the DWD product that supplies it.
get_zip_file_name ¶
Return the DWD ZIP filename for this product and station.
DWDDataError ¶
Bases: NeptoonExternalDataError, RuntimeError
Raised when DWD data cannot be collected or attached as requested.
DWDConfigurationError ¶
DWDInputDataError ¶
Bases: NeptoonInputDataError, ValueError
The local DataFrame supplied for DWD attachment is unusable.
DWDMetadataFileNotFoundError
dataclass
¶
Bases: DWDDataError, FileNotFoundError
The station-product metadata file required by DWD is missing.
DWDStationProductCatalog ¶
Station metadata and product availability for DWD weather stations.
metadata_path
instance-attribute
¶
metadata_path = DWD_STATION_PRODUCTS_FILE if metadata_path is None else Path(metadata_path)
get_stations_by_distance ¶
Return station metadata sorted by distance to the given point.
If variables is provided, stations are filtered to those whose
metadata says they provide all requested variables.
DWDDataFetcher ¶
Download and parse DWD ZIP products for one weather station.
fetch_spec_with_historical ¶
Fetch recent plus historical DWD files and combine their rows.
DWD recent precipitation files usually cover only the last weeks or
months. For multi-year CRNS runs this makes precipitation look almost
completely missing even though historical CDC ZIPs exist in the sibling
historical directory. This method first tries all historical ZIPs
for the station/product discovered from the directory listing and then
appends the recent ZIP. Recoverable errors in one part do not prevent
the other part from being used.
DWDDataAttacher ¶
DWDDataAttacher(data_frame, latitude, longitude, station='nearest', fill_pressure=True, fill_temperature=True, fill_relative_humidity=True, include_precipitation=False, include_solar_radiation=False, precipitation_daily_fallback=True, precipitation_product=None, prefer_dwd_values=False, max_nearest_timedelta='1h', max_distance_km=None, request_timeout=30, request_retries=0, cache_dir=None, use_cache=True, required=False, station_products_path=None)
Attach DWD products to a CRNS DataFrame.
precipitation_daily_fallback
instance-attribute
¶
precipitation_product
instance-attribute
¶
precipitation_product = normalize_precipitation_product(precipitation_product)
max_nearest_timedelta
instance-attribute
¶
max_nearest_timedelta = _to_positive_pandas_timedelta(max_nearest_timedelta, field_name='max_nearest_timedelta')
normalize_precipitation_product ¶
Normalize optional manual DWD precipitation product selection.
get_precipitation_specs_for_request ¶
Return DWD precipitation specs in the order they should be tried.
Without a manual product selection, hourly precipitation has priority. If hourly data are unavailable for the nearest usable station, daily products are tried next. A manual product selection restricts the resolver to that single product.
validate_dwd_variables ¶
Raise a clear error for unsupported DWD variable names.
normalize_dwd_station_id ¶
Normalize DWD station identifiers to five digits.
calculate_haversine_distance_array_km ¶
Return great-circle distances between WGS84 coordinates.
latitude_2 and longitude_2 may be scalars, NumPy arrays or
pandas Series. The function avoids row-wise DataFrame.apply calls when
ranking large station catalogs.
calculate_haversine_distance_km ¶
Return great-circle distance between two WGS84 coordinates.
parse_dwd_product_text ¶
Parse semicolon-separated DWD product text into a DatetimeIndex frame.
select_first_existing_column ¶
Return the first candidate column present in data_frame.
neptoon.external.international_meteo_data_collection¶
Compatibility import facade for the canonical provider modules below. Existing imports remain supported.
neptoon.external.meteo.attachment¶
attachment ¶
International meteorological provider implementation.
This internal module is re-exported through
neptoon.external.international_meteo_data_collection for compatibility.
ExternalMeteoDataAttacher ¶
Attach station-based public meteorological fallback data.
This adapter intentionally focuses on variables that NOAA ISD and Meteostat provide as true station meteorology. It does not synthesize global radiation from sunshine duration or modelled solar products.
Create an external meteo attacher.
Preferred usage is ExternalMeteoDataAttacher(data_frame, config).
Positional/keyword provider options are still accepted for backwards
compatibility with the previous long constructor.
temporal_semantics
instance-attribute
¶
temporal_semantics = {(str(variable)): (coerce_temporal_semantics(value))for (variable, value) in (items())}
include_snow_water_equivalent
instance-attribute
¶
max_nearest_timedelta
instance-attribute
¶
max_nearest_timedelta = _to_positive_pandas_timedelta(max_nearest_timedelta, field_name='max_nearest_timedelta')
effective_open_meteo_model
property
¶
Return the Open-Meteo archive model this provider will request.
serves_era5_land
property
¶
Return whether this configuration actually reads ERA5-Land.
neptoon.external.meteo.common¶
common ¶
EXTERNAL_METEO_SERVICE_LABEL
module-attribute
¶
NOAA_ISD_HISTORY_URL
module-attribute
¶
NOAA_GLOBAL_HOURLY_ACCESS_URL
module-attribute
¶
NOAA_GHCN_DAILY_BASE_URL
module-attribute
¶
NOAA_GHCN_STATIONS_URL
module-attribute
¶
NOAA_GHCN_STATIONS_URL = f'{NOAA_GHCN_DAILY_BASE_URL}/ghcnd-stations.txt'
NOAA_GHCN_INVENTORY_URL
module-attribute
¶
NOAA_GHCN_INVENTORY_URL = f'{NOAA_GHCN_DAILY_BASE_URL}/ghcnd-inventory.txt'
NOAA_USCRN_HOURLY_BASE_URL
module-attribute
¶
NOAA_USCRN_DAILY_BASE_URL
module-attribute
¶
NOAA_USCRN_STATION_LISTING_URL
module-attribute
¶
OPEN_METEO_ARCHIVE_URL
module-attribute
¶
METEOSWISS_STAC_COLLECTION_URL
module-attribute
¶
METEOSWISS_STAC_COLLECTION_URL = 'https://data.geo.admin.ch/api/stac/v1/collections/ch.meteoschweiz.ogd-smn/items'
KNMI_OPEN_DATA_BASE_URL
module-attribute
¶
MET_OFFICE_DATAHUB_BASE_URL
module-attribute
¶
METEO_FRANCE_DEFAULT_API_URL
module-attribute
¶
METEO_FRANCE_STATION_CATALOG_URL
module-attribute
¶
METEO_FRANCE_STATION_CATALOG_URL = 'https://www.data.gouv.fr/api/1/datasets/r/5ec141e4-3650-4365-82b1-db459efe690c'
ExternalMeteoConfigInput
module-attribute
¶
ExternalMeteoConfigInput = ExternalMeteoAttachmentConfig | Mapping[str, Any] | SupportsModelDump
EXTERNAL_METEO_RECOVERABLE_ERRORS
module-attribute
¶
EXTERNAL_METEO_RECOVERABLE_ERRORS = (ExternalMeteoDataError, FileNotFoundError, KeyError, OSError, UnicodeError, ValueError, JSONDecodeError, RequestException, Error, EmptyDataError, ParserError)
EXTERNAL_METEO_VARIABLE_SPECS
module-attribute
¶
EXTERNAL_METEO_VARIABLE_SPECS = {'pressure': ExternalMeteoVariableSpec(name='pressure', output_column=str(AIR_PRESSURE), provider_column='pressure_hpa'), 'temperature': ExternalMeteoVariableSpec(name='temperature', output_column=str(AIR_TEMPERATURE), provider_column='temperature_c'), 'relative_humidity': ExternalMeteoVariableSpec(name='relative_humidity', output_column=str(AIR_RELATIVE_HUMIDITY), provider_column='relative_humidity'), 'precipitation': ExternalMeteoVariableSpec(name='precipitation', output_column=str(PRECIPITATION), provider_column='precipitation_mm', value_semantics='interval_total', interval_duration='1h'), 'solar_radiation': ExternalMeteoVariableSpec(name='solar_radiation', output_column=str(SOLAR_RADIATION), provider_column='solar_radiation'), 'wind_speed': ExternalMeteoVariableSpec(name='wind_speed', output_column=str(WIND_SPEED), provider_column='wind_speed_m_s'), 'reference_et0': ExternalMeteoVariableSpec(name='reference_et0', output_column=str(REFERENCE_ET0), provider_column='reference_et0_mm', value_semantics='interval_total', interval_duration='1h'), None: {(variable): (ExternalMeteoVariableSpec(name=variable, output_column=str(column), provider_column=provider_column))for layer in ERA5_LAND_SOIL_LAYERS}, None: {(variable): (ExternalMeteoVariableSpec(name=variable, output_column=str(column), provider_column=provider_column))for snow in ERA5_LAND_SNOW_VARIABLES}}
EXTERNAL_METEO_DAILY_VARIABLE_SPECS
module-attribute
¶
EXTERNAL_METEO_DAILY_VARIABLE_SPECS = {None: EXTERNAL_METEO_VARIABLE_SPECS, 'pressure': ExternalMeteoVariableSpec(name='pressure', output_column=str(AIR_PRESSURE), provider_column='pressure_hpa', temporal_resolution='daily'), 'precipitation': ExternalMeteoVariableSpec(name='precipitation', output_column=str(PRECIPITATION_DAILY), provider_column='precipitation_mm', temporal_resolution='daily'), 'temperature': ExternalMeteoVariableSpec(name='temperature', output_column=str(AIR_TEMPERATURE), provider_column='temperature_c', temporal_resolution='daily'), 'relative_humidity': ExternalMeteoVariableSpec(name='relative_humidity', output_column=str(AIR_RELATIVE_HUMIDITY), provider_column='relative_humidity', temporal_resolution='daily'), 'solar_radiation': ExternalMeteoVariableSpec(name='solar_radiation', output_column=str(SOLAR_RADIATION), provider_column='solar_radiation', temporal_resolution='daily'), 'wind_speed': ExternalMeteoVariableSpec(name='wind_speed', output_column=str(WIND_SPEED), provider_column='wind_speed_m_s', temporal_resolution='daily')}
OPEN_METEO_VARIABLE_MAP
module-attribute
¶
OPEN_METEO_VARIABLE_MAP = {'pressure': ('surface_pressure', 'pressure_hpa'), 'temperature': ('temperature_2m', 'temperature_c'), 'relative_humidity': ('relative_humidity_2m', 'relative_humidity'), 'precipitation': ('precipitation', 'precipitation_mm'), 'solar_radiation': ('shortwave_radiation', 'solar_radiation'), 'wind_speed': ('wind_speed_10m', 'wind_speed_m_s'), 'reference_et0': ('et0_fao_evapotranspiration', 'reference_et0_mm'), None: {(variable): (api_variable, provider_column)for layer in ERA5_LAND_SOIL_LAYERS}, None: {(variable): (api_variable, provider_column)for snow in ERA5_LAND_SNOW_VARIABLES}}
EOBS_VARIABLE_CANDIDATES
module-attribute
¶
EOBS_VARIABLE_CANDIDATES = {'precipitation': ('rr', 'precipitation', 'precipitation_amount'), 'temperature': ('tg', 'temperature', 'mean_temperature'), 'relative_humidity': ('hu', 'relative_humidity', 'humidity'), 'solar_radiation': ('qq', 'global_radiation', 'solar_radiation')}
EOBS_PROVIDER_COLUMNS
module-attribute
¶
EOBS_PROVIDER_COLUMNS = {'precipitation': 'precipitation_mm', 'temperature': 'temperature_c', 'relative_humidity': 'relative_humidity', 'solar_radiation': 'solar_radiation'}
METEOSWISS_VARIABLE_CANDIDATES
module-attribute
¶
METEOSWISS_VARIABLE_CANDIDATES = {'pressure': ('prestas0', 'pressure_hpa', 'pressure', 'air_pressure'), 'temperature': ('tre200s0', 'temperature_c', 'temperature', 'air_temperature'), 'relative_humidity': ('ure200s0', 'relative_humidity', 'humidity'), 'precipitation': ('rre150h0', 'rre150z0', 'precipitation_mm', 'precipitation'), 'solar_radiation': ('gre000h0', 'gre000z0', 'solar_radiation', 'global_radiation')}
METEOSWISS_PROVIDER_COLUMNS
module-attribute
¶
METEOSWISS_PROVIDER_COLUMNS = {'pressure': 'pressure_hpa', 'temperature': 'temperature_c', 'relative_humidity': 'relative_humidity', 'precipitation': 'precipitation_mm', 'solar_radiation': 'solar_radiation'}
KNMI_VARIABLE_CANDIDATES
module-attribute
¶
KNMI_VARIABLE_CANDIDATES = {'pressure': ('P', 'PG', 'pressure_hpa', 'pressure'), 'temperature': ('T', 'TG', 'temperature_c', 'temperature'), 'relative_humidity': ('U', 'UG', 'relative_humidity', 'humidity'), 'precipitation': ('RH', 'precipitation_mm', 'precipitation'), 'solar_radiation': ('Q', 'solar_radiation', 'global_radiation')}
KNMI_PROVIDER_COLUMNS
module-attribute
¶
KNMI_PROVIDER_COLUMNS = {'pressure': 'pressure_hpa', 'temperature': 'temperature_c', 'relative_humidity': 'relative_humidity', 'precipitation': 'precipitation_mm', 'solar_radiation': 'solar_radiation'}
MET_OFFICE_VARIABLE_CANDIDATES
module-attribute
¶
MET_OFFICE_VARIABLE_CANDIDATES = {'pressure': ('pressure_hpa', 'pressure', 'mslp', 'screenPressure'), 'temperature': ('temperature_c', 'temperature', 'screenTemperature', 'airTemperature'), 'relative_humidity': ('relative_humidity', 'humidity', 'screenRelativeHumidity'), 'precipitation': ('precipitation_mm', 'precipitation', 'totalPrecipitation', 'precipitationAmount')}
MET_OFFICE_PROVIDER_COLUMNS
module-attribute
¶
MET_OFFICE_PROVIDER_COLUMNS = {'pressure': 'pressure_hpa', 'temperature': 'temperature_c', 'relative_humidity': 'relative_humidity', 'precipitation': 'precipitation_mm'}
METEO_FRANCE_VARIABLE_CANDIDATES
module-attribute
¶
METEO_FRANCE_VARIABLE_CANDIDATES = {'pressure': ('pressure_hpa', 'pression_hpa', 'pression_station_hpa', 'pmer_hpa', 'pstat_hpa', 'pressure', 'pression', 'P', 'PMER', 'PSTAT', 'PRES'), 'temperature': ('temperature_c', 'temperature', 'air_temperature', 'temperature_air', 'temperature_2m', 'T', 'TEMP', 'T2M'), 'relative_humidity': ('relative_humidity', 'humidity', 'humidite', 'humidite_relative', 'U', 'RH', 'HUM'), 'precipitation': ('RR1', 'RR', 'RR3', 'RR6', 'RR12', 'RR24', 'precipitation_mm', 'precipitation', 'pluie'), 'solar_radiation': ('solar_radiation', 'global_radiation', 'rayonnement_global', 'GLO', 'RAY_GLO01', 'RAYGLO')}
METEO_FRANCE_PROVIDER_COLUMNS
module-attribute
¶
METEO_FRANCE_PROVIDER_COLUMNS = {'pressure': 'pressure_hpa', 'temperature': 'temperature_c', 'relative_humidity': 'relative_humidity', 'precipitation': 'precipitation_mm', 'solar_radiation': 'solar_radiation'}
FMI_VARIABLE_CANDIDATES
module-attribute
¶
FMI_VARIABLE_CANDIDATES = {'pressure': ('pressure_hpa', 'pressure', 'p_sea', 'psea', 'air_pressure', 'press'), 'temperature': ('temperature_c', 'temperature', 't2m', 't', 'air_temperature'), 'relative_humidity': ('relative_humidity', 'humidity', 'rh', 'relativehumidity'), 'precipitation': ('r_1h', 'precipitation1h', 'prd_pt1h_acc', 'precipitation_mm', 'precipitation', 'rr', 'rain'), 'solar_radiation': ('solar_radiation', 'global_radiation', 'g', 'ghi', 'shortwave_radiation')}
FMI_PROVIDER_COLUMNS
module-attribute
¶
FMI_PROVIDER_COLUMNS = {'pressure': 'pressure_hpa', 'temperature': 'temperature_c', 'relative_humidity': 'relative_humidity', 'precipitation': 'precipitation_mm', 'solar_radiation': 'solar_radiation'}
ECCC_VARIABLE_CANDIDATES
module-attribute
¶
ECCC_VARIABLE_CANDIDATES = {'pressure': ('pressure_hpa', 'station_pressure', 'stn_pres', 'pressure', 'mean_sea_level_pressure'), 'temperature': ('temperature_c', 'air_temp', 'temperature', 'temp', 'air_temperature'), 'relative_humidity': ('relative_humidity', 'rel_hum', 'humidity', 'relative_humidity_percent'), 'precipitation': ('precipitation_mm', 'precipitation', 'precip_amount', 'total_precipitation', 'rain'), 'solar_radiation': ('solar_radiation', 'global_radiation', 'ghi', 'shortwave_radiation')}
ECCC_PROVIDER_COLUMNS
module-attribute
¶
ECCC_PROVIDER_COLUMNS = {'pressure': 'pressure_hpa', 'temperature': 'temperature_c', 'relative_humidity': 'relative_humidity', 'precipitation': 'precipitation_mm', 'solar_radiation': 'solar_radiation'}
BOM_VARIABLE_CANDIDATES
module-attribute
¶
BOM_VARIABLE_CANDIDATES = {'pressure': ('pressure_hpa', 'press', 'press_qnh', 'pressure', 'mslp'), 'temperature': ('temperature_c', 'air_temp', 'apparent_t', 'temperature', 'temp'), 'relative_humidity': ('relative_humidity', 'rel_hum', 'humidity'), 'precipitation': ('precipitation_mm', 'rain_trace', 'rain_since_9am', 'precipitation', 'rain'), 'solar_radiation': ('solar_radiation', 'global_radiation', 'solar_exposure', 'ghi'), 'wind_speed': ('wind_speed_m_s', 'wind_spd_kmh', 'wind_spd_kt', 'wind_speed')}
BOM_PROVIDER_COLUMNS
module-attribute
¶
BOM_PROVIDER_COLUMNS = {'pressure': 'pressure_hpa', 'temperature': 'temperature_c', 'relative_humidity': 'relative_humidity', 'precipitation': 'precipitation_mm', 'solar_radiation': 'solar_radiation', 'wind_speed': 'wind_speed_m_s'}
GHCN_DAILY_VARIABLE_ELEMENTS
module-attribute
¶
GHCN_DAILY_VARIABLE_ELEMENTS = {'precipitation': ('PRCP',), 'temperature': ('TAVG', 'TMAX', 'TMIN'), 'relative_humidity': ('RHAV',), 'pressure': ('ASTP', 'ASLP')}
USCRN_HOURLY_COLUMNS
module-attribute
¶
USCRN_HOURLY_COLUMNS = ['WBANNO', 'UTC_DATE', 'UTC_TIME', 'LST_DATE', 'LST_TIME', 'CRX_VN', 'LONGITUDE', 'LATITUDE', 'T_CALC', 'T_HR_AVG', 'T_MAX', 'T_MIN', 'P_CALC', 'SOLARAD', 'SOLARAD_FLAG', 'SOLARAD_MAX', 'SOLARAD_MAX_FLAG', 'SOLARAD_MIN', 'SOLARAD_MIN_FLAG', 'SUR_TEMP_TYPE', 'SUR_TEMP', 'SUR_TEMP_FLAG', 'SUR_TEMP_MAX', 'SUR_TEMP_MAX_FLAG', 'SUR_TEMP_MIN', 'SUR_TEMP_MIN_FLAG', 'RH_HR_AVG', 'RH_HR_AVG_FLAG', 'SOIL_MOISTURE_5', 'SOIL_MOISTURE_10', 'SOIL_MOISTURE_20', 'SOIL_MOISTURE_50', 'SOIL_MOISTURE_100', 'SOIL_TEMP_5', 'SOIL_TEMP_10', 'SOIL_TEMP_20', 'SOIL_TEMP_50', 'SOIL_TEMP_100']
USCRN_DAILY_COLUMNS
module-attribute
¶
USCRN_DAILY_COLUMNS = ['WBANNO', 'LST_DATE', 'CRX_VN', 'LONGITUDE', 'LATITUDE', 'T_DAILY_MAX', 'T_DAILY_MIN', 'T_DAILY_MEAN', 'T_DAILY_AVG', 'P_DAILY_CALC', 'SOLARAD_DAILY', 'SUR_TEMP_DAILY_TYPE', 'SUR_TEMP_DAILY_MAX', 'SUR_TEMP_DAILY_MIN', 'SUR_TEMP_DAILY_AVG', 'RH_DAILY_MAX', 'RH_DAILY_MIN', 'RH_DAILY_AVG', 'SOIL_MOISTURE_5_DAILY', 'SOIL_MOISTURE_10_DAILY', 'SOIL_MOISTURE_20_DAILY', 'SOIL_MOISTURE_50_DAILY', 'SOIL_MOISTURE_100_DAILY', 'SOIL_TEMP_5_DAILY', 'SOIL_TEMP_10_DAILY', 'SOIL_TEMP_20_DAILY', 'SOIL_TEMP_50_DAILY', 'SOIL_TEMP_100_DAILY']
ExternalMeteoConfigurationError ¶
Bases: NeptoonConfigurationError, ValueError
Invalid user-facing external meteorology configuration.
ExternalMeteoConfigurationTypeError
dataclass
¶
Bases: ExternalMeteoConfigurationError, TypeError
Invalid external-meteorology configuration object shape.
ExternalMeteoDataError ¶
Bases: NeptoonExternalDataError, RuntimeError
Raised when non-DWD external meteorological data cannot be attached.
SupportsModelDump ¶
Bases: Protocol
Protocol for Pydantic-like configuration objects.
ExternalMeteoScalingPolicy
dataclass
¶
Bases: ScalingPolicy
Adjustment of an ERA5-Land reference series inside the meteo block.
The mechanics are shared with the remote-sensing block because reanalysis soil moisture and satellite soil moisture need exactly the same treatment. Only the configuration path differs, so an error message names the key the user wrote.
ExternalMeteoAttachmentConfig
dataclass
¶
ExternalMeteoAttachmentConfig(provider, latitude, longitude, station='nearest', product='auto', fill_pressure=False, fill_temperature=False, fill_relative_humidity=False, include_precipitation=True, include_solar_radiation=False, include_wind_speed=False, include_reference_et0=False, temporal_semantics=None, include_soil_moisture=False, soil_moisture_layers=None, soil_moisture_scaling=None, include_snow_depth=False, include_snow_water_equivalent=False, prefer_external_values=False, max_nearest_timedelta='1h', max_distance_km=None, request_timeout=30, cache_dir=None, use_cache=True, required=False, aemet_api_key=None, aemet_api_key_env=None, aemet_base_url=None, nasa_power_base_url=None, noaa_station_history_path=None, noaa_ghcn_stations_path=None, noaa_ghcn_inventory_path=None, noaa_uscrn_station_catalog_url=None, noaa_uscrn_station_catalog_path=None, nws_base_url=None, nws_user_agent=None, nws_file_paths=None, nws_station_catalog_path=None, inmet_base_url=None, inmet_file_paths=None, inmet_station_catalog_path=None, jma_base_url=None, jma_file_paths=None, jma_station_catalog_path=None, meteostat_station_db_path=None, station_candidate_limit=25, open_meteo_model=None, eobs_data_dir=None, eobs_file_paths=None, meteoswiss_file_paths=None, meteoswiss_data_url=None, meteoswiss_stac_collection_url=None, knmi_api_key=None, knmi_api_key_env=None, knmi_dataset_name=None, knmi_dataset_version=None, knmi_file_paths=None, knmi_max_files=200, met_office_api_key=None, met_office_api_key_env=None, met_office_api_url=None, met_office_file_paths=None, meteo_france_api_key=None, meteo_france_api_key_env=None, meteo_france_api_url=None, meteo_france_file_paths=None, meteo_france_station_catalog_url=None, meteo_france_station_catalog_path=None, chmi_file_paths=None, chmi_station_catalog_path=None, chmi_station_catalog_url=None, chmi_base_url=None, chmi_allowed_quality_values=None, chmi_allowed_flag_values=None, arso_file_paths=None, arso_station_catalog_path=None, arso_station_catalog_url=None, arso_base_url=None, arso_allowed_state_values=None, smhi_base_url=None, smhi_file_paths=None, smhi_period=None, imgw_data_url=None, imgw_file_paths=None, imgw_station_catalog_url=None, imgw_station_catalog_path=None, met_eireann_base_url=None, met_eireann_collection_id=None, met_eireann_file_paths=None, met_eireann_station_catalog_path=None, rmi_belgium_wfs_url=None, vmm_waterinfo_url=None, vmm_waterinfo_pressure_group_id=None, vmm_waterinfo_temperature_group_id=None, vmm_waterinfo_relative_humidity_group_id=None, vmm_waterinfo_precipitation_group_id=None, fmi_data_url=None, fmi_file_paths=None, fmi_stored_query_id='fmi::observations::weather::timevaluepair', fmi_parameters=None, ipma_data_url=None, ipma_file_paths=None, ipma_station_catalog_url=None, ipma_station_catalog_path=None, eccc_api_url=None, eccc_file_paths=None, eccc_collection_id=None, bom_data_url=None, bom_file_paths=None, bom_station_catalog_url=None, bom_station_catalog_path=None, bom_product_id=None, geosphere_austria_base_url=None, geosphere_austria_resource_id=None, geosphere_austria_file_paths=None, geosphere_austria_station_catalog_path=None, frost_client_id=None, frost_client_id_env=None, frost_base_url=None, frost_file_paths=None, frost_source_file=None, dmi_base_url=None, dmi_file_paths=None, dmi_station_catalog_path=None, dwd_request_retries=3, dwd_station_products_path=None, dwd_precipitation_daily_fallback=True, dwd_precipitation_product=None)
Runtime configuration for attaching one external meteo provider.
This object is the typed boundary between YAML/Pydantic configuration, the public CRNSDataHub API, and the provider-specific attacher. It keeps the long provider option set in one place while preserving backwards compatibility for callers that still pass keyword arguments.
include_snow_water_equivalent
class-attribute
instance-attribute
¶
noaa_uscrn_station_catalog_url
class-attribute
instance-attribute
¶
noaa_uscrn_station_catalog_path
class-attribute
instance-attribute
¶
meteoswiss_stac_collection_url
class-attribute
instance-attribute
¶
meteo_france_station_catalog_url
class-attribute
instance-attribute
¶
meteo_france_station_catalog_path
class-attribute
instance-attribute
¶
met_eireann_station_catalog_path
class-attribute
instance-attribute
¶
vmm_waterinfo_pressure_group_id
class-attribute
instance-attribute
¶
vmm_waterinfo_temperature_group_id
class-attribute
instance-attribute
¶
vmm_waterinfo_relative_humidity_group_id
class-attribute
instance-attribute
¶
vmm_waterinfo_precipitation_group_id
class-attribute
instance-attribute
¶
fmi_stored_query_id
class-attribute
instance-attribute
¶
geosphere_austria_resource_id
class-attribute
instance-attribute
¶
geosphere_austria_file_paths
class-attribute
instance-attribute
¶
geosphere_austria_station_catalog_path
class-attribute
instance-attribute
¶
dwd_precipitation_daily_fallback
class-attribute
instance-attribute
¶
variables
property
¶
Return logical meteorological variables enabled by this config.
The property is intentionally derived from the public boolean options so reporting and callers cannot drift from the provider attacher's variable selection.
era5_land_variables
property
¶
Return the ERA5-Land reference variables this config requests.
get_field_names
classmethod
¶
Return dataclass field names for mapping/object coercion.
from_config
classmethod
¶
Build from a Pydantic config, mapping, or compatible object.
ExternalMeteoConfig intentionally does not carry site coordinates;
the workflow injects coordinates from sensor_info here.
ExternalMeteoVariableSpec
dataclass
¶
ExternalMeteoVariableSpec(name, output_column, provider_column, temporal_resolution='hourly', value_semantics='state', interval_duration=None)
Mapping between a logical meteo variable and a Neptoon output column.
coerce_external_meteo_attachment_config ¶
Coerce public API input into ExternalMeteoAttachmentConfig.
Keyword arguments are kept for backwards compatibility with the old long signatures. When both an object and legacy keyword options are supplied, the keyword options override object values.
neptoon.external.meteo.providers.base¶
Shared request and adapter contracts for external meteorological providers.
base ¶
Shared types for external meteorological provider adapters.
ProviderAdapterFn
module-attribute
¶
ProviderAdapterFn = Callable[[ExternalMeteoProviderHost, ProviderRequest], bool]
ProviderRequest
dataclass
¶
ExternalMeteoProviderHost ¶
Bases: Protocol
Structural host contract consumed by provider adapter functions.
neptoon.external.meteo.providers.registry¶
Central provider-to-adapter dispatch registry.
registry ¶
Explicit registry from provider names to attachment adapter callables.
CATALOG_PROVIDER_NAMES
module-attribute
¶
DIRECT_PROVIDER_ADAPTERS
module-attribute
¶
DIRECT_PROVIDER_ADAPTERS = {'aemet': attach_aemet_provider, 'nasa_power': attach_nasa_power_provider, 'open_meteo': attach_open_meteo_provider, 'era5_open_meteo': attach_open_meteo_provider, 'era5_land_open_meteo': attach_open_meteo_provider, 'eobs': attach_eobs_provider, 'noaa_nws': attach_nws_provider, 'inmet': attach_inmet_provider, 'jma_amedas': attach_jma_provider, 'meteoswiss': attach_meteoswiss_provider, 'knmi': attach_knmi_provider, 'fmi': attach_fmi_provider, 'ipma': attach_ipma_provider, 'eccc_geomet': attach_eccc_provider, 'bom': attach_bom_provider, 'geosphere_austria': attach_geosphere_provider, 'frost': attach_frost_provider, 'dmi': attach_dmi_provider, 'dwd': _attach_dwd_provider, 'met_office_datahub': attach_met_office_provider, 'smhi': attach_smhi_provider, 'imgw': attach_imgw_provider, 'met_eireann': attach_met_eireann_provider, 'chmi': attach_chmi_provider, 'arso': attach_arso_provider, 'meteo_france': attach_meteo_france_provider, 'rmi_belgium': attach_rmi_provider, 'vmm_waterinfo': attach_vmm_provider}
get_provider_adapter ¶
Return the adapter for one normalized provider name.
neptoon.external.meteo.providers.gridded¶
Adapters for gridded meteorological data sources.
gridded ¶
neptoon.external.meteo.providers.catalog¶
Adapters and helpers for providers backed by station catalogs.
catalog ¶
neptoon.external.meteo.providers.national¶
Adapters for national meteorological services.
national ¶
Adapters for national meteorological services.
neptoon.external.meteo.providers.station¶
Adapters for direct station-oriented provider APIs.
station ¶
Adapters for station APIs with built-in nearest-station selection.
neptoon.external.meteo.aemet¶
aemet ¶
AEMET OpenData daily climatological station observations.
AEMET_OPEN_DATA_BASE_URL
module-attribute
¶
AEMETDataFetcher ¶
AEMETDataFetcher(latitude, longitude, station='nearest', *, api_key, base_url=None, max_distance_km=None, station_candidate_limit=25, cache_dir=None, request_timeout=30, use_cache=True)
Fetch AEMET OpenData daily climatology with nearest-station selection.
parse_aemet_station_catalog ¶
Parse AEMET climatological station inventory JSON.
parse_aemet_daily_observations ¶
Parse AEMET daily climatological observations into canonical columns.
neptoon.external.meteo.nasa_power¶
nasa_power ¶
NASA POWER hourly point meteorology provider.
NASA_POWER_HOURLY_POINT_URL
module-attribute
¶
NASA_POWER_VARIABLE_MAP
module-attribute
¶
NASA_POWER_VARIABLE_MAP = {'pressure': ('PS', 'pressure_hpa'), 'temperature': ('T2M', 'temperature_c'), 'relative_humidity': ('RH2M', 'relative_humidity'), 'precipitation': ('PRECTOTCORR', 'precipitation_mm'), 'solar_radiation': ('ALLSKY_SFC_SW_DWN', 'solar_radiation'), 'wind_speed': ('WS10M', 'wind_speed_m_s')}
NASAPowerDataFetcher ¶
NASAPowerDataFetcher(latitude, longitude, *, base_url=None, cache_dir=None, request_timeout=30, use_cache=True)
Fetch hourly NASA POWER point meteorology in UTC.
parse_nasa_power_hourly_response ¶
Parse a NASA POWER hourly point response into canonical Neptoon columns.
neptoon.external.meteo.noaa¶
noaa ¶
International meteorological provider implementation.
This internal module is re-exported through
neptoon.external.international_meteo_data_collection for compatibility.
NOAAISDStationCatalog ¶
NOAAISDDataFetcher ¶
Download and parse NOAA ISD Global Hourly station CSV files.
NOAAGHCNDailyStationCatalog ¶
NOAAGHCNDailyStationCatalog(stations_path=None, inventory_path=None, cache_dir=None, request_timeout=30, use_cache=True)
Station catalog for NOAA GHCN-Daily metadata and element inventory.
get_stations_by_distance ¶
get_stations_by_distance(latitude, longitude, start, end, variables=(), max_distance_km=None, limit=25)
NOAAGHCNDailyDataFetcher ¶
Download and parse a single NOAA GHCN-Daily station .dly file.
NOAAUSCRNStationCatalog ¶
NOAAUSCRNStationCatalog(catalog_path=None, catalog_url=None, cache_dir=None, request_timeout=30, use_cache=True)
Station catalog for NOAA USCRN/USRCRN product files.
catalog_url
instance-attribute
¶
catalog_url = strip() if catalog_url else NOAA_USCRN_STATION_LISTING_URL
get_stations_by_distance ¶
get_stations_by_distance(latitude, longitude, product, start, end, variables=(), max_distance_km=None, limit=25)
NOAAUSCRNDataFetcher ¶
NOAAUSCRNDataFetcher(station_file_name, product='hourly', cache_dir=None, request_timeout=30, use_cache=True)
Download and parse NOAA USCRN station hourly or daily text files.
station_file_name
instance-attribute
¶
parse_noaa_isd_global_hourly_csv ¶
Parse NOAA Global Hourly CSV into Neptoon-oriented columns.
Returned columns are a stable subset: temperature_c, pressure_hpa, relative_humidity and precipitation_mm. Missing or unsupported source fields are left absent.
parse_noaa_ghcn_daily_stations ¶
Parse ghcnd-stations.txt fixed-width station metadata.
parse_noaa_ghcn_daily_inventory ¶
Parse ghcnd-inventory.txt fixed-width element availability.
parse_noaa_ghcn_daily_dly ¶
Parse a NOAA GHCN-Daily .dly station file into provider columns.
parse_noaa_uscrn_station_catalog ¶
Parse USCRN station metadata from CSV/TSV or NOAA's station-listing page.
parse_noaa_uscrn_hourly_text ¶
Parse NOAA USCRN Hourly02 station text into provider columns.
parse_noaa_uscrn_daily_text ¶
Parse NOAA USCRN Daily01 station text into provider columns.
neptoon.external.meteo.meteostat¶
meteostat ¶
International meteorological provider implementation.
This internal module is re-exported through
neptoon.external.international_meteo_data_collection for compatibility.
MeteostatStationCatalog ¶
MeteostatDataFetcher ¶
Download and parse Meteostat hourly bulk CSV files.
parse_meteostat_hourly_csv ¶
Parse a Meteostat hourly bulk CSV dump into Neptoon-oriented columns.
neptoon.external.meteo.open_meteo¶
open_meteo ¶
International meteorological provider implementation.
This internal module is re-exported through
neptoon.external.international_meteo_data_collection for compatibility.
OPEN_METEO_ARCHIVE_MODELS
module-attribute
¶
OPEN_METEO_ARCHIVE_MODELS = frozenset({'best_match', 'ecmwf_ifs', 'ecmwf_ifs_analysis_long_window', 'era5_seamless', 'era5', 'era5_land', 'era5_ensemble', 'cerra'})
OpenMeteoDataFetcher ¶
OpenMeteoDataFetcher(latitude, longitude, cache_dir=None, request_timeout=30, use_cache=True, model=None)
Download and parse Open-Meteo Historical Weather API data.
parse_open_meteo_archive_response ¶
Parse Open-Meteo archive JSON into Neptoon-oriented columns.
extract_open_meteo_grid_cell ¶
Return the grid cell Open-Meteo served, where the payload reports it.
neptoon.external.meteo.chmi¶
chmi ¶
Czech Hydrometeorological Institute (CHMI/CHMU) Open Data adapter.
The adapter reads token-free station data published at opendata.chmi.cz.
CHMI observation files use a compact long format with station, element,
timestamp, value and optional quality fields. Neptoon combines 10-minute and
hourly products into its canonical hourly variables and can aggregate those to
daily values.
CHMI_OPEN_DATA_BASE_URL
module-attribute
¶
CHMI_STATION_CATALOG_URL
module-attribute
¶
CHMI_STATION_CATALOG_URL = f'{CHMI_OPEN_DATA_BASE_URL}/historical_csv/metadata/meta1.csv'
CHMIStationCatalog ¶
CHMIStationCatalog(*, catalog_path=None, catalog_url=None, cache_dir=None, request_timeout=30, use_cache=True)
CHMIDataFetcher ¶
CHMIDataFetcher(*, latitude, longitude, station='nearest', product='hourly', file_paths=None, station_catalog_path=None, station_catalog_url=None, base_url=None, allowed_quality_values=None, allowed_flag_values=None, max_distance_km=None, station_candidate_limit=25, cache_dir=None, request_timeout=30, use_cache=True)
parse_chmi_station_catalog ¶
Parse CHMI station metadata into canonical station fields.
Returned columns are station_id, station_name, longitude,
latitude, elevation_m and begin_date when available.
parse_chmi_observations ¶
Parse one CHMI observation product to canonical Neptoon columns.
Unknown CHMI elements are ignored. Quality semantics are intentionally not
inferred: by default every finite observation is retained. Pass explicit
allowlists to filter QUALITY and/or FLAG for a reproducible local policy.
neptoon.external.meteo.arso¶
arso ¶
Slovenian Environment Agency (ARSO / meteo.si) meteorology adapter.
ARSO publishes token-free automatic-station observations as XML under
meteo.arso.gov.si. The public per-station history product is a rolling
roughly 48-hour window, while the all-station latest XML provides the station
metadata needed for nearest-station selection.
The adapter maps interval/representative observations to Neptoon's canonical external-meteorology columns and aggregates the native sub-hourly observations to hourly or daily values. Local XML files are supported for reproducible or longer-history workflows.
ARSO_BASE_URL
module-attribute
¶
ARSO_STATION_CATALOG_URL
module-attribute
¶
ARSO_STATION_CATALOG_URL = f'{ARSO_BASE_URL}/observationAms_si_latest.xml'
ARSOStationCatalog ¶
ARSOStationCatalog(*, catalog_path=None, catalog_url=None, cache_dir=None, request_timeout=30, use_cache=True)
ARSODataFetcher ¶
ARSODataFetcher(*, latitude, longitude, station='nearest', product='hourly', file_paths=None, station_catalog_path=None, station_catalog_url=None, base_url=None, allowed_state_values=None, max_distance_km=None, station_candidate_limit=25, cache_dir=None, request_timeout=30, use_cache=True)
parse_arso_station_catalog ¶
Parse ARSO automatic-station XML to canonical station metadata.
The all-station observationAms_si_latest.xml file is the normal source,
but a per-station history XML is accepted as well. Returned station ids are
ARSO's stable domain_meteosiId values.
parse_arso_observations ¶
Parse ARSO automatic-station XML to canonical provider columns.
allowed_state_values is deliberately opt-in. ARSO publishes per-field
*_state values, but Neptoon does not guess their quality semantics. If
an allowlist is supplied, a value is retained only when its corresponding
state field is present in that allowlist.
neptoon.external.meteo.era5_land¶
era5_land ¶
ERA5-Land soil moisture and snow, served through the Open-Meteo archive.
ERA5-Land is reanalysis of the land surface, not remote sensing, so it
reaches Neptoon through the external_meteo block rather than through
remote_sensing:. It is the same provider, the same endpoint and the same
:class:~neptoon.external.meteo.open_meteo.OpenMeteoDataFetcher that already
supports models=era5_land; exposing soil moisture and snow is an extension
of the variable map and the per-variable flags rather than a second adapter.
The column names carry the origin (era5_land_soil_moisture_7_28cm), so a
reader months later cannot mistake a reanalysis for a measurement.
The module holds three things the rest of the meteo block should not have to know: the native soil layers and which one is the sensible default, the coverage descriptor that lets the shared series-coverage check answer "this is open water", and the measured record of what this route does not serve.
ERA5_LAND_SOIL_LAYERS
module-attribute
¶
ERA5_LAND_SOIL_LAYERS = (ERA5LandSoilLayer(key='0_7cm', variable='soil_moisture_0_7cm', api_variable='soil_moisture_0_to_7cm', provider_column='era5_land_soil_moisture_0_7cm', column=ERA5_LAND_SOIL_MOISTURE_0_7CM, top_cm=0.0, bottom_cm=7.0), ERA5LandSoilLayer(key='7_28cm', variable='soil_moisture_7_28cm', api_variable='soil_moisture_7_to_28cm', provider_column='era5_land_soil_moisture_7_28cm', column=ERA5_LAND_SOIL_MOISTURE_7_28CM, top_cm=7.0, bottom_cm=28.0), ERA5LandSoilLayer(key='28_100cm', variable='soil_moisture_28_100cm', api_variable='soil_moisture_28_to_100cm', provider_column='era5_land_soil_moisture_28_100cm', column=ERA5_LAND_SOIL_MOISTURE_28_100CM, top_cm=28.0, bottom_cm=100.0), ERA5LandSoilLayer(key='100_255cm', variable='soil_moisture_100_255cm', api_variable='soil_moisture_100_to_255cm', provider_column='era5_land_soil_moisture_100_255cm', column=ERA5_LAND_SOIL_MOISTURE_100_255CM, top_cm=100.0, bottom_cm=255.0))
ERA5_LAND_SOIL_LAYERS_BY_KEY
module-attribute
¶
ERA5_LAND_SOIL_LAYERS_BY_KEY = {(key): _adZhfor layer in ERA5_LAND_SOIL_LAYERS}
ERA5_LAND_SOIL_LAYERS_BY_VARIABLE
module-attribute
¶
ERA5_LAND_SOIL_LAYERS_BY_VARIABLE = {(variable): _goyxfor layer in ERA5_LAND_SOIL_LAYERS}
ERA5_LAND_SNOW_DEPTH
module-attribute
¶
ERA5_LAND_SNOW_DEPTH = ERA5LandSnowVariable(variable='snow_depth', api_variable='snow_depth', provider_column='era5_land_snow_depth', column=ERA5_LAND_SNOW_DEPTH, unit='m')
ERA5_LAND_SNOW_WATER_EQUIVALENT
module-attribute
¶
ERA5_LAND_SNOW_WATER_EQUIVALENT = ERA5LandSnowVariable(variable='snow_water_equivalent', api_variable='snow_depth_water_equivalent', provider_column='era5_land_snow_water_equivalent', column=ERA5_LAND_SNOW_WATER_EQUIVALENT, unit='mm')
ERA5_LAND_SNOW_VARIABLES
module-attribute
¶
ERA5_LAND_SNOW_VARIABLES = (ERA5_LAND_SNOW_DEPTH, ERA5_LAND_SNOW_WATER_EQUIVALENT)
ERA5_LAND_SNOW_VARIABLES_BY_NAME
module-attribute
¶
ERA5_LAND_SNOW_VARIABLES_BY_NAME = {(variable): _3adfor item in ERA5_LAND_SNOW_VARIABLES}
ERA5_LAND_COVERAGE_PRODUCT
module-attribute
¶
ERA5_LAND_COVERAGE_PRODUCT = RemoteSensingProduct(product='era5_land_open_meteo', provider='open_meteo', variables=('soil_moisture',), spatial=create_global(description='global land surface; open water is permanently null'), temporal_start=Timestamp('1950-01-01'), temporal_end=None, cadence='PT1H', nodata_meaning='masked', gap_filling='none', extrapolation='forbidden', native_resolution_m=ERA5_LAND_NATIVE_RESOLUTION_M, provenance='reanalysis', credential_tier='token_free', citation='Munoz-Sabater, J. et al. (2021). ERA5-Land: a state-of-the-art global reanalysis dataset for land applications. Earth System Science Data 13, 4349-4383. https://doi.org/10.5194/essd-13-4349-2021 [served through the Open-Meteo Historical Weather API]', license='CC-BY-4.0', literature_key='munoz_sabater_2021_era5_land', unit='m3/m3', notes=("Reanalysis of the land surface, not a satellite retrieval. The provenance class is 'reanalysis' and the column names say so.", 'Open water returns null at every timestamp, so the land boundary is detected by the same coverage rule as a satellite product with no special case.'))
ERA5LandSoilLayer
dataclass
¶
One native ERA5-Land soil layer.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
key
|
str
|
Layer key used in configuration, for example |
required |
variable
|
str
|
Canonical Neptoon meteo variable name for this layer. |
required |
api_variable
|
str
|
Open-Meteo hourly variable name. |
required |
provider_column
|
str
|
Column name inside the parsed provider frame. |
required |
column
|
Name
|
Neptoon output column. |
required |
top_cm
|
float
|
Layer boundaries below the surface, in centimetres. |
required |
bottom_cm
|
float
|
Layer boundaries below the surface, in centimetres. |
required |
ERA5LandSnowVariable
dataclass
¶
get_era5_land_soil_variables ¶
Return canonical meteo variable names for the configured layers.
is_era5_land_variable ¶
Return whether a canonical meteo variable comes from ERA5-Land.
describe_default_layer_choice ¶
Return the stated reason for the default layer, for CLI and reports.
build_point_coverage_observations ¶
Return one coverage observation per timestamp of a point series.
For a point API the valid fraction degenerates to zero or one, because a single cell is either populated or null. That is not a weakness of the rule: an offshore ERA5-Land query returns null at every timestamp, so the land-surface boundary is detected by the same check that reads pixels in a raster window, with no special case.
check_era5_land_coverage ¶
Decide whether ERA5-Land actually covers this location.
Raises:
| Type | Description |
|---|---|
RemoteSensingCoverageError
|
When no timestamp over the whole requested period carries a value. A single null hour is a gap; every hour null is open water or another permanently excluded surface, and attaching a column of NaN for it would read as "we tried" rather than as a configuration mistake. |
record_era5_land_audit ¶
record_era5_land_audit(*, variables, columns, model, content_hash, grid_cell=None, retrieved_at=None, sensor_name=None)
Record what was fetched, from which product version, and its checksum.
The archive model is the product version here: era5 and era5_land are
different datasets served through one endpoint, and the column names commit
to one of them. The content hash sits next to it because a provider version
string has been observed to disagree with the data it names, so a rerun is
verified against the bytes rather than against the label on them.
neptoon.external.meteo.eobs¶
eobs ¶
International meteorological provider implementation.
This internal module is re-exported through
neptoon.external.international_meteo_data_collection for compatibility.
EOBSDataFetcher ¶
Read local E-OBS daily gridded products or daily CSV extracts.
parse_eobs_daily_csv ¶
Parse a small E-OBS-style daily CSV extract into Neptoon columns.
This is mainly useful for reproducible local extracts and tests. Official
E-OBS data are normally distributed as NetCDF, which is handled by
EOBSDataFetcher when xarray is available.
neptoon.external.meteo.meteoswiss¶
meteoswiss ¶
International meteorological provider implementation.
This internal module is re-exported through
neptoon.external.international_meteo_data_collection for compatibility.
METEOSWISS_RECENT_CACHE_MAX_AGE_HOURS
module-attribute
¶
MeteoSwissDataFetcher ¶
MeteoSwissDataFetcher(latitude, longitude, station='nearest', product='hourly', file_paths=None, data_url=None, stac_collection_url=None, cache_dir=None, request_timeout=30, use_cache=True)
Read MeteoSwiss Open Data station files or resolve them via STAC.
The most reproducible mode is to provide file_paths or a direct
data_url from the MeteoSwiss/STAC browser. If neither is provided the
fetcher makes a best-effort STAC lookup in the SwissMetNet collection.
stac_collection_url
instance-attribute
¶
stac_collection_url = stac_collection_url or METEOSWISS_STAC_COLLECTION_URL
parse_meteoswiss_station_csv ¶
Parse a MeteoSwiss/SwissMetNet station file into Neptoon columns.
MeteoSwiss OGD station files are semicolon-separated and commonly contain
columns such as stn, time, tre200s0 (temperature), rre150z0
(precipitation), ure200s0 (relative humidity), prestas0 (station
pressure) and gre000z0 (global radiation). The parser is intentionally
tolerant and also accepts already-normalized column names for local
extracts.
neptoon.external.meteo.knmi¶
knmi ¶
International meteorological provider implementation.
This internal module is re-exported through
neptoon.external.international_meteo_data_collection for compatibility.
KNMIDataFetcher ¶
KNMIDataFetcher(latitude, longitude, station='nearest', product='hourly', api_key=None, dataset_name=None, dataset_version=None, file_paths=None, cache_dir=None, request_timeout=30, use_cache=True, max_files=200)
Read KNMI Data Platform hourly/daily in-situ observation files.
dataset_name
instance-attribute
¶
dataset_name = validate_path_component(str(dataset_name), field_name='KNMI dataset_name')
dataset_version
instance-attribute
¶
dataset_version = validate_path_component(str(dataset_version), field_name='KNMI dataset_version')
parse_knmi_observations_csv ¶
Parse a KNMI hourly/daily observations CSV extract into Neptoon columns.
neptoon.external.meteo.meteo_france¶
meteo_france ¶
International meteorological provider implementation.
This internal module is re-exported through
neptoon.external.international_meteo_data_collection for compatibility.
MeteoFranceStationCatalog ¶
MeteoFranceStationCatalog(catalog_path=None, catalog_url=None, cache_dir=None, request_timeout=30, use_cache=True)
Load Météo-France station metadata and find nearest product candidates.
The default catalog is a compact data.gouv.fr CSV derived from the public Météo-France station API. It contains station ids, coordinates and product availability flags (daily, hourly and 6-minute). Official GeoJSON station metadata or local CSV/GeoJSON files can be supplied through the URL/path overrides for reproducible runs.
catalog_url
instance-attribute
¶
catalog_url = strip() if catalog_url else METEO_FRANCE_STATION_CATALOG_URL
get_stations_by_distance ¶
get_stations_by_distance(latitude, longitude, product='hourly', start=None, end=None, max_distance_km=None, limit=25)
MeteoFranceDataFetcher ¶
MeteoFranceDataFetcher(latitude, longitude, station='nearest', product='auto', api_key=None, api_url=None, file_paths=None, station_catalog_url=None, station_catalog_path=None, max_distance_km=None, station_candidate_limit=25, cache_dir=None, request_timeout=30, use_cache=True)
Read Météo-France observations from files, API and station metadata.
Local files remain the most reproducible input. For API access, the fetcher
can resolve station="nearest" through a cached Météo-France station
catalog with coordinates and product flags, then sends the selected station
id with the provider request.
api_url
instance-attribute
¶
api_url = strip() if api_url else METEO_FRANCE_DEFAULT_API_URL if api_key else None
station_catalog_url
instance-attribute
¶
station_catalog_url = strip() if station_catalog_url else METEO_FRANCE_STATION_CATALOG_URL
station_catalog_path
instance-attribute
¶
parse_meteo_france_observations ¶
Parse Météo-France observation CSV/JSON extracts into Neptoon columns.
The parser accepts local exports and direct API responses. It recognises
common French/OpenData/SYNOP-style column names such as T (temperature,
often Kelvin), U (relative humidity), P/PMER/PSTAT (pressure,
often Pa), RR1/RR (precipitation) and GLO/RAY_GLO01 (global
radiation), as well as already-normalised column names.
parse_meteo_france_station_catalog ¶
Parse Météo-France station catalog CSV/JSON/GeoJSON metadata.
Returned columns are normalized to station_id, latitude,
longitude, optional elevation_m and product flags
is_hourly/is_daily/is_minutely where the source exposes them.
neptoon.external.meteo.rmi_belgium¶
rmi_belgium ¶
Belgian RMI/KMI automatic weather-station WFS provider.
The provider uses the public Royal Meteorological Institute of Belgium WFS
without authentication. Station metadata comes from aws:aws_station and
hourly observations from aws:aws_1hour.
RMI_BELGIUM_WFS_URL
module-attribute
¶
RMI_BELGIUM_VARIABLE_CANDIDATES
module-attribute
¶
RMI_BELGIUM_VARIABLE_CANDIDATES = {'pressure': ('PRESSURE', 'pressure', 'air_pressure', 'pressure_hpa'), 'temperature': ('TEMP_DRY_SHELTER_AVG', 'temp_dry_shelter_avg', 'temperature', 'air_temperature', 'temperature_c'), 'relative_humidity': ('HUMIDITY_REL_SHELTER_AVG', 'humidity_rel_shelter_avg', 'relative_humidity', 'humidity'), 'precipitation': ('PRECIP_QUANTITY', 'precip_quantity', 'precipitation', 'precipitation_mm'), 'solar_radiation': ('SHORT_WAVE_FROM_SKY_AVG', 'short_wave_from_sky_avg', 'global_radiation', 'solar_radiation')}
RMI_BELGIUM_PROVIDER_COLUMNS
module-attribute
¶
RMI_BELGIUM_PROVIDER_COLUMNS = {'pressure': 'pressure_hpa', 'temperature': 'temperature_c', 'relative_humidity': 'relative_humidity', 'precipitation': 'precipitation_mm', 'solar_radiation': 'solar_radiation'}
__all__
module-attribute
¶
__all__ = ['RMIBelgiumDataFetcher', 'RMIBelgiumStationCatalog', 'RMI_BELGIUM_HOURLY_TYPENAME', 'RMI_BELGIUM_PROVIDER_COLUMNS', 'RMI_BELGIUM_STATION_TYPENAME', 'RMI_BELGIUM_VARIABLE_CANDIDATES', 'RMI_BELGIUM_WFS_URL', 'parse_rmi_belgium_hourly_observations', 'parse_rmi_belgium_station_catalog']
RMIBelgiumStationCatalog ¶
RMIBelgiumDataFetcher ¶
RMIBelgiumDataFetcher(latitude, longitude, station='nearest', wfs_url=None, max_distance_km=None, station_candidate_limit=25, cache_dir=None, request_timeout=30, use_cache=True)
Fetch public hourly RMI/KMI AWS observations without an API key.
fetch_hourly ¶
Fetch, normalize, aggregate and clip hourly observations.
parse_rmi_belgium_station_catalog ¶
Parse the public aws:aws_station GeoJSON station catalog.
parse_rmi_belgium_hourly_observations ¶
Parse aws:aws_1hour GeoJSON into normalized Neptoon provider columns.
neptoon.external.meteo.vmm_waterinfo¶
vmm_waterinfo ¶
Anonymous VMM Waterinfo meteorological station provider.
The provider uses the public KIWIS endpoint published by the Flemish Environment Agency (VMM). It discovers station time series for air pressure, air temperature, relative humidity and precipitation and downloads values without an API key. Anonymous Waterinfo access is rate/credit limited, so station metadata and bounded yearly observation requests are cached locally.
VMM_WATERINFO_URL
module-attribute
¶
VMM_WATERINFO_AIR_PRESSURE_GROUP_ID
module-attribute
¶
VMM_WATERINFO_RELATIVE_HUMIDITY_GROUP_ID
module-attribute
¶
VMM_WATERINFO_AIR_TEMPERATURE_GROUP_ID
module-attribute
¶
VMM_WATERINFO_HOURLY_PRECIPITATION_GROUP_ID
module-attribute
¶
VMM_WATERINFO_DEFAULT_GROUP_IDS
module-attribute
¶
VMM_WATERINFO_DEFAULT_GROUP_IDS = {'pressure': VMM_WATERINFO_AIR_PRESSURE_GROUP_ID, 'temperature': VMM_WATERINFO_AIR_TEMPERATURE_GROUP_ID, 'relative_humidity': VMM_WATERINFO_RELATIVE_HUMIDITY_GROUP_ID, 'precipitation': VMM_WATERINFO_HOURLY_PRECIPITATION_GROUP_ID}
VMM_WATERINFO_PROVIDER_COLUMNS
module-attribute
¶
VMM_WATERINFO_PROVIDER_COLUMNS = {'pressure': 'pressure_hpa', 'temperature': 'temperature_c', 'relative_humidity': 'relative_humidity', 'precipitation': 'precipitation_mm'}
__all__
module-attribute
¶
__all__ = ['VMMWaterinfoDataFetcher', 'VMMWaterinfoStationCatalog', 'VMM_WATERINFO_AIR_PRESSURE_GROUP_ID', 'VMM_WATERINFO_AIR_TEMPERATURE_GROUP_ID', 'VMM_WATERINFO_DEFAULT_GROUP_IDS', 'VMM_WATERINFO_HOURLY_PRECIPITATION_GROUP_ID', 'VMM_WATERINFO_PROVIDER_COLUMNS', 'VMM_WATERINFO_RELATIVE_HUMIDITY_GROUP_ID', 'VMM_WATERINFO_URL', 'parse_vmm_waterinfo_hourly_precipitation', 'parse_vmm_waterinfo_station_catalog', 'parse_vmm_waterinfo_timeseries']
VMMWaterinfoStationCatalog ¶
VMMWaterinfoStationCatalog(base_url=None, pressure_group_id=None, temperature_group_id=None, relative_humidity_group_id=None, precipitation_group_id=None, variables=('precipitation',), cache_dir=None, request_timeout=30, use_cache=True)
Discover and rank VMM meteorological station time series.
group_ids
instance-attribute
¶
group_ids = _group_ids(pressure_group_id=pressure_group_id, temperature_group_id=temperature_group_id, relative_humidity_group_id=relative_humidity_group_id, precipitation_group_id=precipitation_group_id)
get_stations_by_distance ¶
Return nearest stations, preferring complete variable coverage.
get_station_by_id ¶
Resolve a fixed station number, internal id, name or time-series id.
VMMWaterinfoDataFetcher ¶
VMMWaterinfoDataFetcher(latitude, longitude, station='nearest', base_url=None, pressure_group_id=None, temperature_group_id=None, relative_humidity_group_id=None, precipitation_group_id=None, max_distance_km=None, station_candidate_limit=25, cache_dir=None, request_timeout=30, use_cache=True)
Fetch anonymous VMM meteorology with nearest-station fallback.
group_ids
instance-attribute
¶
group_ids = _group_ids(pressure_group_id=pressure_group_id, temperature_group_id=temperature_group_id, relative_humidity_group_id=relative_humidity_group_id, precipitation_group_id=precipitation_group_id)
fetch_hourly ¶
Return requested VMM variables aggregated to UTC hours.
parse_vmm_waterinfo_station_catalog ¶
Parse station metadata from a VMM KIWIS value-layer response.
parse_vmm_waterinfo_timeseries ¶
Parse one VMM KIWIS time series and aggregate it to UTC hours.
parse_vmm_waterinfo_hourly_precipitation ¶
Backward-compatible precipitation parser.
neptoon.external.meteo.national_services¶
national_services ¶
International meteorological provider implementation.
This internal module is re-exported through
neptoon.external.international_meteo_data_collection for compatibility.
BOM_STATION_CATALOG_URL
module-attribute
¶
BOM_LATEST_OBSERVATION_BASE_URL
module-attribute
¶
BOM_STATE_PRODUCT_IDS
module-attribute
¶
BOM_STATE_PRODUCT_IDS = {'NSW': 'IDN60801', 'ACT': 'IDN60801', 'VIC': 'IDV60801', 'QLD': 'IDQ60801', 'WA': 'IDW60801', 'SA': 'IDS60801', 'TAS': 'IDT60801', 'NT': 'IDD60801'}
FMIDataFetcher ¶
FMIDataFetcher(latitude, longitude, station='nearest', data_url=None, file_paths=None, stored_query_id='fmi::observations::weather::timevaluepair', parameters=None, cache_dir=None, request_timeout=30, use_cache=True)
Read token-free FMI Open Data observation extracts or configured WFS URLs.
ECCCGeoMetDataFetcher ¶
ECCCGeoMetDataFetcher(latitude, longitude, station='nearest', api_url=None, collection_id=None, file_paths=None, cache_dir=None, request_timeout=30, use_cache=True)
Read token-free ECCC/MSC GeoMet OGC API or local observation extracts.
BOMDataFetcher ¶
BOMDataFetcher(latitude, longitude, station='nearest', data_url=None, file_paths=None, station_catalog_url=None, station_catalog_path=None, product_id=None, max_distance_km=None, station_candidate_limit=25, cache_dir=None, request_timeout=30, use_cache=True)
Read Australian Bureau of Meteorology latest station observations.
station=nearest now resolves an active WMO station from BoM's national
station list and downloads its public 72-hour JSON product. Existing
bom_data_url and bom_file_paths overrides remain supported for
reproducible and historical workflows.
station_catalog_path
instance-attribute
¶
MetOfficeDataHubFetcher ¶
MetOfficeDataHubFetcher(latitude, longitude, station='nearest', api_key=None, api_url=None, file_paths=None, cache_dir=None, request_timeout=30, use_cache=True)
Read Met Office Weather DataHub Land Observations style data.
parse_met_office_observations ¶
Parse Met Office Weather DataHub observation JSON/GeoJSON or CSV extracts.
parse_fmi_open_data_observations ¶
Parse FMI Open Data CSV/JSON extracts into Neptoon provider columns.
FMI WFS XML can vary by stored query and encoding. For reproducible workflows this parser intentionally targets CSV/JSON exports and cached extracts while the fetcher can still download a configured WFS/CSV URL without an API key.
parse_eccc_geomet_observations ¶
Parse ECCC/MSC GeoMet OGC API JSON/GeoJSON or CSV observations.
parse_bom_station_catalog ¶
Parse BoM's national fixed-width station list for live observations.
Only active stations with a WMO identifier are useful for the public
72-hour fwo JSON products, so historic-only stations are retained in
the parser but filtered by the fetcher when resolving nearest live data.
parse_bom_observations ¶
Parse BoM observations and de-accumulate rain since 9am fields.
BoM rain_trace/rain_since_9am values are running accumulations, not
independent observation-period totals. The parser converts them to
non-overlapping increments and carries an explicit interval start for each
increment so later resampling conserves rainfall across arbitrary grids.
neptoon.external.meteo.geosphere_austria¶
geosphere_austria ¶
GeoSphere Austria Data Hub station observations.
The adapter uses the public klima-v2-1h historical station resource. It
supports explicit station ids as well as nearest-station selection from the
resource metadata and exposes the canonical Neptoon hourly meteorological
variables.
GEOSPHERE_AUSTRIA_API_BASE_URL
module-attribute
¶
GEOSPHERE_AUSTRIA_HOURLY_RESOURCE
module-attribute
¶
GEOSPHERE_AUSTRIA_PARAMETER_MAP
module-attribute
¶
GEOSPHERE_AUSTRIA_PARAMETER_MAP = {'pressure': 'p', 'temperature': 'tl', 'relative_humidity': 'rf', 'precipitation': 'rr', 'solar_radiation': 'cglo', 'wind_speed': 'ff'}
GEOSPHERE_AUSTRIA_PROVIDER_COLUMNS
module-attribute
¶
GEOSPHERE_AUSTRIA_PROVIDER_COLUMNS = {'p': 'pressure_hpa', 'tl': 'temperature_c', 'rf': 'relative_humidity', 'rr': 'precipitation_mm', 'cglo': 'solar_radiation', 'ff': 'wind_speed_m_s'}
GeoSphereAustriaDataFetcher ¶
GeoSphereAustriaDataFetcher(latitude, longitude, station='nearest', *, base_url=None, resource_id=None, file_paths=None, station_catalog_path=None, max_distance_km=None, station_candidate_limit=25, cache_dir=None, request_timeout=30, use_cache=True)
Fetch quality-controlled hourly GeoSphere Austria station observations.
station_catalog_path
instance-attribute
¶
parse_geosphere_austria_station_catalog ¶
Parse GeoSphere station metadata to canonical nearest-station fields.
parse_geosphere_austria_observations ¶
Parse klima-v2-1h CSV/JSON station data to canonical columns.
neptoon.external.meteo.frost¶
frost ¶
MET Norway Frost historical station observations.
Frost provides quality-controlled historical observations through an authenticated public API. Open-data requests use HTTP Basic authentication with the Frost client id as username and an empty password.
FROST_OBSERVATION_ENDPOINT
module-attribute
¶
FROST_ELEMENT_MAP
module-attribute
¶
FROST_ELEMENT_MAP = {'pressure': 'surface_air_pressure', 'temperature': 'air_temperature', 'relative_humidity': 'relative_humidity', 'precipitation': 'sum(precipitation_amount PT1H)', 'wind_speed': 'wind_speed'}
FROST_PROVIDER_COLUMNS
module-attribute
¶
FROST_PROVIDER_COLUMNS = {'surface_air_pressure': 'pressure_hpa', 'air_temperature': 'temperature_c', 'relative_humidity': 'relative_humidity', 'sum(precipitation_amount PT1H)': 'precipitation_mm', 'wind_speed': 'wind_speed_m_s'}
FrostDataFetcher ¶
FrostDataFetcher(latitude, longitude, station='nearest', *, client_id=None, base_url=None, file_paths=None, source_file=None, max_distance_km=None, station_candidate_limit=25, cache_dir=None, request_timeout=30, use_cache=True)
Fetch hourly observations from MET Norway's Frost API.
parse_frost_sources ¶
Parse Frost sources response to canonical station metadata.
parse_frost_observations ¶
Parse Frost observations to canonical Neptoon hourly columns.
neptoon.external.meteo.dmi¶
dmi ¶
Danish Meteorological Institute Open Data meteorological observations.
DMI_OPEN_DATA_BASE_URL
module-attribute
¶
DMI_OBSERVATION_ENDPOINT
module-attribute
¶
DMI_PARAMETER_MAP
module-attribute
¶
DMI_PARAMETER_MAP = {'pressure': 'pressure', 'temperature': 'temp_mean_past1h', 'relative_humidity': 'humidity_past1h', 'precipitation': 'precip_past1h', 'solar_radiation': 'radia_glob_past1h', 'wind_speed': 'wind_speed_past1h'}
DMI_PROVIDER_COLUMNS
module-attribute
¶
DMI_PROVIDER_COLUMNS = {'pressure': 'pressure_hpa', 'temp_mean_past1h': 'temperature_c', 'humidity_past1h': 'relative_humidity', 'precip_past1h': 'precipitation_mm', 'radia_glob_past1h': 'solar_radiation', 'wind_speed_past1h': 'wind_speed_m_s'}
DMIDataFetcher ¶
DMIDataFetcher(latitude, longitude, station='nearest', *, base_url=None, file_paths=None, station_catalog_path=None, max_distance_km=None, station_candidate_limit=25, cache_dir=None, request_timeout=30, use_cache=True)
Fetch nearest-station hourly data from DMI Open Data.
station_catalog_path
instance-attribute
¶
parse_dmi_station_catalog ¶
Parse DMI GeoJSON station features to canonical station metadata.
parse_dmi_observations ¶
Parse DMI GeoJSON observation features to canonical provider columns.
neptoon.external.meteo.ipma¶
ipma ¶
IPMA Open Data hourly meteorological station observations for Portugal.
The public IPMA feed is token-free and publishes the latest 24 hours of hourly station observations together with a national station catalogue. This adapter supports explicit station ids and nearest-station resolution from latitude and longitude.
IPMA_OPEN_DATA_BASE_URL
module-attribute
¶
IPMA_STATION_CATALOG_URL
module-attribute
¶
IPMA_STATION_CATALOG_URL = f'{IPMA_OPEN_DATA_BASE_URL}/stations.json'
IPMA_OBSERVATIONS_URL
module-attribute
¶
IPMA_OBSERVATIONS_URL = f'{IPMA_OPEN_DATA_BASE_URL}/observations.json'
IPMA_SOURCE_FIELDS
module-attribute
¶
IPMA_SOURCE_FIELDS = {'pressure': 'pressao', 'temperature': 'temperatura', 'relative_humidity': 'humidade', 'precipitation': 'precAcumulada', 'solar_radiation': 'radiacao', 'wind_speed': 'intensidadeVento'}
IPMA_PROVIDER_COLUMNS
module-attribute
¶
IPMA_PROVIDER_COLUMNS = {'pressure': 'pressure_hpa', 'temperature': 'temperature_c', 'relative_humidity': 'relative_humidity', 'precipitation': 'precipitation_mm', 'solar_radiation': 'solar_radiation', 'wind_speed': 'wind_speed_m_s'}
IPMADataFetcher ¶
IPMADataFetcher(latitude, longitude, station='nearest', *, data_url=None, file_paths=None, station_catalog_url=None, station_catalog_path=None, max_distance_km=None, station_candidate_limit=25, cache_dir=None, request_timeout=30, use_cache=True)
Fetch token-free IPMA hourly observations with nearest-station support.
station_catalog_path
instance-attribute
¶
parse_ipma_station_catalog ¶
Parse IPMA stations.json into nearest-station metadata.
parse_ipma_observations ¶
Parse IPMA's latest-24-hour hourly observation JSON.
IPMA uses -99.0 as its documented no-data sentinel. Solar radiation
is published as hourly energy in kJ/m2 and is converted to mean W/m2 by
dividing by 3.6. precAcumulada is the precipitation accumulated during
the hour ending at the record timestamp.
neptoon.external.meteo.smhi¶
smhi ¶
Token-free SMHI meteorological observations for Sweden.
SMHI's metobs REST API exposes one parameter per endpoint together with station
metadata. The adapter intersects station availability across requested
variables so station: nearest selects a station capable of serving the
whole request whenever possible. Recent and corrected archive periods are
combined automatically for ranges spanning the rolling recent window.
SMHI_METOBS_BASE_URL
module-attribute
¶
SMHI_PARAMETER_IDS
module-attribute
¶
SMHI_PARAMETER_IDS = {'temperature': '1', 'wind_speed': '4', 'relative_humidity': '6', 'precipitation': '7'}
SMHI_PROVIDER_COLUMNS
module-attribute
¶
SMHI_PROVIDER_COLUMNS = {'temperature': 'temperature_c', 'wind_speed': 'wind_speed_m_s', 'relative_humidity': 'relative_humidity', 'precipitation': 'precipitation_mm'}
SMHIDataFetcher ¶
SMHIDataFetcher(latitude, longitude, station='nearest', *, base_url=None, file_paths=None, period=None, max_distance_km=None, station_candidate_limit=25, cache_dir=None, request_timeout=30, use_cache=True)
Fetch SMHI hourly observations without credentials.
parse_smhi_station_catalog ¶
Parse a SMHI parameter endpoint into canonical station metadata.
parse_smhi_observations ¶
Parse one SMHI parameter/station/period JSON response.
The public response stores observations in value with epoch-millisecond
timestamps. variable may be omitted when the response includes the
parameter key; otherwise it is required for local custom extracts.
neptoon.external.meteo.imgw¶
imgw ¶
Token-free IMGW-PIB synoptic observations for Poland.
The public IMGW synop API exposes the latest station observations without credentials. Historical/reproducible processing can use saved API JSON/CSV files. Nearest-station selection uses a built-in synoptic station coordinate inventory derived from IMGW's public meteorological yearbook. A local or remote coordinate catalogue can override that inventory.
IMGW_SYNOP_API_URL
module-attribute
¶
IMGW_STATION_CATALOG_URL
module-attribute
¶
IMGW_STATION_CATALOG_URL = 'https://danepubliczne.imgw.pl/data/dane_pomiarowo_obserwacyjne/dane_meteorologiczne/wykaz_stacji.csv'
IMGW_SOURCE_FIELDS
module-attribute
¶
IMGW_SOURCE_FIELDS = {'pressure': 'cisnienie', 'temperature': 'temperatura', 'relative_humidity': 'wilgotnosc_wzgledna', 'wind_speed': 'predkosc_wiatru'}
IMGW_PROVIDER_COLUMNS
module-attribute
¶
IMGW_PROVIDER_COLUMNS = {'pressure': 'pressure_hpa', 'temperature': 'temperature_c', 'relative_humidity': 'relative_humidity', 'wind_speed': 'wind_speed_m_s'}
IMGWDataFetcher ¶
IMGWDataFetcher(latitude, longitude, station='nearest', *, data_url=None, file_paths=None, station_catalog_url=None, station_catalog_path=None, max_distance_km=None, station_candidate_limit=25, cache_dir=None, request_timeout=30, use_cache=True)
imgw_builtin_station_catalog ¶
Return the built-in IMGW synoptic station coordinate inventory.
parse_imgw_observations ¶
Parse IMGW's public synop JSON/CSV to canonical provider columns.
parse_imgw_station_catalog ¶
Parse IMGW station metadata with flexible Polish/English headers.
neptoon.external.meteo.met_eireann¶
met_eireann ¶
Token-free Met Éireann hourly observations through the public OGC EDR API.
MET_EIREANN_EDR_BASE_URL
module-attribute
¶
MET_EIREANN_HOURLY_COLLECTION
module-attribute
¶
MET_EIREANN_PROVIDER_COLUMNS
module-attribute
¶
MET_EIREANN_PROVIDER_COLUMNS = {'pressure': 'pressure_hpa', 'temperature': 'temperature_c', 'relative_humidity': 'relative_humidity', 'precipitation': 'precipitation_mm', 'solar_radiation': 'solar_radiation', 'wind_speed': 'wind_speed_m_s'}
MET_EIREANN_PARAMETER_ALIASES
module-attribute
¶
MET_EIREANN_PARAMETER_ALIASES = {'pressure': ('station_pressure', 'air_pressure', 'atmospheric_pressure', 'pressure'), 'temperature': ('air_temperature', 'dry_bulb_temperature', 'temperature'), 'relative_humidity': ('relative_humidity', 'humidity'), 'precipitation': ('precipitation_amount', 'rainfall_amount', 'rainfall', 'precipitation', 'rain'), 'solar_radiation': ('global_solar_radiation', 'global_radiation', 'solar_radiation', 'shortwave_radiation'), 'wind_speed': ('mean_wind_speed', 'wind_speed', 'windspeed')}
MetEireannDataFetcher ¶
MetEireannDataFetcher(latitude, longitude, station='nearest', *, base_url=None, collection_id=None, file_paths=None, station_catalog_path=None, max_distance_km=None, station_candidate_limit=25, cache_dir=None, request_timeout=30, use_cache=True)
Fetch public 60-minute Met Éireann observations from OGC EDR.
station_catalog_path
instance-attribute
¶
parse_met_eireann_locations ¶
Parse EDR location metadata into canonical station metadata.
parse_met_eireann_observations ¶
Parse Met Éireann CoverageJSON (preferred) or record-oriented JSON.
neptoon.external.meteo.noaa_nws¶
noaa_nws ¶
Token-free NOAA/NWS surface observations from api.weather.gov.
The NWS API does not require an API key, but it requires an identifying
User-Agent header. The adapter uses station observations only; forecasts
are intentionally outside the meteorological attachment contract.
NWS_DEFAULT_USER_AGENT
module-attribute
¶
NWS_PROVIDER_COLUMNS
module-attribute
¶
NWS_PROVIDER_COLUMNS = {'pressure': 'pressure_hpa', 'temperature': 'temperature_c', 'relative_humidity': 'relative_humidity', 'precipitation': 'precipitation_mm', 'wind_speed': 'wind_speed_m_s'}
NWSDataFetcher ¶
NWSDataFetcher(*, latitude, longitude, station='nearest', base_url=None, user_agent=None, file_paths=None, station_catalog_path=None, max_distance_km=None, station_candidate_limit=25, cache_dir=None, request_timeout=30, use_cache=True)
parse_nws_station_catalog ¶
Parse an NWS station FeatureCollection into canonical metadata.
parse_nws_observations ¶
Parse NWS station observation GeoJSON to canonical Neptoon columns.
neptoon.external.meteo.inmet¶
inmet ¶
Token-free Brazilian INMET SYNOP observations through the public WIS2 OGC API.
INMET_PROVIDER_COLUMNS
module-attribute
¶
INMET_PROVIDER_COLUMNS = {'pressure': 'pressure_hpa', 'temperature': 'temperature_c', 'relative_humidity': 'relative_humidity', 'precipitation': 'precipitation_mm', 'wind_speed': 'wind_speed_m_s'}
INMET_VARIABLE_NAMES
module-attribute
¶
INMET_VARIABLE_NAMES = {'non_coordinate_pressure': 'pressure', 'air_temperature': 'temperature', 'relative_humidity': 'relative_humidity', 'total_precipitation_or_total_water_equivalent': 'precipitation', 'wind_speed': 'wind_speed'}
INMETDataFetcher ¶
INMETDataFetcher(*, latitude, longitude, station='nearest', base_url=None, file_paths=None, station_catalog_path=None, max_distance_km=None, station_candidate_limit=25, cache_dir=None, request_timeout=30, use_cache=True)
parse_inmet_station_catalog ¶
Parse INMET WIS2 station GeoJSON into canonical station metadata.
parse_inmet_observations ¶
Parse variable-wise INMET SYNOP GeoJSON into hourly canonical columns.
requests_url ¶
Return the effective GET URL without exposing query values in cache filenames.
neptoon.external.meteo.jma_amedas¶
jma_amedas ¶
Token-free near-real-time JMA AMeDAS observations.
The public JMA web feed is intended for current/recent observations and does
not provide a stable unlimited archive contract. For reproducible or older
runs, callers can provide saved AMeDAS JSON files through jma_file_paths.
JMA_AMEDAS_STATION_TABLE_PATH
module-attribute
¶
JMA_PROVIDER_COLUMNS
module-attribute
¶
JMA_PROVIDER_COLUMNS = {'pressure': 'pressure_hpa', 'temperature': 'temperature_c', 'relative_humidity': 'relative_humidity', 'precipitation': 'precipitation_mm', 'wind_speed': 'wind_speed_m_s'}
JMA_SOURCE_FIELDS
module-attribute
¶
JMA_SOURCE_FIELDS = {'pressure': 'pressure', 'temperature': 'temp', 'relative_humidity': 'humidity', 'precipitation': 'precipitation1h', 'wind_speed': 'wind'}
JMAAmedasDataFetcher ¶
JMAAmedasDataFetcher(*, latitude, longitude, station='nearest', base_url=None, file_paths=None, station_catalog_path=None, max_distance_km=None, station_candidate_limit=25, cache_dir=None, request_timeout=30, use_cache=True)
parse_jma_amedas_station_catalog ¶
Parse JMA amedastable.json station metadata.
parse_jma_amedas_observations ¶
Parse one or more JMA AMeDAS point JSON blocks to canonical columns.
neptoon.external.nmdb_cli¶
nmdb_cli ¶
Typer CLI for standalone NMDB downloads.
download_nmdb_command ¶
download_nmdb_command(station=Option(..., '--station', '-s', help='NMDB station code, e.g. JUNG.'), start_date=Option(..., '--start-date', help='Start date, YYYY-MM-DD.'), end_date=Option(..., '--end-date', help='End date, YYYY-MM-DD.'), output=Option(..., '--output', '-o', help='CSV file to write.'), resolution=Option('60min', '--resolution', '-r', help='Whole-minute interval as minutes or a pandas duration, for example 1, 10, 10min, 60min, 1h, or 1d.'), table=Option('revori', '--table', help='NMDB table name, commonly revori or ori.'), cache=Option(True, '--cache/--no-cache', help='Reuse and update the local NMDB cache.'), cache_dir=Option(None, '--cache-dir', help="Optional cache directory. Uses Neptoon's platform cache when omitted."), allow_cache_fallback=Option(True, '--allow-cache-fallback/--no-cache-fallback', help='Use available cached data if an NMDB refresh fails.'), request_timeout=Option(30, '--request-timeout', min=1, help='HTTPS timeout in seconds.'), https_proxy=Option(None, '--https-proxy', help='Optional HTTPS proxy URL.'), try_without_proxy=Option(True, '--try-without-proxy/--no-try-without-proxy', help='Retry without proxy if the proxy request fails.'))
Download NMDB data directly to CSV.
list_nmdb_stations ¶
List NMDB station codes with built-in long-term reference values.
neptoon.external.nmdb_data_collection¶
nmdb_data_collection ¶
NMDB_REFERENCES
module-attribute
¶
NMDB_REFERENCES = {'AATB': 157, 'INVK': 111, 'JUNG': 168, 'KERG': 239, 'KIEL': 190, 'MXCO': 227, 'NEWK': 100, 'OULU': 113, 'PSNM': 615, 'SOPO': 308, 'TERA': 124, 'THUL': 130}
NMDB_CUTOFF_RIGIDITIES
module-attribute
¶
NMDB_CUTOFF_RIGIDITIES = {'AATB': 5.2, 'INVK': 0.186, 'JUNG': 5.0, 'KERG': 1, 'KIEL': 2.4, 'MXCO': 7.495, 'NEWK': 2.6, 'OULU': 0.7, 'PSNM': 16.674, 'SOPO': 0.0, 'TERA': 0, 'THUL': 0}
NMDBFetchStatus
dataclass
¶
NMDBFetchError ¶
Bases: NeptoonExternalDataError, RuntimeError
Raised when NMDB data could not be downloaded or parsed.
NMDBResponseError
dataclass
¶
NMDBConfigurationError ¶
NMDBInputDataError ¶
NMDBDataAttacher ¶
NMDBDataAttacher(data_frame, new_column_name=str(INCOMING_NEUTRON_INTENSITY))
This is the core class that a user interacts with when wanting to attach data from the NMDB.eu database to a dataframe. It includes methods for configuring the NMDBConfig class which is then used by other classes for fetching and parsing data from the NMDB.eu API.
Future validation should ensure the returned dataframe has the expected NMDB schema.
Initialisation parameters
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data_frame
|
DataFrame
|
DataFrame which requires data to be attached. It must have a datetime index. |
required |
new_column_name
|
str
|
column name for the new column were neutron count data is appended, by default "incoming_neutron_intensity" |
str(INCOMING_NEUTRON_INTENSITY)
|
configure ¶
configure(station, reference_value=None, resolution='60', nmdb_table='revori', request_timeout=30, request_retries=0, proxies=None, try_without_proxy=True, allow_cache_fallback=True, max_nearest_timedelta='3D')
fetch_data ¶
Creates a NMDBDataHandler using the config and collects the data whilst storing it under self.tmp_data
attach_data ¶
Attaches the data stored in self.tmp_data to self.data_frame. This occurs inplace.
Raises:
| Type | Description |
|---|---|
ValueError
|
When index of the data is not Datetime an error occurs |
return_data_frame ¶
Returns the DataFrame attached in the object.
Returns:
| Type | Description |
|---|---|
DataFrame
|
The DataFrame |
DateTimeHandler ¶
Class that holds Date standardization methods.
This class provides static methods for converting and standardizing date formats to a common format (YYYY-mm-dd) used throughout the NMDB data collection process.
convert_string_to_standard_date
staticmethod
¶
Converts a string date to the standard format (YYYY-mm-dd).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
date_str
|
str
|
The date string to convert. |
required |
Returns:
| Type | Description |
|---|---|
str or None
|
The standardized date string in format (YYYY-mm-dd), or None if the input date string is not a recognizable date format. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the input string cannot be parsed into a valid date. |
format_datetime_to_standard_string
staticmethod
¶
Converts a datetime or pd.Timestamp date to the standard format (YYYY-mm-dd)
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
date_datetime
|
datetime or Timestamp
|
The datetime object to convert. |
required |
Returns:
| Type | Description |
|---|---|
str or None The standardized date string in format (YYYY-mm-dd),
|
|
or None if the input
|
is not a datetime.datetime or pandas.Timestamp instance. |
standardize_date_input
staticmethod
¶
Takes a date as input, checks type, and converts it to the standard format (YYYY-mm-dd)
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
date_input
|
str or TimeStamp or datetime
|
Input date to be converted |
required |
Returns:
| Type | Description |
|---|---|
str
|
String of the date as YYYY-mm-dd |
Raises:
| Type | Description |
|---|---|
ValueError
|
Raise error when neither str or datetime is given |
NMDBConfig ¶
NMDBConfig(start_date_wanted, end_date_wanted, station='JUNG', reference_value=None, cache_dir=None, nmdb_table='revori', resolution='60', cache_exists=False, cache_start_date=None, cache_end_date=None, start_date_needed=None, end_date_needed=None, use_cache=True, request_timeout=30, request_retries=0, proxies=None, try_without_proxy=True, allow_cache_fallback=True, max_nearest_timedelta='3D')
Configuration class for NMDB data retrieval and processing.
This class encapsulates configuration settings required for NMDB data retrieval, including date ranges for data collection, station identification, cache management, and data resolution settings. It ensures that all components of the NMDB data collection module use consistent and standardized configuration parameters.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
start_date_wanted
|
str
|
Start date for data retrieval, formatted as 'YYYY-MM-DD'. |
required |
end_date_wanted
|
str
|
End date for data retrieval, formatted as 'YYYY-MM-DD'. |
required |
station
|
str
|
NMDB station code from which data is retrieved. Defaults to 'JUNG'. |
'JUNG'
|
cache_dir
|
str or None
|
Path to the cache directory for storing retrieved data. If None, uses the default OS cache directory. |
None
|
nmdb_table
|
str
|
Specific NMDB table to query data from. Defaults to 'revori'. |
'revori'
|
resolution
|
str
|
Resolution of the data in minutes. Defaults to '60'. |
'60'
|
cache_exists
|
bool
|
Indicates whether cached data exists for the given parameters. Defaults to False. |
False
|
cache_start_date
|
str or None
|
Start date of the available cached data, formatted as 'YYYY-MM-DD'. None if no cache exists. |
None
|
cache_end_date
|
str or None
|
End date of the available cached data, formatted as 'YYYY-MM-DD'. None if no cache exists. |
None
|
start_date_needed
|
str or None
|
Start date for which data needs to be fetched, considering cached data. None if all data is cached. |
None
|
end_date_needed
|
str or None
|
End date for which data needs to be fetched, considering cached data. None if all data is cached. |
None
|
use_cache
|
bool
|
whether to use cached data, ignore the cache entirely, defaults to True |
True
|
TermsDisplayManager ¶
CacheHandler ¶
Class to handle cache management using downloaded NMDB data
The cache handler managed file paths, naming of files, storage and deletion of the cache. As default it will be stored in the usual operating system cache location.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
NMDBConfig
|
An instance of the NMDBConfig class containing configuration settings |
required |
Attributes:
| Name | Type | Description |
|---|---|---|
config |
NMDBConfig
|
Stores the configuration settings for NMDB data retrieval. |
_cache_file_path |
Path or None
|
The file path to the cache file, dynamically determined based on the NMDBConfig settings. |
Methods:
| Name | Description |
|---|---|
update_cache_file_path |
Updates the cache file path based on the current NMDBConfig settings. |
check_cache_file_exists |
Checks for the existence of the cache file and updates the configuration accordingly. |
read_cache |
Reads the cached NMDB data from the file and returns it as a DataFrame. |
write_cache |
Writes a DataFrame to the cache file location. |
delete_cache |
Deletes the cache file associated with the current NMDBConfig settings. |
check_cache_range |
Determines the range of dates available in the cache and updates the NMDBConfig settings. |
Examples:
>>> config = NMDBConfig(start_date_wanted='2023-01-01',
>>> end_date_wanted='2023-01-31', station='JUNG')
>>> cache_handler = CacheHandler(config)
>>> cache_handler.update_cache_file_path()
>>> print(cache_handler.cache_file_path)
update_cache_file_path ¶
Update the cache file path based on the current configuration.
check_cache_file_exists ¶
Checks the existence of the cache file and sets the property in config
Returns:
| Type | Description |
|---|---|
None
|
|
read_cache ¶
Reads cache nmdb file and formats index
Returns:
| Name | Type | Description |
|---|---|---|
df |
DataFrame
|
DataFrame from the cache file |
write_cache ¶
Write NMDB data to the cache location using the cache_file_path attribute as a name.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
cache_df
|
DataFrame
|
|
required |
Returns:
| Type | Description |
|---|---|
None
|
|
delete_cache ¶
Delete the cache file related to the current instance. E.g. if downloading hourly data for JUNG it will delete the file associated with hourly JUNG from the cache.
Return
None
check_cache_range ¶
Function to find the range of data already available in the cache. It updates the config file depending on availability. It will either declare none existance of the cache, or update the start and end date of the cache.
Returns:
| Type | Description |
|---|---|
None
|
|
DataFetcher ¶
Class to handle sending external requests and fetching data.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
NMDBConfig
|
An instance of the configuration file. |
required |
Methods:
| Name | Description |
|---|---|
get_ymd_from_date |
static method which parses the date into separate values to represent year, month and day |
create_nmdb_url |
creates the url to request data from NMDB.eu, based on values in the configuration file |
fetch_data_http |
uses the created url to request data from NMDB.eu and returns the text from the response |
parse_http_date |
uses the returned text from fetch_data_http() and parses it into a standard format. The format is a pd.Dataframe with a datetime.datetime index |
last_fetch_status
instance-attribute
¶
last_fetch_status = NMDBFetchStatus(ok=False, source='not_started', message='no request made')
get_ymd_from_date
staticmethod
¶
Parses a given date into year, month, and day.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
date
|
datetime or str
|
The date to be parsed. |
required |
Returns:
| Type | Description |
|---|---|
tuple
|
Tuple containing the year, month, and day of the date. |
create_nmdb_url ¶
Creates the URL for obtaining the data using HTTPS
Returns:
| Name | Type | Description |
|---|---|---|
url |
str
|
URL as a string |
fetch_data_http ¶
Fetches data from the HTTPS NMDB.eu endpoint and processes it.
The primary request uses the configured proxy settings, if any.
When that request fails and try_without_proxy is enabled, a
second request is made with environment proxy settings disabled.
Returns:
| Name | Type | Description |
|---|---|---|
Text |
str
|
Returns the text from the HTTPS endpoint |
parse_http_data ¶
Parse the HTTPS response data into a dataframe
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
raw_data
|
str
|
The raw text file collected from NMDB.eu |
required |
Returns:
| Type | Description |
|---|---|
DataFrame
|
A DataFrame with index DateTime and Counts per second |
Raises:
| Type | Description |
|---|---|
ValueError
|
Raised if the requested date is not available at the specified NMDB station, indicated by a specific error message in the raw data. |
RequestException
|
Raised if there's an issue parsing the HTTPS response into a DataFrame, such as an incorrect format or network-related errors during the fetch. |
Examples:
Assuming an instance nmdb_data_handler of a class that
includes this method:
fetch_and_parse_http_data ¶
Fetches raw NMDB data via HTTPS and parses it into a pandas DataFrame.
This method combines the functionalities of fetching NMDB data
from the designated HTTPS source and subsequently parsing that
raw data into a structured DataFrame. It leverages
fetch_data_http to retrieve the data and parse_http_data to
transform it into a usable format.
Returns:
| Type | Description |
|---|---|
DataFrame
|
A DataFrame containing the NMDB data, indexed by datetime with counts per second. |
Raises:
| Type | Description |
|---|---|
Refer to documentation for fetch_data_http() and
|
|
parse_http_data()
|
|
Examples:
Assuming an instance nmdb_data_handler of a class that
includes this method:
DataManager ¶
Manages the integration of cached and newly fetched NMDB data.
This class is responsible for determining the necessity of fetching new NMDB data based on the existing cache and the desired date range. It also handles the combination of cached data with newly fetched data to provide a complete dataset for analysis.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
NMDBConfig
|
Configuration settings for NMDB data retrieval. |
required |
cache_handler
|
CacheHandler
|
Handles operations related to caching of NMDB data. |
required |
data_fetcher
|
DataFetcher
|
Responsible for fetching new data from NMDB. |
required |
Attributes:
| Name | Type | Description |
|---|---|---|
need_data_before_cache |
bool or None
|
Indicates if data before the cached range is needed. |
need_data_after_cache |
bool or None
|
Indicates if data after the cached range is needed. |
Methods:
| Name | Description |
|---|---|
check_if_need_extra_data |
Evaluates the need for fetching data outside the current cache range. |
set_dates_for_nmdb_download |
Updates the configuration with the date ranges that need to be fetched. |
combine_cache_and_new_data |
Merges newly fetched data with existing cached data, ensuring no duplication. |
Initializes the DataManager with the given configuration, cache handler, and data fetcher.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
NMDBConfig
|
Configuration settings for NMDB data retrieval. |
required |
cache_handler
|
CacheHandler
|
Handles operations related to caching of NMDB data. |
required |
data_fetcher
|
DataFetcher
|
Responsible for fetching new data from NMDB. |
required |
check_if_need_extra_data ¶
Updates configuration instance with boolean values stating whether a download of data is required before or after the desired dates.
Returns:
| Type | Description |
|---|---|
None
|
|
set_dates_for_nmdb_download ¶
Updates the configuration instance with the download range for NMDB data based upon the desired data and the available data in the cache.
Returns:
| Type | Description |
|---|---|
None
|
|
fetch_download_or_cache ¶
Download NMDB data, falling back to cache when allowed.
combine_cache_and_new_data ¶
Combines cached and newly downloaded NMDB data into a single DataFrame, ensuring data continuity and no duplication.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df_cache
|
DataFrame
|
The DataFrame containing cached data. |
required |
df_download
|
DataFrame
|
The DataFrame containing newly downloaded data. |
required |
Returns:
| Type | Description |
|---|---|
DataFrame
|
The combined DataFrame, sorted by datetime. |
NMDBDataHandler ¶
Handles the retrieval and management of NMDB data.
This class integrates the CacheHandler, DataFetcher, and
DataManager to manage NMDB data. It ensures that data is fetched
from the NMDB source only when necessary, preferring cached data to
minimize network requests. The class handles cases where new data
needs to be fetched either because it's not present in the cache or
only partial data is available.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
NMDBConfig
|
Configuration settings for NMDB data retrieval, including desired date ranges, station information, and caching preferences. |
required |
Attributes:
| Name | Type | Description |
|---|---|---|
config |
NMDBConfig
|
Stores the provided NMDB configuration settings. |
cache_handler |
CacheHandler
|
Manages caching operations for NMDB data. |
data_fetcher |
DataFetcher
|
Responsible for fetching new NMDB data when required. |
data_manager |
DataManager
|
Determines the need for and manages the retrieval of new data based on cache status and configuration settings. |
Methods:
| Name | Description |
|---|---|
collect_nmdb_data |
Retrieves NMDB data, prioritizing cached data and fetching new data as needed. Returns a DataFrame containing the relevant NMDB data. |
Examples:
>>> config = NMDBConfig(start_date_wanted='2023-01-01',
>>> end_date_wanted='2023-01-31', station='JUNG')
>>> nmdb_handler = NMDBDataHandler(config)
>>> nmdb_data = nmdb_handler.collect_nmdb_data()
>>> print(nmdb_data.head())
Initializes the NMDBDataHandler with the given NMDBConfig instance.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
NMDBConfig
|
Configuration settings for NMDB data retrieval. |
required |
collect_nmdb_data ¶
Collects NMDB data based on the specified configuration, using cached data when available and fetching new data as necessary.
Returns:
| Type | Description |
|---|---|
DataFrame
|
A DataFrame containing NMDB data for the requested range. This may be a combination of cached and newly fetched data, or solely from one source, depending on availability. |
Examples:
Assuming config has been defined and passed to
NMDBDataHandler:
Note: This example assumes that nmdb_handler has been
instantiated with a valid NMDBConfig.
NMDBDataDownloader ¶
Download NMDB neutron-monitor data as a standalone DataFrame.
The downloader is a small public facade over Neptoon's existing NMDB configuration, cache, request, and parsing components. Cached data are reused by default, while the returned frame is always limited to the requested date window.
Examples:
>>> from neptoon import NMDBDataDownloader
>>> df = NMDBDataDownloader.download(
... "JUNG", "2020-01-01", "2020-01-02", "10Min"
... )
>>> isinstance(df, pd.DataFrame)
True
download
staticmethod
¶
download(station, start_date, end_date, resolution='60', *, cache_dir=None, use_cache=True, nmdb_table='revori', request_timeout=30, proxies=None, try_without_proxy=True, allow_cache_fallback=True)
Download data for one NMDB station.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
station
|
str
|
NMDB station code, for example |
required |
start_date
|
str or datetime - like
|
Inclusive date window to download. |
required |
end_date
|
str or datetime - like
|
Inclusive date window to download. |
required |
resolution
|
int, str, or pandas.Timedelta
|
Temporal resolution. Integer values are interpreted as minutes;
pandas-compatible strings such as |
'60'
|
cache_dir
|
str or Path
|
Cache directory. Neptoon's platform cache directory is used when omitted. |
None
|
use_cache
|
bool
|
Reuse and extend cached station data, by default |
True
|
nmdb_table
|
str
|
NMDB table name, by default |
'revori'
|
request_timeout
|
float
|
HTTPS request timeout in seconds. |
30
|
proxies
|
dict[str, str]
|
Optional HTTPS proxy settings. |
None
|
try_without_proxy
|
bool
|
Retry without environment proxy settings after a proxy failure. |
True
|
allow_cache_fallback
|
bool
|
Return available cached data when a refresh fails. |
True
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
UTC |
fetch_nmdb_data ¶
fetch_nmdb_data(start_date, end_date, station, resolution, nmdb_table='revori', request_timeout=30, proxies=None, try_without_proxy=True)
Return a dataframe of data from nmdb.eu without using the cache.
This compatibility function delegates to :class:NMDBDataDownloader.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
start_date
|
str
|
Start date of desired data, format "YYYY-MM-DD" |
required |
end_date
|
str
|
End date of desired data, format "YYYY-MM-DD" |
required |
station
|
str
|
Desired station as string, as available from https://www.nmdb.eu. E.g., "JUNG" or "OULU" |
required |
resolution
|
int or str
|
The desired resolution in minutes or as a pandas-compatible interval |
required |
nmdb_table
|
str
|
The table to collect from nmdb.eu, by default "revori" |
'revori'
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
Datetime indexed dataframe |
neptoon.external.reference_series¶
reference_series ¶
What a gridded reference column is, independent of the block that filled it.
Neptoon attaches gridded reference series through two configuration blocks:
satellite products under remote_sensing: and ERA5-Land reanalysis under
external_meteo:. The split is deliberate and honest, a land-surface
reanalysis is not a satellite retrieval, but a reader, a plot and a report all
need the same three facts about any of these columns regardless of which block
delivered it: what it is, how it was produced, and at what resolution.
Keeping that lookup here rather than in either block is what lets a figure state provenance in its caption without importing one category and guessing about the other. Nothing here decides anything; it describes columns that already exist.
REMOTE_SENSING_VARIABLE_LABELS
module-attribute
¶
REMOTE_SENSING_VARIABLE_LABELS = {'above_ground_biomass': 'Remote-sensing above-ground biomass', 'canopy_height': 'Remote-sensing canopy height', 'ndvi': 'Remote-sensing NDVI', 'evi': 'Remote-sensing EVI', 'lai': 'Remote-sensing LAI', 'fpar': 'Remote-sensing FPAR', 'soil_moisture': 'Satellite soil moisture', 'soil_water_index': 'Copernicus Soil Water Index', 'snow_cover_fraction': 'Remote-sensing snow-cover fraction', 'land_cover_class': 'Remote-sensing land-cover class'}
REFERENCE_COLUMN_DESCRIPTIONS
module-attribute
¶
REFERENCE_COLUMN_DESCRIPTIONS = {None: _build_era5_land_descriptions(), None: _build_remote_sensing_descriptions()}
ReferenceColumnDescription
dataclass
¶
ReferenceColumnDescription(column, label, provenance, unit, native_resolution_m, product, is_scaled=False)
What one gridded reference column holds and where it came from.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
column
|
str
|
The resolved column name. |
required |
label
|
str
|
Short human-readable name, including a depth range where one applies. |
required |
provenance
|
str
|
Provenance class from the closed vocabulary. |
required |
unit
|
str
|
Native unit as written into the column. |
required |
native_resolution_m
|
float | None
|
Native grid resolution in metres, or |
required |
product
|
str
|
Product or dataset the values came from. |
required |
is_scaled
|
bool
|
Whether this is the adjusted companion of a raw column. |
False
|
is_volumetric_water_content
property
¶
Return whether the values share the CRNS soil-moisture axis.
describe_provenance ¶
Return the provenance phrase used in a legend entry or caption.
describe_for_legend ¶
Return the legend label: what it is, then how it was produced.
describe_for_caption ¶
Return the caption line, which names the product as well.
describe_reference_column ¶
Return the description for one column, or None if it is not one.
Returning None rather than raising lets a caller pass a whole frame's
columns without filtering first, which is what a plot and a report both do.
find_reference_columns ¶
Return the gridded reference value columns present, in a stable order.
Marker and diagnostic columns are excluded: they describe a value rather than being one, so they belong in a table and not on an axis.
find_soil_moisture_reference_columns ¶
Return the reference columns that share the CRNS soil-moisture axis.
describe_reference_provenance_caption ¶
Return a caption stating the provenance of every plotted reference series.
The caption is not decoration. A reanalysis and a satellite retrieval look identical once they are lines on the same axes, and the difference is exactly what a reader needs in order to know how much weight to give the comparison.
One series contributes one caption entry. A raw column and its scaled companion are the same series shown twice, so repeating the provenance for both would pad the caption with a line that says nothing new; the adjustment is noted on the entry instead.
is_reference_column_family ¶
Return whether a column belongs to either reference-series family.
neptoon.external.source_metadata¶
source_metadata ¶
Shared helpers for external meteorological source metadata.
EXTERNAL_SOURCE_PREFIXES
module-attribute
¶
EXTERNAL_SOURCE_PREFIXES = ('dwd_', 'meteo_', 'meteostat_', 'noaa_isd_', 'noaa_ghcn_daily_', 'noaa_uscrn_', 'noaa_nws_', 'inmet_', 'jma_amedas_', 'open_meteo_', 'eobs_', 'meteoswiss_', 'knmi_', 'met_office_datahub_', 'meteo_france_', 'chmi_', 'arso_')
EXTERNAL_SOURCE_DIRECT_COLUMNS
module-attribute
¶
has_external_data_sources ¶
Return True if columns include known DWD/public-meteo source metadata.
neptoon.external_meteo_registry¶
Canonical provider names, aliases, product/variable capabilities, and the stable
region_or_country / description metadata exposed for every external meteorological provider.
external_meteo_registry ¶
Shared registry helpers for external meteorological providers.
The YAML configuration layer and the runtime attachment layer both need the same provider aliases and product compatibility rules. Keeping those rules in one lightweight module avoids subtle drift when new providers are added.
EXTERNAL_METEO_PROVIDER_METADATA
module-attribute
¶
EXTERNAL_METEO_PROVIDER_METADATA = {'nasa_power': ExternalMeteoProviderMetadata(region_or_country='Global', description='NASA POWER hourly gridded point meteorology from the POWER API.'), 'open_meteo': ExternalMeteoProviderMetadata(region_or_country='Global', description='Open-Meteo historical gridded meteorology from the Archive API.'), 'era5_open_meteo': ExternalMeteoProviderMetadata(region_or_country='Global', description='ERA5 reanalysis exposed through the Open-Meteo historical API.'), 'era5_land_open_meteo': ExternalMeteoProviderMetadata(region_or_country='Global land', description='ERA5-Land reanalysis exposed through the Open-Meteo historical API.'), 'meteostat': ExternalMeteoProviderMetadata(region_or_country='Global', description='Meteostat bulk station observations with nearest-station selection.'), 'noaa_isd': ExternalMeteoProviderMetadata(region_or_country='Global', description='NOAA Integrated Surface Database / Global Hourly station observations.'), 'noaa_ghcn_daily': ExternalMeteoProviderMetadata(region_or_country='Global', description='NOAA Global Historical Climatology Network daily station observations.'), 'noaa_uscrn': ExternalMeteoProviderMetadata(region_or_country='United States', description='NOAA U.S. Climate Reference Network high-quality reference observations.'), 'noaa_nws': ExternalMeteoProviderMetadata(region_or_country='United States (primarily CONUS)', description='NOAA/NWS near-real-time surface observations from api.weather.gov.'), 'inmet': ExternalMeteoProviderMetadata(region_or_country='Brazil', description='INMET automatic-station SYNOP observations delivered through WIS2.'), 'jma_amedas': ExternalMeteoProviderMetadata(region_or_country='Japan', description='Japan Meteorological Agency AMeDAS recent point observations.'), 'eobs': ExternalMeteoProviderMetadata(region_or_country='Europe', description='E-OBS daily gridded observations loaded from configured local datasets.'), 'aemet': ExternalMeteoProviderMetadata(region_or_country='Spain', description='AEMET OpenData daily climatological station observations.'), 'meteoswiss': ExternalMeteoProviderMetadata(region_or_country='Switzerland', description='MeteoSwiss station observations from open-data resources or local files.'), 'knmi': ExternalMeteoProviderMetadata(region_or_country='Netherlands', description='KNMI station observations from the Data Platform API or local files.'), 'met_office_datahub': ExternalMeteoProviderMetadata(region_or_country='United Kingdom', description='UK Met Office Weather DataHub land observations or local extracts.'), 'meteo_france': ExternalMeteoProviderMetadata(region_or_country='France', description='Météo-France DPClim station observations via API or local files.'), 'rmi_belgium': ExternalMeteoProviderMetadata(region_or_country='Belgium', description='Royal Meteorological Institute of Belgium token-free WFS observations.'), 'vmm_waterinfo': ExternalMeteoProviderMetadata(region_or_country='Flanders, Belgium', description='VMM Waterinfo/KIWIS meteorological observations from Flemish stations.'), 'fmi': ExternalMeteoProviderMetadata(region_or_country='Finland', description='Finnish Meteorological Institute Open Data observations.'), 'smhi': ExternalMeteoProviderMetadata(region_or_country='Sweden', description='SMHI Open Data station observations combining recent and archive feeds.'), 'imgw': ExternalMeteoProviderMetadata(region_or_country='Poland', description='IMGW-PIB synoptic station observations and station metadata.'), 'met_eireann': ExternalMeteoProviderMetadata(region_or_country='Ireland', description='Met Éireann hourly observations from the public OGC EDR service.'), 'ipma': ExternalMeteoProviderMetadata(region_or_country='Portugal', description='IPMA public automatic-station observations and latest-hourly feeds.'), 'eccc_geomet': ExternalMeteoProviderMetadata(region_or_country='Canada', description='Environment and Climate Change Canada observations through GeoMet.'), 'bom': ExternalMeteoProviderMetadata(region_or_country='Australia', description='Australian Bureau of Meteorology station catalogue and rolling observations.'), 'geosphere_austria': ExternalMeteoProviderMetadata(region_or_country='Austria', description='GeoSphere Austria Data Hub hourly climate station observations.'), 'frost': ExternalMeteoProviderMetadata(region_or_country='Norway', description='MET Norway Frost station observations or configured local files.'), 'dmi': ExternalMeteoProviderMetadata(region_or_country='Denmark', description='Danish Meteorological Institute Open Data station observations.'), 'dwd': ExternalMeteoProviderMetadata(region_or_country='Germany', description='DWD Climate Data Center Open Data station observations.'), 'chmi': ExternalMeteoProviderMetadata(region_or_country='Czech Republic', description='Czech Hydrometeorological Institute open station observations.'), 'arso': ExternalMeteoProviderMetadata(region_or_country='Slovenia', description='Slovenian Environment Agency automatic-station observations from meteo.si.')}
EXTERNAL_METEO_PROVIDER_CREDENTIALS
module-attribute
¶
EXTERNAL_METEO_PROVIDER_CREDENTIALS = {'aemet': (ExternalMeteoCredentialMetadata(field_name='aemet_api_key', env_field_name='aemet_api_key_env', default_env_var='NEPTOON_AEMET_API_KEY', label='AEMET OpenData API key', required_for_remote_api=True, description='API key for remote AEMET OpenData access.'),), 'knmi': (ExternalMeteoCredentialMetadata(field_name='knmi_api_key', env_field_name='knmi_api_key_env', default_env_var='NEPTOON_KNMI_API_KEY', label='KNMI API key', required_for_remote_api=True, description='API key for remote KNMI Data Platform access.'),), 'met_office_datahub': (ExternalMeteoCredentialMetadata(field_name='met_office_api_key', env_field_name='met_office_api_key_env', default_env_var='NEPTOON_MET_OFFICE_API_KEY', label='Met Office DataHub API key', required_for_remote_api=True, description='API key for remote UK Met Office Weather DataHub access.'),), 'meteo_france': (ExternalMeteoCredentialMetadata(field_name='meteo_france_api_key', env_field_name='meteo_france_api_key_env', default_env_var='NEPTOON_METEO_FRANCE_API_KEY', label='Météo-France API key', required_for_remote_api=True, description='API key for remote Météo-France DPClim access.'),), 'frost': (ExternalMeteoCredentialMetadata(field_name='frost_client_id', env_field_name='frost_client_id_env', default_env_var='NEPTOON_FROST_CLIENT_ID', label='MET Norway Frost client id', required_for_remote_api=True, description='Frost client id used as the Basic-auth username for API access.'),)}
SUPPORTED_EXTERNAL_METEO_PROVIDERS
module-attribute
¶
SUPPORTED_EXTERNAL_METEO_PROVIDERS = frozenset(EXTERNAL_METEO_PROVIDER_METADATA)
EXTERNAL_METEO_PROVIDER_ALIASES
module-attribute
¶
EXTERNAL_METEO_PROVIDER_ALIASES = {'noaa': 'noaa_isd', 'isd': 'noaa_isd', 'noaa_global_hourly': 'noaa_isd', 'noaa_isd_global_hourly': 'noaa_isd', 'ghcn': 'noaa_ghcn_daily', 'ghcnd': 'noaa_ghcn_daily', 'ghcn_daily': 'noaa_ghcn_daily', 'noaa_daily': 'noaa_ghcn_daily', 'noaa_ghcnd': 'noaa_ghcn_daily', 'noaa_ghcn': 'noaa_ghcn_daily', 'uscrn': 'noaa_uscrn', 'noaa_crn': 'noaa_uscrn', 'noaa_us_climate_reference_network': 'noaa_uscrn', 'nws': 'noaa_nws', 'noaa_nws': 'noaa_nws', 'weather_gov': 'noaa_nws', 'api_weather_gov': 'noaa_nws', 'brazil_inmet': 'inmet', 'inmet_brazil': 'inmet', 'br_inmet': 'inmet', 'jma': 'jma_amedas', 'amedas': 'jma_amedas', 'jma_amedas_japan': 'jma_amedas', 'meteostat_bulk': 'meteostat', 'power': 'nasa_power', 'nasa_power_api': 'nasa_power', 'nasa_power_hourly': 'nasa_power', 'aemet_open_data': 'aemet', 'aemet_opendata': 'aemet', 'spain_aemet': 'aemet', 'openmeteo': 'open_meteo', 'open_meteo_archive': 'open_meteo', 'open_meteo_historical': 'open_meteo', 'era5': 'era5_open_meteo', 'era5_open_meteo': 'era5_open_meteo', 'era5_land': 'era5_land_open_meteo', 'era5_land_open_meteo': 'era5_land_open_meteo', 'ecad': 'eobs', 'eca_d': 'eobs', 'eca&d': 'eobs', 'e_obs': 'eobs', 'meteoschweiz': 'meteoswiss', 'meteo_swiss': 'meteoswiss', 'swissmetnet': 'meteoswiss', 'knmi_open_data': 'knmi', 'knmi_dataplatform': 'knmi', 'metoffice': 'met_office_datahub', 'met_office': 'met_office_datahub', 'metoffice_datahub': 'met_office_datahub', 'met_office_weather_datahub': 'met_office_datahub', 'weather_datahub': 'met_office_datahub', 'meteofrance': 'meteo_france', 'meteo_fr': 'meteo_france', 'meteo_france_dpclim': 'meteo_france', 'dpclim': 'meteo_france', 'kmi_be': 'rmi_belgium', 'rmi': 'rmi_belgium', 'irm': 'rmi_belgium', 'vmm': 'vmm_waterinfo', 'waterinfo': 'vmm_waterinfo', 'waterinfo_be': 'vmm_waterinfo', 'vmm_waterinfo_be': 'vmm_waterinfo', 'fmi_open_data': 'fmi', 'finnish_meteorological_institute': 'fmi', 'ipma_open_data': 'ipma', 'instituto_portugues_do_mar_e_da_atmosfera': 'ipma', 'portugal_ipma': 'ipma', 'msc_geomet': 'eccc_geomet', 'geomet': 'eccc_geomet', 'eccc': 'eccc_geomet', 'environment_canada': 'eccc_geomet', 'weather_canada': 'eccc_geomet', 'bureau_of_meteorology': 'bom', 'australian_bureau_of_meteorology': 'bom', 'geosphere': 'geosphere_austria', 'geosphere_at': 'geosphere_austria', 'geosphere_austria_data_hub': 'geosphere_austria', 'zamg': 'geosphere_austria', 'zamg_datahub': 'geosphere_austria', 'met_norway': 'frost', 'met_norway_frost': 'frost', 'metno': 'frost', 'metno_frost': 'frost', 'dmi_open_data': 'dmi', 'danish_meteorological_institute': 'dmi', 'dwd_open_data': 'dwd', 'deutscher_wetterdienst': 'dwd', 'germany_dwd': 'dwd', 'chmu': 'chmi', 'čhmú': 'chmi', 'czech_hydrometeorological_institute': 'chmi', 'czech_hydrometeorological_institute_chmi': 'chmi', 'meteosi': 'arso', 'meteo.si': 'arso', 'meteo_si': 'arso', 'arso_meteo': 'arso', 'slovenian_environment_agency': 'arso', 'slovenian_environment_agency_arso': 'arso', 'smhi_open_data': 'smhi', 'swedish_meteorological_and_hydrological_institute': 'smhi', 'swedish_meteorological_hydrological_institute': 'smhi', 'imgw_pib': 'imgw', 'imgw_open_data': 'imgw', 'polish_imgw': 'imgw', 'met_eireann_open_data': 'met_eireann', 'meteireann': 'met_eireann', 'met_ie': 'met_eireann', 'irish_meteorological_service': 'met_eireann'}
EXTERNAL_METEO_PRODUCT_ALIASES
module-attribute
¶
EXTERNAL_METEO_PRODUCT_ALIASES = {'global_hourly': 'hourly', 'isd_hourly': 'hourly', 'ghcn_daily': 'daily', 'ghcnd_daily': 'daily', 'noaa_ghcn_daily': 'daily', 'uscrn_hourly': 'hourly', 'noaa_uscrn_hourly': 'hourly', 'uscrn_daily': 'daily', 'noaa_uscrn_daily': 'daily', 'nws_hourly': 'hourly', 'noaa_nws_hourly': 'hourly', 'inmet_hourly': 'hourly', 'inmet_synop_hourly': 'hourly', 'jma_hourly': 'hourly', 'amedas_hourly': 'hourly', 'jma_amedas_hourly': 'hourly', 'meteostat_hourly': 'hourly', 'nasa_power_hourly': 'hourly', 'power_hourly': 'hourly', 'aemet_daily': 'daily', 'open_meteo_hourly': 'hourly', 'openmeteo_hourly': 'hourly', 'era5_hourly': 'hourly', 'era5_land_hourly': 'hourly', 'eobs_daily': 'daily', 'e_obs_daily': 'daily', 'meteoswiss_hourly': 'hourly', 'swissmetnet_hourly': 'hourly', 'knmi_hourly': 'hourly', 'knmi_daily': 'daily', 'met_office_hourly': 'hourly', 'metoffice_hourly': 'hourly', 'meteo_france_hourly': 'hourly', 'meteofrance_hourly': 'hourly', 'dpclim_hourly': 'hourly', 'meteo_france_daily': 'daily', 'meteofrance_daily': 'daily', 'dpclim_daily': 'daily', 'rmi_belgium_hourly': 'hourly', 'kmi_hourly': 'hourly', 'rmi_hourly': 'hourly', 'irm_hourly': 'hourly', 'vmm_hourly': 'hourly', 'waterinfo_hourly': 'hourly', 'vmm_waterinfo_hourly': 'hourly', 'fmi_hourly': 'hourly', 'ipma_hourly': 'hourly', 'ipma_open_data_hourly': 'hourly', 'eccc_hourly': 'hourly', 'eccc_geomet_hourly': 'hourly', 'bom_hourly': 'hourly', 'geosphere_hourly': 'hourly', 'geosphere_austria_hourly': 'hourly', 'frost_hourly': 'hourly', 'met_norway_hourly': 'hourly', 'dmi_hourly': 'hourly', 'dmi_open_data_hourly': 'hourly', 'dwd_hourly': 'hourly', 'dwd_open_data_hourly': 'hourly', 'chmi_hourly': 'hourly', 'chmi_daily': 'daily', 'chmu_hourly': 'hourly', 'chmu_daily': 'daily', 'arso_hourly': 'hourly', 'arso_daily': 'daily', 'meteo_si_hourly': 'hourly', 'meteo_si_daily': 'daily', 'smhi_hourly': 'hourly', 'smhi_open_data_hourly': 'hourly', 'imgw_hourly': 'hourly', 'imgw_open_data_hourly': 'hourly', 'met_eireann_hourly': 'hourly', 'met_eireann_open_data_hourly': 'hourly'}
DAILY_ONLY_EXTERNAL_METEO_PROVIDERS
module-attribute
¶
HOURLY_OR_DAILY_EXTERNAL_METEO_PROVIDERS
module-attribute
¶
HOURLY_OR_DAILY_EXTERNAL_METEO_PROVIDERS = frozenset({'knmi', 'meteo_france', 'noaa_uscrn', 'chmi', 'arso'})
EXTERNAL_METEO_VARIABLES
module-attribute
¶
EXTERNAL_METEO_VARIABLES = frozenset({'pressure', 'temperature', 'relative_humidity', 'precipitation', 'solar_radiation', 'wind_speed', 'reference_et0'})
EXTERNAL_METEO_PROVIDER_VARIABLES
module-attribute
¶
EXTERNAL_METEO_PROVIDER_VARIABLES = {'aemet': _TEMP_HUMIDITY_PRECIP | {'wind_speed'}, 'arso': _CORE_WITH_SOLAR_AND_WIND, 'bom': _CORE_WITH_SOLAR_AND_WIND, 'chmi': _CORE_WITH_SOLAR_AND_WIND, 'dmi': _CORE_WITH_SOLAR_AND_WIND, 'dwd': _CORE_WITH_SOLAR, 'eccc_geomet': _CORE_WITH_SOLAR, 'eobs': _TEMP_HUMIDITY_PRECIP | {'solar_radiation'}, 'era5_land_open_meteo': frozenset({'pressure', 'temperature', 'relative_humidity', 'reference_et0'}), 'era5_open_meteo': EXTERNAL_METEO_VARIABLES, 'fmi': _CORE_WITH_SOLAR, 'frost': _CORE_WITH_WIND, 'geosphere_austria': _CORE_WITH_SOLAR_AND_WIND, 'imgw': frozenset({'pressure', 'temperature', 'relative_humidity', 'wind_speed'}), 'inmet': _CORE_WITH_WIND, 'ipma': _CORE_WITH_SOLAR_AND_WIND, 'jma_amedas': _CORE_WITH_WIND, 'knmi': _CORE_WITH_SOLAR, 'nasa_power': _CORE_WITH_SOLAR_AND_WIND, 'met_eireann': _CORE_WITH_SOLAR_AND_WIND, 'met_office_datahub': _CORE_METEO_VARIABLES, 'meteo_france': _CORE_WITH_SOLAR, 'meteostat': _CORE_METEO_VARIABLES, 'meteoswiss': _CORE_WITH_SOLAR, 'noaa_ghcn_daily': _CORE_METEO_VARIABLES, 'noaa_isd': _CORE_METEO_VARIABLES, 'noaa_nws': _CORE_WITH_WIND, 'noaa_uscrn': _TEMP_HUMIDITY_PRECIP | {'solar_radiation'}, 'open_meteo': EXTERNAL_METEO_VARIABLES, 'rmi_belgium': _CORE_WITH_SOLAR, 'smhi': _TEMP_HUMIDITY_PRECIP | {'wind_speed'}, 'vmm_waterinfo': _CORE_METEO_VARIABLES}
ExternalMeteoProviderMetadata
dataclass
¶
ExternalMeteoCredentialMetadata
dataclass
¶
ExternalMeteoCredentialMetadata(field_name, env_field_name, default_env_var, label, required_for_remote_api, description)
Describe one runtime credential accepted by an external meteo provider.
field_name is the explicit secret field accepted by configuration and
runtime APIs. env_field_name names the non-secret config field that can
point at an environment variable instead. default_env_var is the
conventional variable a UI or deployment tool can offer without hardcoding
provider knowledge outside the core registry.
normalize_external_meteo_token ¶
Normalize provider/product tokens before alias lookup.
normalize_external_meteo_provider ¶
Normalize external provider names and validate provider support.
get_external_meteo_provider_metadata ¶
Return region/country and description for one provider or alias.
get_external_meteo_provider_credentials ¶
Return runtime credential metadata for one provider or alias.
Providers with anonymous/token-free APIs return an empty tuple. This is a stable core discovery API intended for CLIs, GUIs and deployment tooling.
get_external_meteo_provider_region_or_country ¶
Return the documented primary region or country for one provider.
get_external_meteo_provider_description ¶
Return the concise provider description used by code and documentation.
get_supported_external_meteo_variables ¶
Return canonical variables supported by one provider.
external_meteo_provider_supports_variable ¶
Return whether a provider exposes one canonical meteorological variable.
get_default_external_meteo_product ¶
Return the default product for a normalized or alias provider name.
normalize_external_meteo_product ¶
Normalize product aliases and optionally validate provider compatibility.
keep_auto is useful in the Pydantic field validator where the provider
is not yet guaranteed to be available. Runtime code passes a provider and
receives the effective concrete product instead.
validate_external_meteo_product_for_provider ¶
Validate a normalized product for a normalized or alias provider.
neptoon.external._http_cache¶
_http_cache ¶
Shared HTTP download cache for Neptoon's external data categories.
These helpers were grown inside :mod:neptoon.external.meteo.common while
meteorology was the only category that downloaded provider payloads. Remote
sensing needs exactly the same behaviour, a per-provider cache subdirectory,
freshness and stale-fallback policy from the environment, atomic publication of
downloaded bytes, and UTC clipping of the resulting frame, so the mechanics
live here instead of being reached for across a category boundary.
The module is deliberately category-neutral: the caller supplies the user-facing service label and the exception type used for its own boundary, so error messages keep naming the provider the user actually configured.
CACHE_MAX_AGE_ENV_VARIABLE
module-attribute
¶
ALLOW_STALE_CACHE_ENV_VARIABLE
module-attribute
¶
SupportsRequest ¶
Bases: Protocol
Protocol for the retrying HTTP request helper used by downloads.
clip_frame_to_utc_range ¶
Clip a time-indexed dataframe to an inclusive UTC time range.
get_cache_subdir ¶
Return the per-provider cache directory, creating it when possible.
A cache is an optimisation, so a directory that cannot be created is reported as a warning and the caller proceeds without caching.
get_cache_policy ¶
Return the cache freshness limit in hours and the stale-fallback flag.
effective_cache_max_age_hours ¶
Return the strictest configured cache age limit.
NEPTOON_CACHE_MAX_AGE_HOURS remains the global operator override. A
mutable provider product can additionally declare a conservative built-in
maximum age, so that a default cache never turns a rolling feed into an
immutable snapshot.
can_use_stale_cache_after ¶
Return whether a stale cache is safe to serve after this failure.
Stale data is a resilience fallback for transient connectivity and service failures only. Authentication, authorization, missing-resource and other deterministic client errors must surface instead of being hidden behind an old payload that no longer describes what the endpoint holds.
read_cache_candidate ¶
read_cache_candidate(cache_file, *, error_factory=None, service='External provider', max_age_hours=None, max_response_bytes=HTTP_CACHE_MAX_RESPONSE_BYTES, read_failures=None)
Return cached bytes and whether they are fresh under the cache policy.
An empty cache file is treated as a failed earlier write: it is moved aside
with a .corrupt-<epoch> suffix so the evidence survives, and the caller
downloads again.
A cache entry is optional by definition, so an unreadable one is reported
and stepped over rather than raised: the download that follows is the real
answer, and turning a permission problem on a temporary file into a hard
failure would deny a user data the provider is willing to serve. The reason
is appended to read_failures when the caller supplies one, so that a
download failing too can say both things at once.
download_to_cache ¶
download_to_cache(url, cache_file, request_timeout, use_cache, *, service, error_factory, headers=None, params=None, requester=None, max_age_hours=None, max_response_bytes=HTTP_CACHE_MAX_RESPONSE_BYTES)
Download a URL through the shared cache and return its bytes.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
url
|
str
|
Absolute URL to fetch with a retrying GET. |
required |
cache_file
|
Path
|
Cache destination. Written atomically, and read first when
|
required |
request_timeout
|
float
|
HTTP timeout in seconds. |
required |
use_cache
|
bool
|
Read from and write to |
required |
service
|
str
|
User-facing label for the calling category, used in error messages. |
required |
error_factory
|
ErrorFactory
|
Builds the caller's boundary exception from a message. |
required |
headers
|
dict[str, str] | None
|
Optional request headers and query parameters. |
None
|
params
|
dict[str, str] | None
|
Optional request headers and query parameters. |
None
|
requester
|
SupportsRequest | None
|
Optional request callable. Defaults to the shared retrying helper; callers pass their own so existing monkeypatch surfaces keep working. |
None
|
max_age_hours
|
float | None
|
Optional per-product age limit for a mutable feed, applied on top of the global policy. The stricter of the two wins. |
None
|
max_response_bytes
|
int
|
Maximum accepted response or cache payload size. Defaults to 3 GB to keep malformed or unexpectedly large provider responses bounded. |
HTTP_CACHE_MAX_RESPONSE_BYTES
|
Returns:
| Type | Description |
|---|---|
bytes
|
Response body, or a stale cached body when the download failed and stale fallback is permitted. |
neptoon.external.remote_sensing.attachment¶
attachment ¶
Dispatch remote-sensing providers and join their series onto the CRNS index.
The CRNS index is master. Remote-sensing series are joined onto it and the CRNS series is never resampled down to a product's cadence.
Two things happen here that are easy to conflate and must not be:
- Filling within the product's own support. A daily value holds for its day; spreading it across that day's hourly timestamps is what the product says, not an assumption, so it carries no marker.
- Filling gaps between observations. For a 16-day composite the days in
between were not measured. Covering them is a modelling decision, permitted
only where the catalog says so, bounded by
max_gap, and always marked.
The order is fixed as quality filter, then gap, then gap filling, then marker. A value the provider's own quality band rejects becomes a gap and is then treated like any other gap. That is the only order in which the marker stays honest: "interpolated" then means "no usable observation was here", whether it was missing or rejected. The reverse order would push interpolated values through a quality test built for observations.
Providers stay duck-typed. There is no base class and no shared fetch_*
signature, because how the data is retrieved is exactly where providers differ.
What is shared is a registry and a request/result shape at the configuration
level, so a new product becomes a registry entry rather than a change here.
ProviderFetcher
module-attribute
¶
ProviderFetcher = Callable[[RemoteSensingFetchRequest], RemoteSensingFetchResult]
RemoteSensingFetchRequest
dataclass
¶
RemoteSensingFetchRequest(product, variables, latitudes, longitudes, footprint_radius_m, start, end, extraction='footprint', epoch=None, substitute_epochs=(), bracketing_epochs=(), cache_dir=None, use_cache=True, request_timeout=30.0)
What a provider is asked for.
The coordinate collections may describe the spatial extent a provider has to inspect, but the current fetch-result model is still a station time series. Any provider that reduces to one scalar must therefore require exactly one effective target position. Future rover support needs timestamp-position targets and a multi-position result model rather than silently using point 0.
CompanionColumn
dataclass
¶
A second column a provider produces alongside a semantic variable.
Not every product reduces to one number. A biomass product shipping a
per-pixel uncertainty band produces a value and its uncertainty; a land-cover
product produces a class and the share of each class in the footprint.
Neither is a semantic variable in its own right, nobody configures
variables: [land_cover_fraction_tree_cover], and neither may be invented
by the attachment layer, because only the provider knows whether the product
publishes it.
A companion is joined onto the CRNS index exactly like its parent variable, under the parent's gap-filling policy, and it inherits the parent's markers rather than growing its own: the markers say how the observation got to a timestamp, and that answer is the same for both.
RemoteSensingFetchResult
dataclass
¶
RemoteSensingFetchResult(data_frame, provenance, observations=(), artefacts=(), companion_columns=(), valid_fraction=None, warnings=(), findings=())
What a provider returns.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data_frame
|
DataFrame
|
Observations indexed by UTC timestamp, one column per semantic variable plus any companion columns. Only genuine observations belong here: gap filling happens downstream, so a provider must not pre-fill. |
required |
observations
|
tuple[CoverageObservation, ...]
|
Per-timestamp coverage results, which are stage 2 of the spatial check. |
()
|
provenance
|
RemoteSensingProvenanceRecord
|
Record making a rerun verifiable. Where a fetch rests on several
artefacts, this is the first of |
required |
artefacts
|
tuple[RemoteSensingProvenanceRecord, ...]
|
One provenance record per fetched artefact. An annual raster product
contributes one file per epoch and an uncertainty band is a second file,
so "which files produced this column" needs more than one record. Left
empty, |
()
|
companion_columns
|
tuple[CompanionColumn, ...]
|
Extra columns to join alongside the variable. |
()
|
valid_fraction
|
float | None
|
Footprint valid fraction, written as a column only where the catalog says nodata means the phenomenon is absent. |
None
|
warnings
|
tuple[str, ...]
|
Provider-level notes for the report and the audit trail. |
()
|
findings
|
tuple[str, ...]
|
Provider-level notes for the report and the audit trail. |
()
|
provenance_records
property
¶
Return every artefact behind this result, never an empty tuple.
RemoteSensingAttachmentResult
dataclass
¶
RemoteSensingAttachmentResult(data_frame, value_columns=(), marker_columns=(), column_aggregations=dict(), provenance=(), selections=dict(), report=None, warnings=(), findings=())
Outcome of attaching one remote-sensing configuration.
column_aggregations
class-attribute
instance-attribute
¶
RemoteSensingDataAttacher ¶
RemoteSensingDataAttacher(data_frame, config, *, configured_tiers=(TOKEN_FREE_TIER,), catalog=None)
Resolve, fetch and join one remote-sensing configuration.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data_frame
|
DataFrame
|
Current CRNS frame. Its index defines the target time axis. |
required |
config
|
RemoteSensingAttachmentConfig
|
Attachment configuration including the site coordinates. |
required |
configured_tiers
|
Iterable[str]
|
Credential tiers usable without prompting the user. |
(TOKEN_FREE_TIER,)
|
catalog
|
dict[str, RemoteSensingProduct] | None
|
Catalog to resolve against, including user-declared products. |
None
|
catalog
instance-attribute
¶
catalog = get_catalog(include_user_declared=local_products) if catalog is None else catalog
target_index
property
¶
Return the CRNS time index, in UTC, that everything joins onto.
register_provider_fetcher ¶
Register the fetch callable for one provider.
Providers register themselves as their modules land, so adding a product is a registry entry rather than an edit to the dispatch code.
unregister_provider_fetcher ¶
Remove a provider registration. Primarily for tests.
get_registered_providers ¶
Return the providers that currently have a fetcher.
get_provider_fetcher ¶
Return the fetch callable for one provider.
Raises:
| Type | Description |
|---|---|
RemoteSensingDataError
|
When no fetcher is registered. The message says which providers are available rather than failing with an attribute error. |
attach_remote_sensing_data ¶
attach_remote_sensing_data(data_frame, config, *, configured_tiers=(TOKEN_FREE_TIER,), catalog=None)
Attach one remote-sensing configuration to a CRNS frame.
describe_multi_sensor_selections ¶
Return warnings where one variable resolved to different products.
Automatic selection can legitimately resolve differently per station, and that confounds cross-station comparison: different resolution, different retrieval, different bias. Logging it is not optional, so the disagreement is named with the stations involved.
select_consistent_products ¶
Return one product per variable that every station can use.
The opt-in for network studies. It is deliberately not the default, because forcing consistency costs resolution at the stations that had a better regional product available, and that cost should be a decision rather than a silent preference.
neptoon.external.remote_sensing.catalog¶
catalog ¶
Declarative product catalog for remote-sensing reference data.
The catalog stores metadata, not data: a handful of numbers and strings per
product saying where it is valid, when, how fine, how one reaches it and how it
may be interpreted. No raster tiles live here. The values are what makes the
preflight in :mod:.eligibility possible before anything is downloaded.
Three properties are deliberate:
- It is data, not an interface. Adding a product is a catalog entry, not a change to existing code, and nothing inherits from anything.
- It does not need editing when a provider extends its time series.
Available epochs are discovered from the provider at runtime and cached; the
catalog carries the declared extent, and a discovered extent overrides it.
What genuinely cannot be discovered (licence, citation,
nodata_meaning, provenance class) stays here and is a release-checklist review item. - Where declared metadata and observed data disagree, the mismatch is a logged finding rather than a silent discard. Two real cases motivated this: a collection declaring an open-ended temporal extent while its items stop years earlier, and a query returning an item whose pixels are entirely nodata. Each says either the provider or our understanding is wrong.
Authority is unchanged throughout: metadata informs, pre-filters and cross-checks, but pixels decide coverage.
REMOTE_SENSING_PRODUCTS
module-attribute
¶
REMOTE_SENSING_PRODUCTS = {'dlr_forest_structure_agbd': RemoteSensingProduct(product='dlr_forest_structure_agbd', provider='dlr', variables=('above_ground_biomass',), correction_inputs=frozenset({'above_ground_biomass'}), extraction='footprint', spatial=_DLR_SPATIAL, temporal_start=Timestamp('2017-01-01'), temporal_end=None, observed_temporal_end=Timestamp('2023-12-31'), available_epochs=tuple(range(2017, 2024)), cadence='P1Y', nodata_meaning='phenomenon_absent', absent_value=0.0, gap_filling='linear', max_gap=None, extrapolation='nearest_epoch', native_resolution_m=10.0, provenance='satellite_derived', credential_tier='token_free', citation='Kacic, P., Thonfeld, F., Gessner, U., Kuenzer, C. (2023). Forest Structure Characterization in Germany: Novel Products and Analysis Based on GEDI, Sentinel-1 and Sentinel-2 Data. Remote Sensing 15, 1969. https://doi.org/10.3390/rs15081969 [collection FOREST_STRUCTURE_DE_AGBD_P1Y]', license='CC-BY-4.0', literature_key='dlr_forest_structure_de', unit='Mg/ha', bands=(BandValidity(band='agbd', fill_value=-9999.0, valid_min=0.0, units='Mg/ha'),), notes=('Forest product: farmland and urban footprints return nodata, which states that no forest is present rather than that the product is unavailable. The vegetation indices describe the non-forest part.', 'EPSG:4326 with metrically non-square pixels: about 10.00 m north-south and 6.14 m east-west at 52 degrees north.', 'No uncertainty band is published for this asset.', 'Coverage stops short of the national border in places, and the Baltic islands are missing entirely: measured on the 2023 map, a 4 km box over Jasmund National Park on Ruegen holds no valid pixel at all. The extent is therefore the measured data extent rather than Germany, and a station there is reported as uncovered instead of receiving a biomass of zero.')), 'esa_cci_biomass_v6': RemoteSensingProduct(product='esa_cci_biomass_v6', provider='esa_cci_biomass', variables=('above_ground_biomass',), correction_inputs=frozenset({'above_ground_biomass'}), extraction='footprint', spatial=create_global(description='global land'), temporal_start=Timestamp('2007-01-01'), temporal_end=Timestamp('2022-12-31'), available_epochs=(2007, 2010, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022), cadence='P1Y', nodata_meaning='outside_coverage', gap_filling='linear', max_gap=None, extrapolation='nearest_epoch', native_resolution_m=100.0, provenance='satellite_derived', credential_tier='token_free', citation='Santoro, M., Cartus, O. (2025). ESA Biomass Climate Change Initiative (Biomass_cci): Global datasets of forest above-ground biomass for the years 2007, 2010, 2015-2022, v6.0. NERC EDS Centre for Environmental Data Analysis. https://doi.org/10.5285/95913ffb6467447ca72c4e9d8cf30501', license='esa-cci-data-policy', literature_key='esa_cci_biomass_v6', unit='Mg/ha', bands=(BandValidity(band='AGB', fill_value=None, valid_min=0.0, units='Mg/ha'), BandValidity(band='AGB_SD', fill_value=None, valid_min=0.0, units='Mg/ha')), notes=('Ships a per-pixel uncertainty layer (AGB_SD), which the biomass product from the national provider does not.', 'Uncertainty propagates under full spatial correlation: the footprint uncertainty is the weighted mean of the pixel standard deviations, not sigma over the square root of n.', 'Tile identifiers name the north-west corner of the tile.', 'Open water is not separable by pixel value. Measured on 2026-08-24: a pure-ocean tile does not exist and answers HTTP 404, which is a clean coverage answer, but inside a coastal tile the sea reads as zero with every pixel flagged valid. With no nodata value and zero a legal biomass, a wholly-zero footprint is reported as an ambiguity rather than resolved.')), 'esa_worldcover_v200': RemoteSensingProduct(product='esa_worldcover_v200', provider='esa_worldcover', variables=('land_cover_class',), spatial=create_global(description='global land'), temporal_start=Timestamp('2021-01-01'), temporal_end=Timestamp('2021-12-31'), available_epochs=(2021,), cadence='P1Y', nodata_meaning='outside_coverage', gap_filling='step', extrapolation='epoch_map', native_resolution_m=10.0, provenance='satellite_retrieval', credential_tier='token_free', citation='Zanaga, D., Van De Kerchove, R., Daems, D., De Keersmaecker, W., Brockmann, C., Kirches, G., Wevers, J., Cartus, O., Santoro, M., Fritz, S., Lesiv, M., Herold, M., Tsendbazar, N. E., Xu, P., Ramoino, F., Arino, O. (2022). ESA WorldCover 10 m 2021 v200. https://doi.org/10.5281/zenodo.7254221', license='CC-BY-4.0', literature_key='esa_worldcover_v200', unit='class_code', discovers_epochs_at_runtime=False, bands=(BandValidity(band='Map', fill_value=0.0, valid_min=10.0, valid_max=100.0),), notes=('Informative only: the class enters no computation anywhere in Neptoon.', 'Reachable anonymously over public object storage; no account, token or catalogue query is required.', "'exact' is meaningless for this product: the 2021 map *is* the map.", "Using a map as a layer in a published figure additionally requires the attribution '(c) ESA WorldCover project <year> / Contains modified Copernicus Sentinel data (<year>) processed by ESA WorldCover consortium'. Citing the dataset and attributing a rendered layer are different duties.", 'EPSG:4326 with metrically non-square pixels: about 9.28 m north-south and 5.70 m east-west at 52 degrees north.')), 'esa_worldcover_v100': RemoteSensingProduct(product='esa_worldcover_v100', provider='esa_worldcover', variables=('land_cover_class',), spatial=create_global(description='global land'), temporal_start=Timestamp('2020-01-01'), temporal_end=Timestamp('2020-12-31'), available_epochs=(2020,), cadence='P1Y', nodata_meaning='outside_coverage', gap_filling='step', extrapolation='epoch_map', native_resolution_m=10.0, provenance='satellite_retrieval', credential_tier='token_free', citation='Zanaga, D., Van De Kerchove, R., De Keersmaecker, W., Souverijns, N., Brockmann, C., Quast, R., Wevers, J., Grosu, A., Paccini, A., Vergnaud, S., Cartus, O., Santoro, M., Fritz, S., Georgieva, I., Lesiv, M., Carter, S., Herold, M., Li, Linlin, Tsendbazar, N. E., Ramoino, F., Arino, O. (2021). ESA WorldCover 10 m 2020 v100. https://doi.org/10.5281/zenodo.5571936', license='CC-BY-4.0', literature_key='esa_worldcover_v100', unit='class_code', discovers_epochs_at_runtime=False, bands=(BandValidity(band='Map', fill_value=0.0, valid_min=10.0, valid_max=100.0),), priority=-1, notes=('Earlier WorldCover epoch, kept so a study covering 2020 can pin the map that was current then.', "Using a map as a layer in a published figure additionally requires the attribution '(c) ESA WorldCover project <year> / Contains modified Copernicus Sentinel data (<year>) processed by ESA WorldCover consortium'. Citing the dataset and attributing a rendered layer are different duties.")), 'modis_ornl_mod13q1': RemoteSensingProduct(product='modis_ornl_mod13q1', provider='modis_ornl', variables=('ndvi', 'evi'), spatial=create_global(description='global'), temporal_start=Timestamp('2000-02-18'), temporal_end=None, cadence='P16D', support='P1D', nodata_meaning='masked', gap_filling='linear', max_gap='P32D', extrapolation='forbidden', native_resolution_m=250.0, provenance='satellite_retrieval', credential_tier='token_free', citation='Didan, K. (2021). MODIS/Terra Vegetation Indices 16-Day L3 Global 250 m SIN Grid V061. NASA Land Processes Distributed Active Archive Center. https://doi.org/10.5067/MODIS/MOD13Q1.061 [subsets served by the ORNL DAAC MODIS/VIIRS subsetting service]', license='nasa-earthdata-open-access', literature_key='modis_mod13q1', unit='dimensionless', bands=(BandValidity(band='250m_16_days_NDVI', fill_value=-3000.0, valid_min=-2000.0, valid_max=10000.0, scale_factor=0.0001), BandValidity(band='250m_16_days_EVI', fill_value=-3000.0, valid_min=-2000.0, valid_max=10000.0, scale_factor=0.0001), BandValidity(band='250m_16_days_pixel_reliability', fill_value=-1.0, valid_min=0.0, valid_max=3.0), BandValidity(band='250m_16_days_composite_day_of_the_year', fill_value=-1.0, valid_min=1.0, valid_max=366.0)), notes=('A maximum-value composite, not a 16-day mean: the value is deliberately the upper envelope of the period. The day-of-year band corrects the timestamp but not this selection bias.', 'The composite day-of-year band places each value on its actual acquisition day instead of the window start, which is worth up to 15 days of interpolation accuracy and costs nothing.', 'Fill values differ per band within this product, and the provider publishes them as strings with the valid range as free text. Comparing an integer cell against a string is always unequal, which reports open ocean as fully covered.', 'Requesting a subset without naming a band returns blue reflectance, whose fill value differs again, so the mistake survives as data. This is why configuration names variables semantically.')), 'modis_ornl_mcd15a3h': RemoteSensingProduct(product='modis_ornl_mcd15a3h', provider='modis_ornl', variables=('lai', 'fpar'), spatial=create_global(description='global'), temporal_start=Timestamp('2002-07-04'), temporal_end=None, cadence='P4D', support='P1D', nodata_meaning='masked', gap_filling='linear', max_gap='P8D', extrapolation='forbidden', native_resolution_m=500.0, provenance='satellite_retrieval', credential_tier='token_free', citation='Myneni, R., Knyazikhin, Y., Park, T. (2021). MODIS/Terra+Aqua Leaf Area Index/FPAR 4-Day L4 Global 500 m SIN Grid V061. NASA Land Processes Distributed Active Archive Center. https://doi.org/10.5067/MODIS/MCD15A3H.061 [subsets served by the ORNL DAAC MODIS/VIIRS subsetting service]', license='nasa-earthdata-open-access', literature_key='modis_mcd15a3h', unit='m2/m2', bands=(BandValidity(band='Lai_500m', fill_value=255.0, valid_min=0.0, valid_max=100.0, scale_factor=0.1, units='m2/m2'), BandValidity(band='Fpar_500m', fill_value=255.0, valid_min=0.0, valid_max=100.0, scale_factor=0.01), BandValidity(band='FparLai_QC', fill_value=255.0, valid_min=0.0, valid_max=254.0)), notes=('A four-day cadence makes linear gap filling uncontroversial and is standard practice in phenology, but the land-use caveat on mowing and harvest still applies.',)), 'copernicus_lcm10': RemoteSensingProduct(product='copernicus_lcm10', provider='local_raster', variables=('land_cover_class',), spatial=create_from_bounding_box(-31.5, 27.0, 45.0, 72.0, description='Europe (declared)', is_precise=False), temporal_start=Timestamp('2021-01-01'), temporal_end=Timestamp('2021-12-31'), available_epochs=(2021,), cadence='P1Y', nodata_meaning='outside_coverage', gap_filling='step', extrapolation='epoch_map', native_resolution_m=10.0, provenance='satellite_retrieval', credential_tier='token_free', citation='Copernicus Land Monitoring Service (2024). CLC+ Backbone 10 m raster land cover and land use, Europe. European Environment Agency. https://land.copernicus.eu/en/products/clc-backbone', license='copernicus-land-monitoring-data-policy', literature_key='copernicus_lcm10', unit='class_code', discovers_epochs_at_runtime=False, notes=('Not fetchable by Neptoon: not present in the Copernicus Data Space catalogue, and reachable only through a Sentinel Hub bring-your-own-COG deployment or a manual export. Export it once and declare it under remote_sensing.local_products with this product name.', 'The fetchable 10 m alternative is ESA WorldCover. The 100 m Copernicus land cover is fetchable but coarser, which is a step back for a footprint of a few hundred metres.', 'Informative only: land cover enters no computation anywhere in Neptoon.')), 'copernicus_swi_ts': RemoteSensingProduct(product='copernicus_swi_ts', provider='cdse', variables=('soil_water_index',), spatial=create_global(description='global land'), temporal_start=Timestamp('2007-01-02'), temporal_end=None, cadence='P1D', support='P1D', nodata_meaning='outside_coverage', gap_filling='none', extrapolation='forbidden', native_resolution_m=12500.0, provenance='satellite_derived', credential_tier='registration', citation='Copernicus Land Monitoring Service (2017). Soil Water Index 2007-present (raster 12.5 km), global, daily. European Commission Joint Research Centre. https://doi.org/10.2909/98c5e00e-3580-4bb3-9509-50a572b1e935 [accessed as V4.0.1 through the Copernicus Data Space Ecosystem]', license='copernicus-global-land-data-policy', literature_key='copernicus_soil_water_index', unit='index_0_100', notes=('Not soil moisture. It is a dimensionless index and Neptoon performs no conversion to m3/m3, because none is defensible in general.', 'Carries a characteristic time T. The support is roughly the last T days, so comparing an index at T=10 against unsmoothed CRNS soil moisture is not a comparison on equal terms.', 'Catalogue discovery answers anonymously while download requires an account, so the preflight can state what a free registration would add before anyone runs into a wall.', 'Tile existence is not coverage: an ocean query also returns a tile, because tiles are rectangles.'))}
VARIABLES_WITHOUT_CATALOGUED_PRODUCT
module-attribute
¶
VARIABLES_WITHOUT_CATALOGUED_PRODUCT = {'canopy_height': "The catalogued product was withdrawn on 2026-08-26 rather than kept for completeness. It was the German forest-structure RH95 map, a second catalogue entry and a second fetch beside the biomass map rather than a by-product of it, and it stopped at the German border: measured over 2019-2022, it reached 38 of the 66 COSMOS-Europe stations and none of the other 28. Nothing in Neptoon read the column. It entered no correction, no calibration and no soil-moisture product, and its one use was as a third witness to whether a footprint is really forest. That question is already open without it: measured at the same 38 German stations, the DLR forest fraction and the WorldCover tree-cover share disagree by more than 5 points at 13 of them and by up to 20, in both directions. Disagreeing witnesses are a diagnosis to run when the forest fraction looks wrong, not a column to publish. Ask for 'land_cover_class' for the forest question, and read 'rs_above_ground_biomass_valid_fraction' beside it for what the biomass product itself considers forest.", 'snow_cover_fraction': "No token-free source exists, and the credentialed one has the wrong time axis for masking. Measured against the live services on 2026-08-20: the subsetting service serving the vegetation products lists 46 products and none of them is snow, and MOD10A1, MYD10A1, MOD10A2, VNP10A1 and MCD10A1 each answer HTTP 404 'Product not found'. The reanalysis route does not close the gap either: its snow water equivalent is accepted as a variable but returns null for every model. Measured again on 2026-08-26, this time on the Copernicus Data Space Ecosystem: the Sentinel-2 HR-S&I fractional snow cover at 20 m IS in the OData catalogue as 'clms_wsi_fractional-snow-cover_europe_utm_20m_daily_v2', anonymously searchable, spatially queryable with OData.CSC.Intersects, running from 2016-08 to the present, one granule being a directory of COGs (FSCTOC, FSCOG, their QA layers, NDSI, QAFLAGS, CLD). It is invisible to the same catalogue's STAC interface, which lists ten collections and no snow at all. What it cannot do is mask a continuous record: at one CRNS station the point is covered by 16 granules in the 90 days of January to March 2021, one every 5.6 days, and the single granule covering the station on 2021-02-10 reports 89.8 percent cloud. Fractional snow cover is therefore a footprint-scale validator of a mask rather than the mask itself, and the mask runs on the reanalysis snow depth, which answers at every timestamp. Building the fetch path also means closing a credential gap: an anonymous product download answers HTTP 401 'Token not found', the Keycloak realm accepts the password grant and refuses client credentials on the public client, the S3 route answers InvalidAccessKeyId, and CREDENTIAL_ENV_VARIABLES['cdse'] declares no username field for any of them. MOD10A1 remains a standard Earth Engine dataset, so the variable also has a cloud-project route rather than disappearing. Snow depth from the reanalysis is unaffected and available today.", 'soil_moisture': 'Satellite soil moisture is served through a cloud-project provider, which is deliberately outside the token-free scope.'}
SpatialExtent
dataclass
¶
SpatialExtent(kind='global', west=None, south=None, east=None, north=None, outline=(), description='global', is_precise=True)
Declared spatial validity of a product.
Stage 1 of the spatial check reads this offline, so neptoon rs catalog
stays fast and usable without a network. It answers "definitely outside" and
nothing more: a bounding box alone is insufficient for a national product,
because Germany's box also contains Szczecin. Stage 2 reads the pixels.
outline exists for products whose box fits especially badly, and is
checked by ray casting so no geometry library is needed. It is left empty
unless a provider publishes an outline or one has been measured: an outline
that is slightly too tight turns a genuine near-border station into a hard
error, which is worse than deferring to the pixels.
create_from_bounding_box
classmethod
¶
Return a bounding-box extent with an optional simplified outline.
An extent counts as precise when an outline is supplied, because only an outline can answer "outside the product" for a shape that is not a rectangle. Callers may state it explicitly where a product's true extent genuinely is a rectangle.
is_point_inside ¶
Return whether a point lies inside the declared extent.
This is the offline pre-filter, not the coverage answer. A True here
means "not definitely outside"; whether the product actually has data at
the point is decided by reading the values.
BandValidity
dataclass
¶
BandValidity(band, fill_value=None, valid_min=None, valid_max=None, scale_factor=1.0, offset=0.0, units=None)
Per-band predicate deciding whether a returned cell counts as valid.
What counts as valid is per product and per band, not a universal nodata
constant: one vegetation product uses fill -3000 for its indices,
-1000 for its reflectance bands and -1 for pixel reliability, each
with its own valid range and scale factor.
Where the provider publishes this the catalog fetches it instead of
restating it, so it cannot drift, but the fetched values must be coerced.
A provider returning every field as a string, with the valid range as free
text such as '-2000 to 10000', makes an integer-against-string
comparison always unequal, and the resulting coverage fraction is 1.000
over open ocean.
convert_to_physical_value ¶
Return the scaled physical value for one valid raw cell value.
LocalReaderOptions
dataclass
¶
LocalReaderOptions(kind='raster', crs=None, timestamp_column=None, value_column=None, source_timezone=None)
How to read a user-supplied file, as declared by the user.
Separate from :class:BandValidity, which says what a value means. This
says how to get at the values in the first place, and the fields are the ones
that cannot be inferred without risking a plausible wrong answer: a raster
without embedded georeferencing has no CRS to read, and a table has no
convention naming its timestamp column.
RemoteSensingProduct
dataclass
¶
RemoteSensingProduct(product, provider, variables, spatial, temporal_start, temporal_end, cadence, nodata_meaning, gap_filling, extrapolation, native_resolution_m, provenance, credential_tier, citation, license, literature_key=None, extraction='single_cell', correction_inputs=frozenset(), observed_temporal_end=None, available_epochs=(), absent_value=None, support=None, max_gap=None, bands=(), discovers_epochs_at_runtime=True, unit='dimensionless', priority=0, notes=(), uri=None, is_user_declared=False, reader=None)
Coverage envelope and interpretation policy for one product.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
product
|
str
|
Canonical catalog key, as normalised by |
required |
provider
|
str
|
Canonical provider key. |
required |
variables
|
tuple[str, ...]
|
Semantic variables this product can serve. |
required |
correction_inputs
|
frozenset[str]
|
Variables this product is admissible as an input to a correction for. Empty, the default, means reference only. It is per quantity rather than a product-wide flag because admissibility differs per correction: an annual biomass map is a defensible biomass input and says nothing about whether the same map may drive a vegetation-interception correction, and a new correction has to be able to arrive without a schema change. Serving a variable and being admissible as a correction input for it are therefore two separate statements. |
frozenset()
|
extraction
|
ExtractionMode
|
How this product's cells are reduced to one value: a footprint-weighted
mean, or the single cell holding the station. Declared per product for
the same reason |
'single_cell'
|
spatial
|
SpatialExtent
|
Declared spatial extent, used by stage 1 of the spatial check. |
required |
temporal_start
|
Timestamp
|
Declared first epoch. |
required |
temporal_end
|
Timestamp | None
|
Declared last epoch, or |
required |
observed_temporal_end
|
Timestamp | None
|
Last epoch actually observed. Where this disagrees with
|
None
|
available_epochs
|
tuple[int, ...]
|
Declared epoch years for products with an irregular annual series. Empty means regular cadence or runtime discovery. |
()
|
cadence
|
str
|
ISO-8601 duration between observations. |
required |
support
|
str | None
|
ISO-8601 duration over which one observation applies, which is not the same as the interval between observations. A daily mean genuinely holds for its day, so spreading it over 24 hourly timestamps is what the product says rather than an assumption. A 16-day maximum-value composite does not hold for 16 days: it is one acquisition, and the other fifteen days were not measured. Filling within the support is always allowed and needs no marker; anything beyond it is gap filling and is always marked. Defaults to the cadence. |
None
|
nodata_meaning
|
NodataMeaning
|
What a nodata cell means for this product. |
required |
absent_value
|
float | None
|
Value contributed by an absent phenomenon, required when
|
None
|
gap_filling
|
GapFilling
|
Policy for filling between observations. |
required |
max_gap
|
str | None
|
Optional ISO-8601 limit on |
None
|
extrapolation
|
Extrapolation
|
Policy beyond the product extent. Independent of |
required |
native_resolution_m
|
float
|
Native resolution in metres, used by automatic product selection. |
required |
provenance
|
Provenance
|
Mandatory provenance class. |
required |
credential_tier
|
CredentialTier
|
Access effort demanded of the user. |
required |
citation
|
str
|
Attribution text, which is a licence condition for several providers rather than a courtesy. |
required |
license
|
str
|
Licence identifier, separate from the citation: attribution and redistribution are different permissions, and this design redistributes provider payloads whenever one is checked in as a test fixture. |
required |
literature_key
|
str | None
|
Key in |
None
|
bands
|
tuple[BandValidity, ...]
|
Per-band validity predicates, where they are known statically. |
()
|
discovers_epochs_at_runtime
|
bool
|
Whether available epochs must be discovered from the provider. |
True
|
unit
|
str
|
Native unit as delivered by the provider, before Neptoon converts it. |
'dimensionless'
|
priority
|
int
|
Manual override of automatic ranking, for the cases where a coarser product is scientifically preferable. |
0
|
notes
|
tuple[str, ...]
|
Short factual notes worth surfacing in the CLI and documentation. |
()
|
requires_user_supplied_file
property
¶
Return whether this entry documents a product Neptoon cannot fetch.
Some products exist, are worth describing, and have no automated route: the 10 m Copernicus land-cover map is reachable only through a Sentinel Hub bring-your-own-COG deployment or a manual export. Leaving it out of the catalogue would make it look unknown; leaving it in as a normal entry would let automatic selection choose something it cannot fetch. So it is catalogued and marked, and the user supplies the file.
effective_temporal_end
property
¶
Return the last epoch to trust offline.
An observed end wins over a declared one, because observation is the authority everywhere else in this design too.
has_temporal_extent_discrepancy
property
¶
Return whether declared and observed extents disagree.
__post_init__ ¶
Normalise the declared timestamps and reject an unusable product.
A catalog entry and a user-declared product reach this the same way, so every check here is a contract the rest of the package may then assume: timestamps are UTC, the enumerated policy fields hold known values, gap filling is compatible with the variables it would fill, and the attribution fields a published figure needs are present. Failing in the constructor keeps an unusable product from reaching a fetch, where the same fault would surface as a confusing provider error instead.
describe_temporal_extent ¶
Return a compact human-readable temporal extent.
get_temporal_extent_finding ¶
Return a finding when declared and observed extents disagree.
does_serve_variable ¶
Return whether this product can serve a semantic variable.
does_admit_correction_input ¶
Return whether this product may feed a correction with a variable.
Reference use needs no declaration; feeding a neutron correction does, because the value then reaches the soil-moisture product rather than a comparison plot.
count_footprint_cells ¶
Return how many native cells a circular footprint spans.
Pure geometry, pi r^2 over the square of the cell edge, and the
number that decides whether an aggregate is a footprint mean or one
pixel with a rounding error. At 100 m native resolution a 250 m radius
holds about 20 cells, which is an average; a 100 m radius holds about 3,
which is not.
apply_discovered_extent ¶
Return a copy carrying an extent discovered from the provider.
convert_to_dict ¶
Return a JSON-serialisable description for CLI and audit output.
get_catalog ¶
Return the product catalog, optionally extended by user-declared entries.
A user-declared product is a first-class source, not a special case: it gets the same preflight, the same coverage checks and the same provenance obligation as a built-in one.
get_product ¶
Return one catalog entry by canonical product name.
iter_products ¶
Yield catalog entries matching the supplied filters, in stable order.
get_products_for_variable ¶
Return every catalogued product able to serve one semantic variable.
get_catalogued_variables ¶
Return the set of variables at least one catalogued product serves.
get_catalog_findings ¶
Return metadata discrepancies worth reporting for the whole catalog.
create_user_declared_product ¶
create_user_declared_product(*, product, variables, uri, correction_inputs=(), unit, provenance, nodata_meaning, native_resolution_m, temporal_start, temporal_end, citation, license, cadence='P1Y', gap_filling='linear', extrapolation='forbidden', extraction='single_cell', absent_value=None, spatial=None, max_gap=None, bands=(), priority=0, allow_s3=False, notes=(), reader=None)
Build a catalog entry for a local or otherwise user-supplied product.
Many products exist only as a bulk download, and institutional layers are sometimes not public at all, so a local source is a normal route rather than an afterthought. The work is not the reading but the declaration: CRS, unit, variable, nodata semantics, provenance and temporal validity have to come from the user, so those fields are mandatory and validated here rather than defaulted into something plausible and wrong.
neptoon.external.remote_sensing.common¶
common ¶
Shared contracts for the remote-sensing external data category.
This module holds the pieces every part of the category needs and that none of them owns: the closed provenance vocabulary, the temporal and nodata policy vocabularies, the variable specifications that map a semantic variable onto a Neptoon column, the runtime attachment configuration, and the error types.
Remote sensing deliberately shares mechanics with the meteorological category
rather than an interface. There is no provider base class and no common
fetch_* signature: providers stay duck-typed with their own retrieval
shapes, because how the data is retrieved is exactly where they differ. What is
shared is the configuration abstraction, one attachment config, one variable
mapping, one registry, one dispatching attacher, plus the download cache in
neptoon.external._http_cache.
REMOTE_SENSING_SERVICE_LABEL
module-attribute
¶
Provenance
module-attribute
¶
Provenance = Literal['satellite_retrieval', 'satellite_derived', 'model_assimilated', 'reanalysis', 'derived']
PROVENANCE_VALUES
module-attribute
¶
PROVENANCE_VALUES = frozenset({'satellite_retrieval', 'satellite_derived', 'model_assimilated', 'reanalysis', 'derived'})
NodataMeaning
module-attribute
¶
NODATA_MEANING_VALUES
module-attribute
¶
EXTRAPOLATION_VALUES
module-attribute
¶
EXTRACTION_MODE_VALUES
module-attribute
¶
CredentialTier
module-attribute
¶
CREDENTIAL_TIER_VALUES
module-attribute
¶
CREDENTIAL_TIER_ORDER
module-attribute
¶
REMOTE_SENSING_VARIABLE_SPECS
module-attribute
¶
REMOTE_SENSING_VARIABLE_SPECS = {'above_ground_biomass': RemoteSensingVariableSpec(name='above_ground_biomass', output_column=str(RS_ABOVE_GROUND_BIOMASS), unit='kg/m2', provenance='satellite_derived', description='Above-ground biomass density from a satellite-driven model.'), 'canopy_height': RemoteSensingVariableSpec(name='canopy_height', output_column=str(RS_CANOPY_HEIGHT), unit='m', provenance='satellite_derived', description='Canopy height metric from a satellite forest-structure model.'), 'ndvi': RemoteSensingVariableSpec(name='ndvi', output_column=str(RS_NDVI), unit='dimensionless', provenance='satellite_retrieval', description='Normalized difference vegetation index.'), 'evi': RemoteSensingVariableSpec(name='evi', output_column=str(RS_EVI), unit='dimensionless', provenance='satellite_retrieval', description='Enhanced vegetation index.'), 'lai': RemoteSensingVariableSpec(name='lai', output_column=str(RS_LAI), unit='m2/m2', provenance='satellite_retrieval', description='Leaf area index.'), 'fpar': RemoteSensingVariableSpec(name='fpar', output_column=str(RS_FPAR), unit='dimensionless', provenance='satellite_retrieval', description='Fraction of absorbed photosynthetically active radiation.'), 'soil_moisture': RemoteSensingVariableSpec(name='soil_moisture', output_column=str(RS_SOIL_MOISTURE), unit='m3/m3', provenance='model_assimilated', supports_scaling=True, description='Satellite soil-moisture reference series.'), 'soil_water_index': RemoteSensingVariableSpec(name='soil_water_index', output_column=str(RS_SOIL_WATER_INDEX), unit='index_0_100', provenance='satellite_derived', supports_scaling=True, description='Copernicus Soil Water Index, a dimensionless index, not soil moisture.'), 'snow_cover_fraction': RemoteSensingVariableSpec(name='snow_cover_fraction', output_column=str(RS_SNOW_COVER_FRACTION), unit='fraction_0_1', provenance='satellite_retrieval', description='Fraction of the footprint under snow cover.'), 'land_cover_class': RemoteSensingVariableSpec(name='land_cover_class', output_column=str(RS_LAND_COVER_CLASS), unit='class_code', provenance='satellite_retrieval', is_categorical=True, is_informative_only=True, description='Land-cover class code. Informative only; enters no computation.')}
RemoteSensingConfigInput
module-attribute
¶
RemoteSensingConfigInput = RemoteSensingAttachmentConfig | Mapping[str, Any] | SupportsModelDump
REMOTE_SENSING_RECOVERABLE_ERRORS
module-attribute
¶
REMOTE_SENSING_RECOVERABLE_ERRORS = (RemoteSensingDataError, FileNotFoundError, KeyError, OptionalDependencyError, OSError, UnicodeError, ValueError)
RemoteSensingDataError ¶
Bases: NeptoonExternalDataError, RuntimeError
Raised when remote-sensing reference data cannot be attached.
RemoteSensingConfigurationError ¶
Bases: NeptoonConfigurationError, ValueError
Raised when the remote-sensing configuration itself cannot be honoured.
Deliberately not a :class:RemoteSensingDataError. A product that does not
serve a variable is not an unavailable reference series, it is a
configuration mistake, and the two need opposite handling: an unavailable
series may degrade under required: false, while a configuration mistake
has to be corrected before the run means anything. It reached the user as a
NeptoonInternalError before, which says "this is an unexpected internal
failure, not a normal user/data error" and invites a bug report for something
the user can fix in one line.
It is a ValueError as well, so the existing configuration boundaries that
catch native value errors keep catching it.
RemoteSensingConfigurationTypeError
dataclass
¶
Bases: RemoteSensingConfigurationError, TypeError
Invalid remote-sensing configuration object shape or field type.
RemoteSensingCoverageError
dataclass
¶
Bases: RemoteSensingDataError
Raised when a product does not cover the requested location.
A fetch outside a product's extent returns either nothing or edge values, so this is a hard failure rather than a warning: a warning scrolls past in a log and leaves behind a column of nodata that reads as measurement. The message always names the products that do cover the location.
RemoteSensingTemporalError
dataclass
¶
Bases: RemoteSensingDataError
Raised when a product cannot serve the requested period.
The message states the product's own extent, what was requested, and the single configuration line that would make the approximation the user's explicit decision.
RemoteSensingCredentialError
dataclass
¶
Bases: RemoteSensingDataError
Raised when a credentialed product is requested without credentials.
Never a bare 401: the message names the command that configures access
and, where relevant, the registration route and quota model.
SupportsModelDump ¶
Bases: Protocol
Protocol for Pydantic-like configuration objects.
RemoteSensingVariableSpec
dataclass
¶
RemoteSensingVariableSpec(name, output_column, unit, provenance, block='remote_sensing', is_categorical=False, is_informative_only=False, supports_scaling=False, description='')
Mapping between a semantic remote-sensing variable and its output column.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name
|
str
|
Semantic variable name used in configuration, for example |
required |
output_column
|
str
|
Resolved Neptoon column name for the value. |
required |
unit
|
str
|
Physical unit or an explicit dimensionless marker. |
required |
provenance
|
Provenance
|
Mandatory provenance class from the closed vocabulary. |
required |
block
|
Literal['remote_sensing', 'external_meteo']
|
Configuration block the variable is reached through. ERA5-Land is
reanalysis and belongs to |
'remote_sensing'
|
is_categorical
|
bool
|
Whether the value is a class code. Categorical values are never interpolated and never averaged. |
False
|
is_informative_only
|
bool
|
Whether the variable is forbidden from entering any computation. Land cover is informative only, and that restriction is enforced here rather than described in prose. |
False
|
supports_scaling
|
bool
|
Whether an offset or factor may be applied, producing a scaled companion column alongside the raw one. |
False
|
description
|
str
|
Short human-readable description used by the CLI and reports. |
''
|
RemoteSensingCoveragePolicy
dataclass
¶
RemoteSensingCoveragePolicy(spatial_policy='allow_partial', temporal_policy='auto', max_extrapolation_gap=None, anchor_year=None)
User-facing coverage policy for one attachment.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
spatial_policy
|
SpatialPolicy
|
|
'allow_partial'
|
temporal_policy
|
TemporalPolicy
|
|
'auto'
|
max_extrapolation_gap
|
str | None
|
Optional ISO-8601 limit on |
None
|
anchor_year
|
int | None
|
Optional explicit epoch, generalised from the |
None
|
RemoteSensingScalingPolicy
dataclass
¶
Bases: ScalingPolicy
Offset/factor adjustment for a remote-sensing reference series.
Remote-sensing soil moisture often reproduces the dynamics well while the absolute value is off, so an adjustment must be possible. It is explicit, reversible and audited: raw and scaled columns are both written, the default performs nothing, and a multiplicative factor is documented as hard to justify for an assimilated product.
The mechanics live in :mod:neptoon.external.remote_sensing.scaling because
the external_meteo block needs exactly the same treatment for ERA5-Land
soil moisture. This subclass adds only the configuration path, so a message
names the key the user actually wrote.
RemoteSensingAttachmentConfig
dataclass
¶
RemoteSensingAttachmentConfig(latitude, longitude, provider=AUTO_PROVIDER, product=None, variables=(), footprint_radius_m=250.0, extraction=AUTO_EXTRACTION_MODE, footprint_radius_is_explicit=False, coverage=RemoteSensingCoveragePolicy(), scaling=RemoteSensingScalingPolicy(), cache_dir=None, use_cache=True, request_timeout=30.0, prefer_remote_sensing_values=False, required=False, sensor_name=None, correction_input_variables=(), local_products=())
Runtime configuration for attaching one remote-sensing product.
This is the typed boundary between YAML/Pydantic configuration, the public
CRNSDataHub API and the provider-specific fetchers. Site coordinates are
injected by the workflow because the configuration section deliberately does
not repeat them.
footprint_radius_is_explicit
class-attribute
instance-attribute
¶
coverage
class-attribute
instance-attribute
¶
coverage = field(default_factory=RemoteSensingCoveragePolicy)
scaling
class-attribute
instance-attribute
¶
scaling = field(default_factory=RemoteSensingScalingPolicy)
prefer_remote_sensing_values
class-attribute
instance-attribute
¶
has_correction_inputs
property
¶
Return whether any requested variable feeds a neutron correction.
is_provider_automatic
property
¶
Return whether Neptoon selects the product for this attachment.
variable_specs
property
¶
Return specifications for the requested variables, in order.
resolve_extraction_mode ¶
Return the extraction mode that applies to one product here.
The catalogue declares what the product allows; this entry may override it. Both readings are legitimate, a user who wants the 250 m vegetation index averaged over a wider box is entitled to ask, so the override is honoured and the preflight reports which reading is in force.
is_correction_input ¶
Return whether one variable feeds a neutron correction in this run.
get_field_names
classmethod
¶
Return dataclass field names for mapping/object coercion.
from_config
classmethod
¶
Build from a Pydantic config, mapping, or compatible object.
RemoteSensingProvenanceRecord
dataclass
¶
RemoteSensingProvenanceRecord(product, provider, provenance, native_resolution_m, retrieved_at, url=None, product_version=None, content_hash=None, catalog_version=None, license=None, citation=None, notes=())
Everything needed to make a rerun verifiable rather than hoped for.
Version strings are not trustworthy, one product collection declares an
open-ended temporal extent while its items stop years earlier, and another
ships files named fv5.0 inside a v5.01 directory, so the content
hash is what makes "same configuration, six months later" checkable.
get_remote_sensing_variable_spec ¶
Return the specification for one semantic variable name.
coerce_remote_sensing_attachment_config ¶
Coerce public API input into a RemoteSensingAttachmentConfig.
neptoon.external.remote_sensing.correction_inputs¶
correction_inputs ¶
Turn a remote-sensing series into an input a neutron correction may read.
This is the second attachment point, and the only place where a satellite value stops being a reference column and becomes part of the soil-moisture product. The step is deliberately small and explicit, because everything that makes it defensible is bookkeeping rather than physics:
-
One derived column, named for what it is. The provider column
rs_above_ground_biomassstays an observation and is never written to. The correction readssite_biomass, which is the observation plus, where the user declared one, a static site value for the rows the product does not cover. Keeping them apart is what lets a reader see which rows rest on a map and which on a number somebody typed. -
Markers travel in the direction of consumption. A wrong biomass value produces a wrong soil moisture, so the interpolation, substitution and observation-age markers are propagated onto the derived CRNS columns rather than left beside the input. Reference series keep their markers where they are; they consume nothing.
-
The uncovered part stays uncovered. Where neither the product nor a declared static value has anything to say, the input is NaN, the correction factor is NaN, and no soil moisture is produced for those rows. The alternative, borrowing the nearest epoch, which the product's own policy permits for reference use, would put a map of another year inside a soil-moisture number, and inside N0 whenever a calibration day falls in the borrowed range.
CORRECTION_INPUT_COLUMNS
module-attribute
¶
CORRECTION_INPUT_COLUMNS = {'above_ground_biomass': str(SITE_BIOMASS)}
CorrectionInputResult
dataclass
¶
CorrectionInputResult(variable, source_column, column, marker_columns=(), value_count=0, observed_count=0, interpolated_count=0, static_fill_count=0, missing_count=0, static_value=None, warnings=tuple())
What building one correction-input column produced.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
variable
|
str
|
Semantic remote-sensing variable that was consumed. |
required |
source_column
|
str
|
Provider column the values came from. |
required |
column
|
str
|
Derived column the correction reads. |
required |
marker_columns
|
tuple[str, ...]
|
Marker and diagnostic columns written beside the derived column. |
()
|
value_count
|
int
|
Rows carrying a usable value. |
0
|
observed_count
|
int
|
Rows whose value came from an observation of the product's own epoch. |
0
|
interpolated_count
|
int
|
Rows filled between observations by the product's gap-filling policy. |
0
|
static_fill_count
|
int
|
Rows filled from the declared static site value. |
0
|
missing_count
|
int
|
Rows left NaN, where the correction is therefore not applied. |
0
|
static_value
|
float | None
|
The declared static site value, when one was used. |
None
|
RemoteSensingCorrectionSummary
dataclass
¶
RemoteSensingCorrectionSummary(variable, product, provider, native_resolution_m, footprint_radius_m, correction_method, correction_factor_column, input_column, factor_minimum=None, factor_mean=None, factor_maximum=None, epochs=(), substitutions=(), input_result=None, n0_before=None, n0_after=None, calibration_footprint_radius_m=None)
What a satellite-fed correction actually did, for the audit and report.
A corrected neutron series whose correction came from a satellite product has to carry the product, the epochs behind it, the resolution it was aggregated at and the range of the correction factor. The factor range is the number a reviewer can judge without rerunning anything: it says how large an intervention the correction was.
calibration_footprint_radius_m
class-attribute
instance-attribute
¶
get_correction_input_column ¶
Return the derived column a correction reads for one input quantity.
build_correction_input_column ¶
Derive the column a correction reads from an attached provider column.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data_frame
|
DataFrame
|
CRNS frame carrying the attached provider column and its markers. It is modified in place, which is the convention for column derivation here. |
required |
variable
|
str
|
Semantic remote-sensing variable, for example |
required |
static_value
|
float | None
|
Declared static site value used only on rows explicitly authorised by
|
None
|
static_fill_mask
|
Series | ndarray | None
|
Boolean mask identifying rows where the selected product is genuinely outside its declared coverage. Arbitrary provider/QA/temporal gaps must not be converted into a static site value merely because they are NaN. |
None
|
Returns:
| Type | Description |
|---|---|
CorrectionInputResult
|
The derived column, its markers and the row counts behind them. |
propagate_correction_input_markers ¶
Copy an input's provenance markers onto a column derived from it.
The rule, stated once: a marker sits with the remote-sensing column while the column is a reference series, and propagates onto the derived CRNS column as soon as the value is consumed. Written only where the derived column actually has a value, so the markers describe rows that exist.
Returns:
| Type | Description |
|---|---|
tuple of str
|
The marker columns written, in a stable order. |
summarise_correction_factor ¶
Return the minimum, mean and maximum of a correction factor column.
neptoon.external.remote_sensing.credentials¶
credentials ¶
Credential tiers for remote-sensing providers.
Access models differ, and the difference is a tier rather than a per-provider special case. The criterion is not whether access is free but whether the credential goes cleanly into the configuration:
token_free: no account, no key.registration: a free account, once. Discovery may still be anonymous while download is not, so preflight can state what an account would add before anybody runs into a wall.cloud_project: a registered cloud project, an eligibility questionnaire and a recurring compute quota. That cannot be abstracted away; the user has to act. The design goal is to keep it off the default path and make the opt-in short, not to hide it.
Two rules follow and are enforced here rather than described:
- The token-free path works with no user action at all. Credentials upgrade the available product set; they never gate the feature, and no credentialed provider is ever required for a workflow a user can run today.
- Automatic selection never picks an unconfigured credentialed product, so it cannot trigger an authentication prompt in the middle of a run.
The configuration file is the primary place to put a credential, because that
was the stated criterion. One consequence is documented rather than hidden: a
configuration carrying a credential must not be shared or committed, and the
*_env environment-variable indirection is the answer for configurations that
are shared. No keyring dependency is added.
CREDENTIALED_PROVIDERS
module-attribute
¶
CREDENTIALED_PROVIDERS = {'cdse': {'tier': 'registration', 'label': 'Copernicus Data Space Ecosystem', 'command': 'neptoon rs auth cdse', 'effort': 'free registration', 'detail': 'Create a free Copernicus Data Space Ecosystem account, then either put the credential in the sensor configuration or run the auth command. Catalogue discovery answers anonymously, so the preflight can already state which product, period and size would be fetched.'}, 'earth_engine': {'tier': 'cloud_project', 'label': 'Google Earth Engine', 'command': 'neptoon rs auth earth_engine', 'effort': 'cloud project plus eligibility questionnaire', 'detail': 'Register a Google Cloud project, enable the Earth Engine API and complete the eligibility questionnaire for noncommercial use. Noncommercial projects carry a recurring monthly quota of free compute units applied per project, so a run can fail for reasons unrelated to the data. Earth Engine is therefore never a default provider.'}}
CREDENTIAL_ENV_VARIABLES
module-attribute
¶
CREDENTIAL_ENV_VARIABLES = {'cdse': ('NEPTOON_CDSE_TOKEN', 'NEPTOON_CDSE_PASSWORD', 'CDSE_PASSWORD'), 'earth_engine': ('NEPTOON_EARTH_ENGINE_PROJECT', 'EARTHENGINE_PROJECT', 'GOOGLE_CLOUD_PROJECT')}
CredentialStatus
dataclass
¶
Whether one provider's credential is available, and how it was found.
get_credential_directory ¶
Return the user-level directory holding stored provider credentials.
find_credential_in_environment ¶
Return the environment variable naming a configured credential, if any.
get_credential_status ¶
Return whether a provider is usable, and where its credential came from.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
provider
|
str
|
Canonical provider name. |
required |
configured_secrets
|
dict[str, object] | None
|
Secrets supplied through configuration, keyed by provider. Values may be
|
None
|
credential_directory
|
Path | None
|
Override for the stored-credential directory, for tests. |
None
|
get_configured_tiers ¶
Return the credential tiers currently usable without further action.
token_free is always present, which is the point: the default path never
depends on a credential.
is_provider_usable ¶
Return whether a provider can be used without prompting the user.
describe_credential_gain ¶
Return the preflight line naming what authenticating would add.
get_credential_setup_hint ¶
Return the actionable setup instructions for one provider.
raise_for_missing_credential ¶
Raise an actionable error when a credentialed provider is unconfigured.
neptoon.external.remote_sensing.eligibility¶
eligibility ¶
Preflight: is this product suitable here, now, at this resolution?
Remote-sensing products are bounded in space and time in ways a station network is not, and both failure modes produce output that looks like data: a station outside a product's extent returns nodata or edge values, and a period past the last epoch silently carries the last value forward, inventing dynamics that did not occur. So eligibility is decided before the download rather than interpreted afterwards.
Four mechanisms, each answering one question:
-
:func:
check_spatial_eligibility, two-stage. Stage 1 is the catalog bounding box, offline and cheap, and answers only "definitely outside". Stage 2 reads the values and computes the fraction that are valid. A catalogue item search is not a coverage oracle: a national product may publish one country-wide item per year whose geometry is the country's bounding box, so item existence answers "is there a product for this box", not "does the product have data at this point". -
:func:
check_temporal_eligibility, two independent catalog fields, because filling between observations and substituting beyond the extent are orthogonal questions. A single field conflating them left the vegetation indices classified inconsistently, and neither answer fitted: refusing to fill would have put almost every hourly timestamp on NaN, while allowing substitution would have carried a seasonal index across a year boundary. -
:func:
select_product_for_variable, the user asks for a variable and Neptoon picks the product, mirroring the existing nearest-station idiom. The choice states its reasoning, including the runner-up it beat and, where the runner-up was finer, what choosing against it gave up. Temporal fit enters the ranking as a coarse grade rather than as an overlap fraction: a continuous key ahead of resolution let a single day of record decide a tenfold difference in pixel size. -
:func:
check_footprint_resolution, is the product fine enough for this footprint? A cell edge that is coarse against the radius delivers a pixel value wearing the name of a footprint mean. At 100 m native resolution a 250 m radius spans about 20 cells, which is an average; a 100 m radius spans about 3, which is not. The check names the count rather than only complaining. -
:func:
build_eligibility_report, the whole picture for one attachment, which is whatneptoon rs checkprints before a run starts.
One more distinction runs through all of them: whether a variable is a
reference series or an input to a correction. Reference series are the case
this module was written for, and they stay exactly as they were. A correction
input reaches the soil-moisture product, so three things tighten: the catalog
must declare the product admissible for that quantity (correction_inputs),
the footprint-resolution check refuses instead of warning, and substitution
beyond the product extent is forbidden regardless of what the product's own
extrapolation field permits, a borrowed biomass value would otherwise
travel into a soil-moisture number.
Nothing is silent. Every substitution, partial coverage and fallback product produces a marker column, a report line and an audit entry.
SpatialOutcome
module-attribute
¶
SpatialOutcome = Literal['covered', 'boundary', 'phenomenon_absent', 'masked_gap', 'outside', 'unverified']
TemporalOutcome
module-attribute
¶
ResolutionOutcome
module-attribute
¶
CoverageObservation
dataclass
¶
CoverageObservation(valid_fraction, cell_count, valid_cell_count, source_shape='raster_window', source_coverage_fraction=None)
Result of reading the values, which is stage 2 of the spatial check.
The unit of "valid" changes with the source shape but the rule does not: for a raster window it is the fraction of non-nodata pixels, for a gridded REST subset the fraction of non-fill cells, and for a point API it degenerates to zero or one because a single cell is either populated or null. An offshore point returns all-null, so the land-surface boundary is detected by the same rule with no special case.
SpatialEligibility
dataclass
¶
SpatialEligibility(product, outcome, is_inside_declared_extent, valid_fraction=None, absent_value=None, writes_valid_fraction_column=False, warnings=(), findings=())
Verdict of the spatial check for one product at one location.
writes_valid_fraction_column
class-attribute
instance-attribute
¶
EpochSubstitution
dataclass
¶
One uncovered sub-range and the epoch that stands in for it.
A request that reaches outside a product's extent has two parts, and they are not interchangeable: the part the product covers, which has observations of its own, and the part it does not, which has to borrow one. Collapsing both onto a single epoch is what emptied 91.5 % of a biomass column at Hohes Holz while the run reported success, so the two travel separately from here on and each uncovered range names the epoch it borrowed and how far away it was.
ResolutionEligibility
dataclass
¶
ResolutionEligibility(product, variable, footprint_radius_m, native_resolution_m, cell_count, outcome, is_correction_input=False, extraction='footprint', message='')
Verdict on a product's native resolution against one footprint.
Separate from the spatial check, which asks whether the product has data here. This asks whether what it has can be averaged here. Both can pass and fail independently: a 12.5 km product covers a German forest perfectly and still cannot describe a 250 m footprint.
is_usable
property
¶
Return whether the attachment may proceed on this verdict.
A coarse reference series is still worth attaching: the comparison is one the reader can judge for themselves once the cell count is stated, and refusing it would remove information rather than protect a result. There is also often no finer alternative, the 250 m vegetation product is the only token-free one, so refusing would amount to removing the variable. A coarse correction input is refused, because it becomes part of a soil-moisture number instead of sitting beside one.
Single-cell extraction is refused for a correction input on the same grounds and for the same reason the cell count exists: one pixel driving a correction is exactly the case the minimum guards against. It reaches this point only when a configuration overrode the mode, since no product that is declared single-cell is admissible as a correction input.
TemporalEligibility
dataclass
¶
TemporalEligibility(product, outcome, gap_filling, max_gap, extrapolation, resolved_temporal_policy, overlap_fraction=0.0, resolved_epoch=None, pinned_epoch=None, substitution_gap=None, missing_ranges=(), substitutions=(), marks_extrapolated=False, warnings=(), findings=())
Verdict of the temporal check for one product over one period.
describe_missing_ranges ¶
Return the exact missing ranges, which a warning must name.
describe_substitutions ¶
Return which epoch stood in for which range, which a report must say.
A reader has to be able to see, without opening the data, that the rows before 2017 in a record starting in 2014 carry the 2017 map rather than a measurement of 2014.
ProductSelection
dataclass
¶
ProductSelection(variable, product, reason, runner_up=None, alternatives=(), rejected=(), credential_gains=())
Chosen product for one variable, with the reasoning that chose it.
VariableEligibility
dataclass
¶
VariableEligibility(variable, selection, spatial, temporal, resolution=None, is_correction_input=False)
Everything the preflight knows about one requested variable.
describe_extraction ¶
Return the one line saying how this product is reduced to a value.
Printed for every variable rather than only where it is surprising. The mode decides what the number in the column is, one pixel, or a mean over an area, and a reader cannot check that against the catalog for every product they use.
UnservedVariable
dataclass
¶
A requested variable no catalogued product can serve here, and why.
Recorded rather than raised. The preflight already produces a specific,
actionable message for this, whether the variable has no catalogued product
at all, as snow cover does not, or its only candidate does not reach this
station. Raising it from inside the report meant one such variable took the
whole attachment down before the per-variable degradation path could run,
required: false notwithstanding.
The verdict belongs in the report; what to do about it is the caller's
decision, and it depends on required.
EligibilityReport
dataclass
¶
EligibilityReport(sensor_name, latitude, longitude, start, end, variables=(), unserved=(), catalog_version=CATALOG_VERSION, warnings=(), findings=())
Preflight result for one whole attachment configuration.
check_footprint_resolution ¶
check_footprint_resolution(product, *, variable, footprint_radius_m, is_correction_input=False, extraction='footprint')
Decide whether a product is fine enough for one footprint.
The count is geometry, pi r^2 over the cell area, and the thresholds are
policy. What differs between a reference series and a correction input is
only the consequence: a coarse reference series is a comparison the reader
can judge for themselves once the cell count is stated, while a coarse
correction input silently becomes part of a soil-moisture number, so it is
refused instead.
Under single-cell extraction the grading does not apply, because nothing is averaged. The cell count is still computed and reported, since it is the number that shows why the product is extracted that way, but it is not a verdict on a mean nobody took.
check_spatial_stage_one ¶
Return whether a point is not definitely outside a product's extent.
This is the offline pre-filter. It never confirms coverage, only the values can do that, but it is what makes the nodata semantics decidable at all: the same nodata value means "no forest here" inside the declared extent and "not covered" outside it.
check_spatial_eligibility ¶
check_spatial_eligibility(product, latitude, longitude, *, observation=None, spatial_policy='allow_partial', catalog=None)
Decide the spatial verdict for one product at one location.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
product
|
RemoteSensingProduct
|
Catalog entry under test. |
required |
latitude
|
float
|
Station position in degrees. |
required |
longitude
|
float
|
Station position in degrees. |
required |
observation
|
CoverageObservation | None
|
Stage-2 result from reading the values. |
None
|
spatial_policy
|
SpatialPolicy
|
|
'allow_partial'
|
catalog
|
dict[str, RemoteSensingProduct] | None
|
Catalog used to name covering alternatives in an error. |
None
|
Raises:
| Type | Description |
|---|---|
RemoteSensingCoverageError
|
When the location is outside the product's coverage, or when a partial
footprint is combined with |
check_series_coverage ¶
check_series_coverage(product, observations, *, latitude, longitude, spatial_policy='allow_partial', catalog=None)
Decide the spatial verdict from every observation over the period.
A single-timestamp check cannot tell a cloudy composite from open ocean: both come back entirely masked. Over a whole requested period the two separate cleanly, and the distinction matters, because a footprint that is masked at every timestamp is not a run of gaps that gap filling will handle, it is absence of coverage, and leaving it to gap filling produces a column of NaN that reads as "we tried" instead of an error that says why.
Raises:
| Type | Description |
|---|---|
RemoteSensingCoverageError
|
When no observation over the whole period holds a single valid cell for a product whose nodata means a transient mask. |
resolve_effective_extrapolation ¶
Return the extrapolation policy that actually applies.
A product's own field says what the quantity permits. Feeding a correction adds a second constraint that the catalog cannot express, because it is a property of the use rather than of the product: a borrowed epoch is a stated approximation in a reference column and an invented input to a soil-moisture number in a correction. The reference implementation for this correction substituted a constant for the years before its product's first epoch; that constant then carried the N0 fit, which is exactly the kind of thing that has to be visible in configuration rather than buried in a script.
resolve_temporal_policy ¶
Resolve auto against the extrapolation policy that applies.
auto is the default so a user does not have to know a product's temporal
character, and can still override it. A product whose extrapolation is
forbidden behaves like exact; the others allow their declared
substitution. A correction input is treated as forbidden whatever the
product declares.
check_temporal_eligibility ¶
Decide the temporal verdict for one product over one requested period.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
product
|
RemoteSensingProduct
|
Catalog entry being judged. |
required |
start
|
Timestamp | str
|
Requested processing period. |
required |
end
|
Timestamp | str
|
Requested processing period. |
required |
coverage
|
RemoteSensingCoveragePolicy | None
|
User-facing coverage policy for the attachment. |
None
|
is_correction_input
|
bool
|
Whether this variable feeds a neutron correction. When it does, substitution beyond the product extent is forbidden whatever the product declares: the uncovered rows stay NaN and the correction is simply not applied there, rather than being computed on a borrowed map. |
False
|
Raises:
| Type | Description |
|---|---|
RemoteSensingTemporalError
|
When the period cannot be served and the configuration does not permit the substitution that would serve it. The message states the product's extent, what was requested, and the single line that makes the approximation an explicit decision. |
select_product_for_variable ¶
select_product_for_variable(variable, *, latitude, longitude, start, end, configured_tiers=(TOKEN_FREE_TIER,), requested_product=None, catalog=None, is_correction_input=False, footprint_radius_m=None, extraction_override=AUTO_EXTRACTION_MODE)
Choose the product serving one variable, and state why.
The ranking is, in order: spatial coverage as a hard filter; credential tier,
token-free first, so automatic selection never triggers an authentication
prompt; the coarse temporal grade of :func:_grade_temporal_fit, which asks
whether the product has data inside the period rather than measuring the
overlap; an explicit priority override for cases where product knowledge makes a
scientifically preferable source more important than nominal pixel size; and
finally finer native resolution as the default tie-breaker within a priority.
Two further filters apply when the variable feeds a neutron correction, and they are hard rather than ranked: the catalog must declare the product admissible for that quantity, and the product's cells must be fine enough for the configured footprint. A pinned product that fails either is a configuration error naming the reason; under automatic selection it is a recorded rejection like any other, so a coarse global product loses to a fine regional one instead of quietly serving a correction it cannot support.
Temporal fit sits before resolution but is graded rather than measured, on purpose. A raw overlap fraction is a continuous key, and a continuous key placed before resolution makes an arbitrarily small difference decisive: one day of record outside a product's extent, or a leap day, outranked a factor of ten in resolution at 66 measured stations.
build_eligibility_report ¶
build_eligibility_report(config, *, start, end, configured_tiers=(TOKEN_FREE_TIER,), observations=None, catalog=None)
Run the whole preflight for one attachment configuration.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
RemoteSensingAttachmentConfig
|
Attachment configuration, including the site coordinates. |
required |
start
|
Timestamp | str
|
Requested processing period. |
required |
end
|
Timestamp | str
|
Requested processing period. |
required |
configured_tiers
|
Iterable[str]
|
Credential tiers usable without prompting the user. |
(TOKEN_FREE_TIER,)
|
observations
|
dict[str, CoverageObservation] | None
|
Optional stage-2 results keyed by product name. Absent entries mean the
values were not read, which the verdict states rather than glossing over;
this is why |
None
|
catalog
|
dict[str, RemoteSensingProduct] | None
|
Catalog to resolve against, including any user-declared products. |
None
|
neptoon.external.remote_sensing.footprint_aggregation¶
footprint_aggregation ¶
Footprint weighting and aggregation shared by every gridded provider.
This is CRNS domain knowledge, which is why it sits here and not in
:mod:neptoon.geo: that layer is deliberately domain-neutral and knows nothing
about where a neutron signal comes from. The formula itself is not re-derived
here either, :mod:neptoon.footprint owns the horizontal footprint and every
caller in Neptoon reaches it through that one boundary, this module included.
What is added here is the adapter that turns a set of cell distances into
weights and a set of cell values into one footprint number.
One implementation, two shapes of input. The gridded-subset provider hands over a square box of cells in a sinusoidal grid; the raster providers hand over a window read in geographic coordinates. Both reduce to "distance per cell, value per cell, usable per cell", so both call the same three functions. The predecessor package this work absorbs carried the Schroen weighting twice with divergent geodesy, one path projecting with pyproj and the other using a flat-earth cosine, and the two disagreed. Sharing the adapter is what makes that class of defect impossible rather than merely fixed.
Three aggregates are produced and they are not interchangeable:
- the weighted mean, which is the footprint value;
- the uncertainty, propagated from a per-pixel uncertainty band under the
assumption of full spatial correlation, so it is the weighted mean of the
pixel standard deviations rather than
sigma / sqrt(n); - the spatial heterogeneity, the weighted standard deviation of the values themselves, which describes the site and is not an uncertainty.
Conflating the second and the third is the single most likely way to publish a confident wrong number from this module, so they are separate functions with separate names and neither is derived from the other.
FootprintWeighting
dataclass
¶
Parameters of the radial weighting applied to the cells of a footprint.
The weighting function takes soil moisture and air humidity as shape parameters. Neither is known when a vegetation index or a biomass map is fetched, so both carry the published reference values and are exposed rather than hidden: a footprint drawn for a static map is an approximation of where the neutron signal comes from, not a measurement of it.
select_single_cell ¶
Return a mask selecting only the cell holding the station.
The cell whose centre is nearest, which on a regular grid is the cell the station falls inside. Used where averaging neighbours is not an improvement on one value but a different, wrong quantity: a mean of class codes, or a mean over cells that are all the same pixel because the product is orders of magnitude coarser than the footprint.
An all-NaN distance array cannot happen for a window that was read
around the station, so the nearest cell always exists.
calculate_footprint_weights ¶
Return unnormalised footprint weights for a set of cell distances.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
distances
|
ndarray
|
Distance from the station to each cell centre, in metres. |
required |
cellsize
|
float
|
Representative cell side in metres. Only the cell holding the station depends on it, through the intra-cell distance floor. |
required |
weighting
|
FootprintWeighting
|
Shape parameters of the radial weighting function. |
required |
Notes
The radial weighting function is zero at zero distance and peaks about a metre out, so the cell containing the sensor cannot be represented by the distance to its own centre. It is represented by the mean distance from that centre to a point drawn uniformly inside the cell.
aggregate_over_footprint ¶
Return the footprint-weighted and the unweighted mean over usable cells.
Both means run over the same cells, so the only difference between them is the weighting. That is what makes the unweighted value a comparison rather than a second measurement of a different area.
propagate_correlated_uncertainty ¶
Propagate a per-pixel uncertainty band under full spatial correlation.
The footprint uncertainty is the weighted mean of the pixel standard
deviations, not sigma / sqrt(n).
Retrieval errors in above-ground-biomass products are strongly spatially
correlated at 100 m inside a roughly 200 m CRNS footprint; the pixels are not
independent samples of a common mean. Treating them as independent would
divide the stated uncertainty by about sqrt(n), a factor of roughly five
for 100 m pixels in a 600 m window, and would make the most
scientific-looking number in the output the least trustworthy. Assuming full
correlation is conservative and defensible, and the assumption is stated in
the documentation rather than buried here.
This is deliberately not the same quantity as
:func:calculate_spatial_heterogeneity. Heterogeneity describes how much the
site varies; it may be reported alongside, but it is never the uncertainty.
calculate_spatial_heterogeneity ¶
Return the weighted standard deviation of the values over the footprint.
A description of the site, not a measurement error. The estimator carries Kish's effective-sample-size correction, because footprint weights are reliability weights rather than frequency counts and the plain weighted variance is biased low. The correction is largest for a small or sharply peaked footprint and vanishes as the effective sample size grows.
calculate_weighted_class_fractions ¶
Return the footprint-weighted share of every class code present.
Categorical values are never averaged, so a class field reduces to shares rather than to a mean. The shares sum to one over the usable cells; cells the mask rejects contribute to nothing, because a class code has no absent value the way a physical quantity does.
calculate_majority_class ¶
Return the class code holding the largest weighted share, or NaN.
substitute_absent_phenomenon ¶
Fill cells where the phenomenon is absent, and say which cells now count.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
values
|
ndarray
|
Cell values, with invalid cells already |
required |
valid
|
ndarray
|
Cells carrying a real observation. |
required |
inside_window
|
ndarray
|
Cells that belong to the read window at all. A window is a rectangle and a footprint is a circle, and a reprojected window is neither, so the cells that fall outside the source entirely must not be mistaken for cells where the phenomenon is absent. Substituting there would dilute the footprint mean with data that was never read. |
required |
absent_value
|
float
|
What an absent phenomenon contributes, |
required |
coverage_mask
|
ndarray | None
|
Optional mask distinguishing cells geometrically inside the raster source from boundless padding outside it. Outside-source cells must never be reinterpreted as an absent phenomenon. |
None
|
Returns:
| Type | Description |
|---|---|
tuple
|
The filled values and the mask of cells that now enter the aggregate. |
Notes
This is the substitution :mod:neptoon.geo deliberately does not perform.
That layer carries nodata_meaning and reports the valid fraction; what an
absent phenomenon contributes to a footprint aggregate is a domain decision
and is made here.
Without it, a station at a forest edge reports the forest mean as the footprint mean while half its footprint carries almost no woody biomass, a silent factor of about two, in the direction of overestimating.
neptoon.external.remote_sensing.masking¶
masking ¶
Say where a gridded observation disputes a CRNS product, and leave the product alone.
This is the third crossing between the gridded families and the CRNS columns, and the only one that is about a CRNS value rather than beside it. Biomass entered as a correction input; soil moisture and land cover stay reference series. Snow does neither: it is a confounder, not a covariate. Snow is hydrogen above the detector, so a CRNS under a snowpack reports wetter soil than it has.
Flagging rather than correcting, and rather than withholding. A snow correction is badly determined -- it needs the snow water mass, its layering and its position relative to the detector -- while a flag needs one number and a threshold. Deleting the affected soil moisture was the earlier behaviour and is deliberately not what happens now: the snow quantity that is actually available is a reanalysis depth over a 9 km cell, which is not evidence strong enough to remove a measurement on the user's behalf. Neptoon states the condition, the output carries it in one column, and whether those hours are dropped is the user's decision, taken on their own data with their own threshold.
The quantity is second best, and the code says so rather than hiding it. The
physically right variable is snow water equivalent in millimetres, because snow
density runs from roughly 100 to 500 kg/m3 and a depth alone does not give the
hydrogen mass: 0.20 m of snow is anywhere between 20 and 100 mm of water. Measured
against the live services, no token-free route serves it. The
Open-Meteo archive accepts snow_depth_water_equivalent and answers null for
every archive model; the Copernicus CDS point time-series collection, the one
whose shape would fit, does not offer the variable at all, and only the gridded
collection does, behind an account and a queued job; the token-free copies that
do carry it (ARCO-ERA5 on public GCS, NSF NCAR's ERA5 on AWS) are chunked for
whole-globe access, so a point-year costs gigabytes. Snow depth is what is
actually served, so snow depth is what the flag reads, and the ambiguity is
pushed into the threshold the user has to declare rather than into a number
Neptoon reports.
Three states, not two. True is snow at or above the threshold, False is a snow value that was read and stayed below it, and missing is missing: a gap in the reanalysis, a cloudy overpass, a period outside the product. Collapsing the third into False would print "checked, snow-free" for hours nobody checked, which is the one reading a user cannot recover from the output afterwards.
SNOW_AFFECTED_COLUMNS
module-attribute
¶
SNOW_AFFECTED_COLUMNS = (SOIL_MOISTURE_GRAV, SOIL_MOISTURE_VOL, SOIL_MOISTURE_VOL_FINAL, SOIL_MOISTURE_UNCERTAINTY_VOL_UPPER, SOIL_MOISTURE_UNCERTAINTY_VOL_LOWER, SOIL_MOISTURE_COUNTING_UNCERTAINTY_VOL, SOIL_MOISTURE_STATISTICAL_UNCERTAINTY_VOL, SOIL_MOISTURE_STATISTICAL_UNCERTAINTY_VOL_UPPER, SOIL_MOISTURE_STATISTICAL_UNCERTAINTY_VOL_LOWER, SOIL_MOISTURE_SYSTEMATIC_UNCERTAINTY_VOL, SOIL_MOISTURE_SYSTEMATIC_UNCERTAINTY_VOL_UPPER, SOIL_MOISTURE_SYSTEMATIC_UNCERTAINTY_VOL_LOWER, SOIL_MOISTURE_TOTAL_UNCERTAINTY_VOL, SOIL_MOISTURE_MEASURMENT_DEPTH, SOIL_MOISTURE_MEASUREMENT_RADIUS, SOIL_MOISTURE_MEASUREMENT_VOLUME)
SnowMaskResult
dataclass
¶
SnowMaskResult(source, threshold, flagged_count=0, evaluated_count=0, unavailable_count=0, flag_column='', affected_columns=())
What one snow flag says.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
SnowMaskSource
|
The gridded series the flag was built from. |
required |
threshold
|
float
|
Declared threshold, in :attr: |
required |
flagged_count
|
int
|
Rows at or above the threshold. |
0
|
evaluated_count
|
int
|
Rows carrying a usable snow value, so the flag could answer for them. |
0
|
unavailable_count
|
int
|
Rows where the snow series itself is missing. The flag is unknown for them rather than false: absence of an observation is not evidence of a snow-free hour, and writing one would be indistinguishable in the output from an hour that was actually checked. |
0
|
flag_column
|
str
|
Column the flag was written to. |
''
|
affected_columns
|
tuple[str, ...]
|
CRNS columns the flag speaks about and deliberately leaves untouched. |
()
|
get_snow_affected_columns ¶
Resolve the CRNS columns a snow flag speaks about, at execution time.
ColumnInfo labels are mutable because smoothing relabels the active final
soil-moisture column. Resolving here avoids freezing the pre-smoothing label
when this module is imported.
coerce_snow_mask ¶
Return a three-state snow flag from a boolean or numeric carrier.
The workflow carries the flag through temporal aggregation as a numeric column, because aggregation is defined on numbers; this converts either shape back into the nullable boolean the output column uses, keeping missing missing.
build_snow_mask ¶
Return the three-state snow flag for a frame.
The comparison is >=, so a threshold reads as "this much snow is already
too much", and a missing snow value gives pd.NA rather than False:
absence of an observation is not evidence of a snow-free hour.
Raises:
| Type | Description |
|---|---|
KeyError
|
When the source column is not in the frame. The caller decides whether that is a configuration error or a skipped step; defaulting here would produce an all-false flag, which is indistinguishable from a snow-free record. |
attach_snow_mask ¶
Write the snow flag beside the CRNS product, and change nothing else.
Mutates data_frame in place by adding exactly one column,
:func:get_snow_mask_column_name. The soil-moisture columns and their
uncertainties are read to report which of them the flag speaks about, and are
left as they are.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data_frame
|
DataFrame
|
Frame carrying both the snow series and the CRNS product. |
required |
source
|
SnowMaskSource
|
Declared source the flag is built from. |
required |
threshold
|
float
|
Threshold in :attr: |
required |
mask
|
Series | None
|
Flag computed earlier in the run, before temporal aggregation could average a short snow event away. Recomputed from the frame when omitted. |
None
|
neptoon.external.remote_sensing.providers.dlr_agbd¶
dlr_agbd ¶
German forest-structure rasters from the DLR EOC geoservice.
One cloud-optimised GeoTIFF per year covers the whole country, so a station reads a window rather than a file: the 2023 biomass map is 102120 by 86697 cells, about 71 GB as float64, and a 600 m window over HTTP range requests measures 0.03 s. The window is the only thing ever transferred.
This is a forest product, and that is the whole difficulty. Non-forest is nodata, not zero. Measured on the 2023 map with 600 m windows: a forest site is 100 % valid at 169 Mg/ha, a farmland site in the Magdeburger Boerde and a city site are both 0 % valid, and a forest edge is 97.6 % valid. Two failures follow from reading that as a coverage problem:
- farmland would be reported as "the product does not cover this location", which is a wrong diagnosis (Germany is covered, there is simply no forest) and
- at a half-forested footprint, averaging only the valid pixels reports the forest mean as the footprint mean while half the area carries almost no woody biomass. A silent factor of about two, upward.
So nodata contributes absent_value to the weighted aggregate, and the valid
fraction is written as a forest-fraction column: it is a property of the site,
and it also says how much of the footprint the vegetation indices have to
describe instead.
Metadata informs; pixels decide; disagreements are reported. Three discrepancies are harvested rather than discarded. The collection declares an open-ended temporal extent while items exist only for 2017-2023. The item geometry is the bounding box, so it can never answer coverage, Szczecin returns an item and no data. And the product's coverage stops short of the national border in places and is absent from the Baltic islands entirely, which is why the catalogue carries a measured outline instead of a rectangle.
DLR_STAC_BASE_URL
module-attribute
¶
DLR_COLLECTIONS
module-attribute
¶
DLR_COLLECTIONS = {'dlr_forest_structure_agbd': DlrCollection(collection='FOREST_STRUCTURE_DE_AGBD_P1Y', asset='agbd_cog', variable='above_ground_biomass', unit='Mg/ha', band='agbd')}
DlrCollection
dataclass
¶
Which collection and asset answer which semantic variable.
converts_to_kg_per_m2
property
¶
Return whether values need unit alignment to Neptoon's biomass column.
DlrEpochAsset
dataclass
¶
One annual item, reduced to what the fetch and the provenance need.
DlrStacClient ¶
DlrStacClient(*, base_url=DLR_STAC_BASE_URL, request_timeout=30.0, use_cache=True, cache_dir=None, requester=None)
Discover the annual items of one forest-structure collection.
The item search is not a coverage oracle and is never used as one: DLR publishes one Germany-wide raster per year whose item geometry is the German bounding box, so a query at Szczecin returns an item whose pixels are entirely nodata. Discovery answers "which epochs exist", which is a question the collection metadata gets wrong.
get_payload ¶
Return one parsed catalogue document, through the shared cache.
get_collection ¶
Return the collection document, for licence and declared extent.
get_epoch_assets ¶
Return the available epochs of one collection, keyed by year.
get_content_hash_input ¶
Return every catalogue payload used, in request order.
get_dlr_collection ¶
Return the collection serving one catalogued product.
build_dlr_raster_source ¶
Return the raster source for one epoch, carrying the catalogued validity.
The validity rule comes from the catalogue entry where one is declared, so a band's fill value and range live in one place rather than being restated beside the URL.
select_dlr_epochs ¶
select_dlr_epochs(available, *, start, end, requested_epoch, substitute_epochs=(), bracketing_epochs=())
Return the epochs to read for one request.
An annual product inside its own extent contributes one observation per year, so the attachment layer can interpolate between maps rather than holding one map constant across several years of neutron data.
Two kinds of epoch arrive from the preflight and they are not the same
instruction. requested_epoch pins one map for the whole period, which is
what an epoch map or an explicit coverage.anchor_year means.
substitute_epochs are borrowed for the parts of the period the extent
does not reach, and they are read in addition to the maps inside it: a
record running 2014-2021 needs its own 2017 to 2021 maps and one borrowed map
for 2014-2016, not one map for all of it.
fetch_dlr_forest_structure ¶
fetch_dlr_forest_structure(request, *, base_url=DLR_STAC_BASE_URL, requester=None, weighting=None)
Fetch one forest-structure variable over the requested period.
create_dlr_fetcher ¶
create_dlr_fetcher(*, base_url=DLR_STAC_BASE_URL, requester=None, weighting=None)
Return a fetcher bound to one endpoint, for tests and for a mirror.
register_dlr_provider ¶
Register the DLR fetcher under its catalogue provider key.
neptoon.external.remote_sensing.providers.esa_cci_biomass¶
esa_cci_biomass ¶
ESA CCI global above-ground biomass, with the uncertainty band it publishes.
Token-free over CEDA with byte ranges, ten epochs between 2007 and 2022, 100 m
globally. The tile arithmetic and the URL construction already live in
:mod:neptoon.geo.providers.remote_raster, because they are geometry rather than
science; what this module adds is the footprint, the coverage semantics and the
uncertainty.
The uncertainty is the reason this product is here. The national biomass map
publishes no uncertainty band at all, its item carries only the value COG, an
overview and a thumbnail, while this one ships AGB_SD beside every AGB
tile. That band is propagated under the assumption of full spatial
correlation: the footprint uncertainty is the weighted mean of the pixel
standard deviations, not sigma / sqrt(n). Retrieval errors in biomass
products are strongly correlated at 100 m inside a roughly 200 m footprint, so
treating the pixels as independent would divide the stated uncertainty by about
five for a 600 m window and would make the most scientific-looking number in the
output the least trustworthy.
Zero is a real value here, unlike in the national forest product. This is a
merged global map: a cell with no woody biomass carries zero, so an invalid cell
means the product has nothing at this location rather than that the phenomenon is
absent. The catalogue says so with nodata_meaning: outside_coverage, and
nothing is substituted.
Three traps this module is shaped around. The files declare no nodata value at
all, so validity comes from valid_min/valid_max, the same conclusion the
vegetation-index provider reached from the opposite direction. The published
directory version and the version inside the file name are not always the same
string: files under v5.01/ are named fv5.0, and asking for fv5.01
returns HTTP 404.
And the third one is open water, which this project has now hit for the third
time in a third product. Measured on 2026-08-24: a tile over pure ocean does not
exist and answers HTTP 404, which is a clean coverage answer. But a coastal
tile does exist (N60E000 is 35 MB because it contains Britain and Norway) and
inside it the North Sea reads as 0 with the mask reporting every pixel valid.
With no nodata value and zero a legal biomass, no pixel rule can separate open
water from a genuinely treeless site. So the ambiguity is reported as a finding
rather than resolved by a guess, exactly as the national product's bounding-box
ambiguity is. A land-only test suite would pass while this returned 0 Mg/ha at
valid_fraction = 1.000 over the sea.
UNCERTAINTY_SOURCE_COLUMN
module-attribute
¶
__all__
module-attribute
¶
__all__ = ['ESA_CCI_BIOMASS_UNIT', 'ESA_CCI_UNCERTAINTY_LAYER', 'ESA_CCI_VALUE_LAYER', 'UNCERTAINTY_COLUMN', 'EsaCciEpochResult', 'build_esa_cci_source', 'create_esa_cci_biomass_fetcher', 'esa_biomass_cci_geotiff_url', 'fetch_esa_cci_biomass', 'register_esa_cci_biomass_provider', 'select_esa_cci_epochs', 'UriResolver']
EsaCciEpochResult
dataclass
¶
EsaCciEpochResult(year, timestamp, value, uncertainty, footprint, value_grid, value_url, uncertainty_grid, uncertainty_url)
One epoch read: the value footprint, its uncertainty and the artefacts.
esa_biomass_cci_geotiff_url ¶
esa_biomass_cci_geotiff_url(*, tile, year, layer='agb', version=ESA_BIOMASS_CCI_DEFAULT_VERSION, file_version=None)
Return a public CEDA DAP GeoTIFF URL for one ESA CCI Biomass tile.
select_esa_cci_epochs ¶
select_esa_cci_epochs(product, *, start, end, requested_epoch, substitute_epochs=(), bracketing_epochs=())
Return the epoch years to read for one request.
The series is irregular (a five-year step separates 2010 from 2015), so the epochs inside the period are read and the attachment layer interpolates between them under the catalogue's gap-filling policy, every filled value carrying its marker and its observation age.
requested_epoch pins one map for the whole period and is reserved for the
case where the period really does resolve to one map. substitute_epochs
are borrowed for the ranges outside the extent and are read alongside the
maps inside it, never instead of them.
build_esa_cci_source ¶
build_esa_cci_source(*, product, tile, year, layer, version, cache_dir=None, timeout_s=300.0, uri_resolver=None)
Return a raster source for one tile, layer and epoch.
The validity rule comes from the catalogue band, which is where it belongs:
these files publish no nodata value, so valid_min is the only thing
standing between an unset cell and a biomass reading.
fetch_esa_cci_biomass ¶
fetch_esa_cci_biomass(request, *, version=ESA_BIOMASS_CCI_DEFAULT_VERSION, weighting=None, include_uncertainty=True, uri_resolver=None)
Fetch above-ground biomass and its uncertainty over the requested period.
create_esa_cci_biomass_fetcher ¶
create_esa_cci_biomass_fetcher(*, version=ESA_BIOMASS_CCI_DEFAULT_VERSION, weighting=None, include_uncertainty=True, uri_resolver=None)
Return a fetcher bound to one release, for tests and for a pinned study.
register_esa_cci_biomass_provider ¶
Register the ESA CCI Biomass fetcher under its catalogue provider key.
neptoon.external.remote_sensing.providers.local_products¶
local_products ¶
Products the user supplies as files, read through the same path as the rest.
Local files are the dominant pattern in Neptoon's external layer rather than an afterthought (one meteorological provider works no other way) and remote sensing without them would be the outlier. Many gridded products exist only as a bulk download, institutional layers are sometimes not public at all, loading once and computing offline afterwards is a real workflow, and a local route is the fallback when a provider API drifts.
Architecturally this costs almost nothing, which is the point of the design:
:class:~neptoon.geo.sources.RasterSource already accepts a local path, so a
local GeoTIFF falls out as a capability rather than as a second code path. The
footprint, the weighting, the coverage oracle and the provenance are the same
code the fetched products use, and the raster tests exercise this path anyway
because their fixtures are synthetic local files.
Two shapes are supported and no more. A raster (GeoTIFF, COG or VRT) read window-wise, and a table (CSV or Parquet) of timestamped values read whole. "Any file the user has" is not a shape.
The declaration is the work. CRS, unit, variable, nodata, provenance and temporal validity come from the user and are mandatory rather than defaulted, because each of them turns a wrong file into a plausible number instead of into an error. They are validated when the configuration loads, not when the file is first opened, so the failure arrives before the run rather than during it.
LOCAL_TABLE_SUFFIXES
module-attribute
¶
get_local_reader ¶
Return the reader declaration of a user-supplied product.
fetch_local_raster ¶
Read a user-supplied raster over the footprint, exactly as a fetched one.
fetch_local_timeseries ¶
Read a user-supplied table of timestamped values for one variable.
No footprint weighting happens here and none is pretended: a table has no geometry, so the value is whatever the user's own processing produced for this station. That is recorded in the provenance rather than left for a reader to assume.
register_local_product_providers ¶
Register both local readers under their catalogue provider keys.
describe_local_products ¶
Return one line per user-declared product, for the preflight and report.
neptoon.external.remote_sensing.providers.modis_ornl¶
modis_ornl ¶
MODIS/VIIRS point subsets from the ORNL DAAC subsetting service.
The service hands back a small grid of cells around a coordinate rather than a
raster window, so nothing here reads a COG and nothing here needs neptoon.geo.
What arrives is JSON: a box of raw integers, its corner in the MODIS sinusoidal
grid, and a per-band description of what those integers mean.
Four things in that description are easy to get wrong, and three of them return plausible-looking numbers rather than an error:
- Fill values differ per band inside one product. MOD13Q1 uses
-3000for NDVI and EVI,-1000for the reflectance bands and-1for pixel reliability, each with its own valid range and scale factor. A single nodata constant for the product is wrong for most of its bands. - Every metadata field arrives as a string, and the valid range as free text
such as
'-2000 to 10000'. Comparing an integer cell against the string'-3000'is always unequal, so every ocean cell counts as valid and the coverage fraction is 1.000 over open water. Both of those were measured, twice. - A subset requested without a band returns the first band, blue reflectance, whose fill value differs again, so the two mistakes compound into a confident wrong answer. This module addresses bands only through the semantic variable.
- The date list is not a coverage oracle.
/datesanswers for open ocean too. Only the values answer coverage, which is why every composite contributes a :class:CoverageObservationand the series decides, not one timestamp.
The unit of aggregation is the returned box. Cell distances to the station go through the shared horizontal footprint model, so the reported value is the footprint-weighted mean rather than a box mean; the unweighted mean over the same cells travels alongside as a comparison column.
MODIS_ORNL_PRODUCTS
module-attribute
¶
MODIS_ORNL_PRODUCTS = {'modis_ornl_mod13q1': ModisProductBands(code='MOD13Q1', variable_bands={'ndvi': '250m_16_days_NDVI', 'evi': '250m_16_days_EVI'}, quality_band='250m_16_days_pixel_reliability', quality_kind='pixel_reliability', day_of_year_band='250m_16_days_composite_day_of_the_year'), 'modis_ornl_mcd15a3h': ModisProductBands(code='MCD15A3H', variable_bands={'lai': 'Lai_500m', 'fpar': 'Fpar_500m'}, quality_band='FparLai_QC', quality_kind='fpar_lai_qc', day_of_year_band=None)}
ModisProductBands
dataclass
¶
ModisProductBands(code, variable_bands, quality_band=None, quality_kind=None, day_of_year_band=None)
Which band of an ORNL product answers which semantic variable.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
code
|
str
|
Product identifier as the service spells it, for example |
required |
variable_bands
|
Mapping[str, str]
|
Semantic variable to band name. Configuration never names a band. |
required |
quality_band
|
str | None
|
Band carrying the provider's own quality judgement, if any. |
None
|
quality_kind
|
QualityKind | None
|
How that band is read. A reliability rank and a packed bit field are not interchangeable and are not treated as if they were. |
None
|
day_of_year_band
|
str | None
|
Band giving the actual acquisition day of the selected cell inside a composite window, if the product publishes one. |
None
|
get_band_for_variable ¶
Return the band serving one semantic variable.
ModisQualityPolicy
dataclass
¶
Configurable filtering on the provider's own quality bands.
Filtering happens before gap filling, never after: a cell the provider's quality band rejects becomes a gap and is then treated like any other gap. That is the only order in which the interpolation marker keeps meaning "no usable observation was here".
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
max_pixel_reliability
|
int | None
|
Highest accepted vegetation-index reliability rank: |
1
|
max_scf_qc
|
int | None
|
Highest accepted LAI/FPAR retrieval rank from the |
1
|
get_limit_for_kind ¶
Return the accepted rank limit for one kind of quality band.
OrnlBandPredicate
dataclass
¶
OrnlBandPredicate(band, fill_value=None, valid_min=None, valid_max=None, scale_factor=1.0, units=None, scale_factor_missing=False)
Per-band validity rule, coerced from the strings the service returns.
The service publishes fill value, valid range, scale factor and units at
/{product}/bands, so the rule is fetched rather than restated and cannot
drift. Coercion is the whole point: every field arrives as text, and the
valid range as free text such as '-2000 to 10000'.
ModisSubsetGrid
dataclass
¶
Geometry of the box the service returned.
The corner is given in the MODIS sinusoidal projection in metres, so cell distances are computed there directly and no projection library is needed: the sinusoidal forward transform is one cosine.
is_aligned_with ¶
Return whether two returned bands address the same pixel grid.
create_from_payload
classmethod
¶
Return the grid described by a /subset payload.
Every geometry field is coerced: the corners arrive as strings.
calculate_cell_distances ¶
Return each cell's distance to the station in metres.
Cells come back row-major from the north-west corner, which is the service's documented order; the corner it reports is the lower left, so row zero is the northernmost row. A box centred on the station is very nearly symmetric about its middle row, so the consequence of the two conventions disagreeing would be a vertical flip of a near-symmetric weight field rather than a wrong value.
ModisComposite
dataclass
¶
One date of one band, as returned inside a /subset payload.
ModisOrnlSubsetClient ¶
ModisOrnlSubsetClient(*, base_url=ORNL_BASE_URL, cache_dir=None, use_cache=True, request_timeout=30.0, requester=None)
Cached access to the ORNL DAAC subsetting endpoints.
Every response is cached by product, band, position, box size and date range, because the request that matters is the whole period at once: a 16-day product asked per CRNS timestamp would be tens of thousands of requests for the same few hundred composites.
get_band_predicates ¶
Return the per-band validity rules the service publishes.
get_available_dates ¶
Return the composite dates the service holds for one position.
This is availability, not coverage: the endpoint answers for open ocean too. Only the values decide whether a position is covered.
get_subset ¶
Return one chunk of one band, at most ten composites wide.
fetch_band_series ¶
Return every composite of one band, chunked across the date limit.
get_content_hash ¶
Return a hash over every payload used, in request order.
Version strings from providers are not trustworthy, so this is what makes "same configuration, six months later" checkable.
coerce_ornl_number ¶
Return a metadata field as a number, or None when it is not one.
The service answers with text throughout, and a band without a scale factor
reports the string 'Not Available' rather than omitting the field.
parse_ornl_valid_range ¶
Return the low and high bounds of a free-text valid range.
The field is prose, not a pair: '-2000 to 10000'. An unparsable range
yields two None bounds. The resolver treats those as missing metadata
and fills them from the catalog declaration; they are never interpreted as
an instruction to broaden validity silently.
parse_band_predicates ¶
Return the per-band validity rules from a /{product}/bands payload.
convert_catalog_band ¶
Return the catalog's own predicate for a band, used as a fallback.
describe_predicate_disagreement ¶
Return a finding where the fetched and catalogued rules disagree.
Neither is corrected against the other. A disagreement says the provider changed something or that the catalog misread it, and both are worth a line.
convert_to_sinusoidal ¶
Return sinusoidal grid coordinates in metres for one position.
MODIS products are gridded on a sphere in the sinusoidal projection, whose forward transform needs no projection library.
build_quality_mask ¶
Return the mask of cells the provider's own quality band accepts.
A reliability rank and a packed bit field are read differently, and the
difference matters: reading FparLai_QC as a rank would compare a byte
whose top bits carry the retrieval method against a threshold meant for the
values zero to three, and accept almost nothing.
parse_subset_payload ¶
Return the grid and the per-date cell arrays of a /subset payload.
calculate_acquisition_timestamp ¶
Return the day a composite was actually acquired on.
A 16-day composite carries the day of year of the cell each value was selected from, so a value can be placed on the day it was measured instead of on the first day of its window. That is worth up to fifteen days of interpolation accuracy and the band costs nothing.
It fixes the timing only. MOD13Q1 is a maximum-value composite, deliberately the upper envelope of its period, and no timestamp corrects that selection.
calculate_subset_half_width_km ¶
Return the half-width of the box to request, in kilometres.
The service takes whole kilometres, so a 250 m footprint asks for the smallest box it offers. Cells beyond the footprint carry weights orders of magnitude smaller and cost nothing, while a box that is too small leaves the coverage fraction resting on a handful of cells.
chunk_dates ¶
chunk_dates(dates, size=MAX_DATES_PER_SUBSET_REQUEST)
Return date identifiers grouped into requests the service accepts.
get_product_bands ¶
Return the band mapping for one catalogued product.
Raises:
| Type | Description |
|---|---|
RemoteSensingDataError
|
When the catalog routes a product to this provider that the module does not know how to address. The message names what it does serve rather than failing on a missing key. |
resolve_band_predicate ¶
Return the validity rule for one band, preferring the service's own.
A band with no rule anywhere is an error rather than a default. "Treat every cell as valid" is precisely the failure that reports open ocean as fully covered, so it is never the fallback.
select_composite_dates ¶
Return the composites needed to cover a period, plus one on each side.
The bracketing composites are what let the leading and trailing part of the period be filled by interpolation between two real observations rather than left empty or carried outward from one.
fetch_modis_ornl_subset ¶
fetch_modis_ornl_subset(request, *, quality_policy=None, weighting=None, base_url=ORNL_BASE_URL, requester=None)
Fetch one catalogued MODIS or VIIRS product for one station.
The whole period is fetched once and joined afterwards, never one request per CRNS timestamp: a 16-day product over three years is under seventy composites, and asking per hour would be tens of thousands of requests for the same few hundred numbers.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
request
|
RemoteSensingFetchRequest
|
What to fetch, from the attachment layer. |
required |
quality_policy
|
ModisQualityPolicy | None
|
Filtering on the provider's own quality bands. Defaults to accepting good and marginal vegetation-index cells and main-algorithm LAI/FPAR retrievals. |
None
|
weighting
|
ModisFootprintWeighting | None
|
Shape parameters of the radial weighting function. |
None
|
base_url
|
str
|
Service base URL, overridden by tests. |
ORNL_BASE_URL
|
requester
|
Any | None
|
Optional request callable, so a test can answer without a network. |
None
|
Returns:
| Type | Description |
|---|---|
RemoteSensingFetchResult
|
Observations only. Gap filling belongs to the attachment layer, so no value here was invented to fill a hole. |
create_modis_ornl_fetcher ¶
create_modis_ornl_fetcher(*, quality_policy=None, weighting=None, base_url=ORNL_BASE_URL, requester=None)
Return a fetcher bound to one quality and weighting configuration.
The dispatch signature carries no provider options, so a study that wants different filtering builds its own fetcher and registers it, rather than every provider growing a configuration field nobody else uses.
register_modis_ornl_provider ¶
register_modis_ornl_provider(*, quality_policy=None, weighting=None, base_url=ORNL_BASE_URL, requester=None)
Register the MODIS/VIIRS fetcher, optionally reconfigured.
neptoon.external.remote_sensing.product_outlines¶
product_outlines ¶
Measured outlines of products whose bounding box cannot answer coverage.
A bounding box says "definitely outside" for a global product. It cannot do that for a national one, and the German biomass product shows why twice: Szczecin lies inside Germany's box and returns entirely nodata, exactly as a German farmland footprint does, and the product has no data at all on Ruegen, Usedom or Fehmarn. Measured on the 2023 map, 2026-08-24: a 4 km box over Jasmund National Park returns 0 valid pixels of 275600, while the Darss peninsula 60 km west returns 90.8 %. An administrative boundary would have called Jasmund covered and handed back zero biomass for old beech forest, so the outline is measured from the product's own data. Provider metadata cannot supply it: the collection publishes one bounding box and every item geometry equals it.
DLR_FOREST_STRUCTURE_DE_OUTLINE
module-attribute
¶
DLR_FOREST_STRUCTURE_DE_OUTLINE = (((8.8124, 54.8856), (8.9849, 54.8971), (9.0309, 54.8511), (9.5371, 54.8511), (9.5831, 54.8741), (9.6061, 54.8051), (9.7326, 54.7936), (9.7441, 54.6901), (9.9167, 54.6786), (9.9167, 54.6325), (9.9627, 54.6095), (9.9742, 54.414), (10.0892, 54.414), (10.1352, 54.3219), (10.4803, 54.3219), (10.5033, 54.368), (10.5493, 54.3219), (10.8254, 54.3219), (10.8484, 54.345), (10.8599, 54.2874), (10.9174, 54.2759), (10.9404, 54.2299), (11.09, 54.2184), (11.1015, 53.9998), (11.5386, 53.9768), (11.5616, 54.0919), (11.6996, 54.0919), (11.7456, 54.1609), (12.1367, 54.1609), (12.1827, 54.2529), (12.2978, 54.2989), (12.4358, 54.2989), (12.4703, 54.4715), (12.5278, 54.483), (12.5508, 54.46), (12.7809, 54.46), (12.8154, 54.4255), (12.8269, 54.3219), (12.8959, 54.2759), (13.0109, 54.2759), (13.126, 54.1839), (13.264, 54.1839), (13.31, 54.1379), (13.7241, 54.1609), (13.7816, 54.1034), (13.7816, 53.9653), (13.9197, 53.9423), (13.9312, 53.7928), (14.2648, 53.7583), (14.2878, 53.6432), (14.3338, 53.6202), (14.3338, 53.5052), (14.3798, 53.4592), (14.3798, 53.2981), (14.4373, 53.2866), (14.4488, 53.2521), (14.3568, 53.2061), (14.3568, 53.114), (14.3223, 53.0795), (14.2303, 53.0565), (14.2188, 52.999), (14.1497, 52.976), (14.1497, 52.6769), (14.2303, 52.6424), (14.2533, 52.5964), (14.3223, 52.5964), (14.3683, 52.5043), (14.5868, 52.4928), (14.5868, 52.2397), (14.6559, 52.1937), (14.6559, 52.0787), (14.7479, 52.0557), (14.7019, 52.0097), (14.7019, 51.7106), (14.7709, 51.6645), (14.7709, 51.5495), (14.8859, 51.4805), (14.8859, 50.9743), (14.8169, 50.9283), (14.8169, 50.8363), (14.7824, 50.8018), (14.5983, 50.8708), (14.2533, 50.8708), (14.0922, 50.8018), (13.9772, 50.8018), (13.9082, 50.7327), (13.5631, 50.7097), (13.5286, 50.6292), (13.4711, 50.5947), (13.241, 50.5717), (13.218, 50.5257), (13.0339, 50.4797), (12.9649, 50.4106), (12.5048, 50.3876), (12.4013, 50.3071), (12.4013, 50.031), (12.4933, 49.985), (12.5393, 49.893), (12.5393, 49.6399), (12.6544, 49.5248), (12.6544, 49.4558), (12.6889, 49.4213), (12.7579, 49.4213), (12.8039, 49.3523), (12.9879, 49.3523), (13.218, 49.1222), (13.356, 49.0992), (13.4135, 48.9957), (13.4711, 48.9611), (13.6321, 48.9611), (13.6781, 48.8921), (13.7471, 48.8921), (13.7586, 48.8346), (13.8161, 48.8231), (13.8507, 48.7656), (13.8046, 48.5585), (13.7241, 48.501), (13.379, 48.478), (13.356, 48.3629), (13.287, 48.3629), (13.241, 48.2939), (13.0109, 48.2709), (12.9304, 48.1904), (12.9304, 47.9143), (12.9994, 47.8683), (12.9994, 47.7302), (13.1145, 47.6382), (13.0455, 47.5922), (13.0109, 47.4656), (12.9189, 47.5347), (12.8039, 47.5347), (12.7579, 47.6037), (12.2057, 47.6267), (12.1827, 47.6037), (11.6306, 47.5807), (11.5846, 47.5117), (11.4926, 47.5117), (11.4235, 47.4886), (11.4005, 47.4426), (11.3085, 47.4196), (10.9634, 47.4196), (10.9174, 47.4656), (10.8714, 47.4656), (10.8484, 47.5117), (10.4803, 47.5117), (10.4343, 47.4196), (10.0662, 47.4196), (10.0432, 47.4886), (9.9282, 47.5117), (9.8131, 47.5807), (9.5371, 47.6037), (9.491, 47.6727), (8.6398, 47.6497), (8.5938, 47.6037), (8.4558, 47.6037), (8.4328, 47.5577), (7.7426, 47.5577), (7.6736, 47.5347), (7.6046, 47.6497), (7.524, 47.6612), (7.524, 47.7992), (7.5701, 47.9143), (7.5701, 48.1214), (7.6851, 48.3054), (7.7311, 48.3284), (7.7541, 48.4665), (7.8691, 48.5355), (7.8921, 48.6735), (7.9612, 48.7196), (7.9612, 49.0187), (7.6506, 49.0302), (7.4895, 49.1222), (6.7303, 49.1452), (6.6958, 49.2257), (6.6498, 49.2487), (6.6498, 49.3178), (6.6038, 49.3408), (6.5923, 49.4443), (6.3737, 49.4788), (6.3737, 49.8239), (6.2472, 49.8814), (6.1207, 50.031), (6.1207, 50.146), (6.1897, 50.2151), (6.1897, 50.5142), (6.1437, 50.6982), (6.0287, 50.7212), (6.0057, 51.0203), (6.0517, 51.0433), (6.0517, 51.2274), (6.1667, 51.3424), (6.1667, 51.6876), (6.0402, 51.6991), (6.0057, 51.7796), (6.1552, 51.8141), (6.1782, 51.8831), (6.2012, 51.7911), (6.4773, 51.7911), (6.5003, 51.8601), (6.7763, 51.8601), (6.7879, 52.1247), (6.9949, 52.1477), (6.9949, 52.4468), (6.7763, 52.4813), (6.7073, 52.5043), (6.6958, 52.5388), (7.1214, 52.5273), (7.1329, 52.8149), (7.179, 52.8379), (7.202, 52.999), (7.524, 53.045), (7.524, 53.3671), (7.432, 53.3901), (7.3975, 53.5627), (7.225, 53.5742), (7.2595, 53.6317), (7.6046, 53.6317), (7.6276, 53.5857), (8.0187, 53.5857), (8.0302, 53.6432), (8.0647, 53.6547), (8.0762, 53.4131), (8.1452, 53.3901), (8.1567, 53.2636), (8.5938, 53.2406), (8.6283, 53.3211), (8.6283, 53.6432), (8.5823, 53.8043), (8.5938, 53.8618), (8.6743, 53.8273), (8.7088, 53.6777), (9.0079, 53.6777), (9.0194, 53.7583), (9.0769, 53.7698), (9.1229, 53.6547), (9.4795, 53.6662), (9.468, 53.9538), (9.0884, 53.9883), (9.0884, 54.2184), (9.2035, 54.2414), (9.2035, 54.5175), (9.169, 54.552), (9.0769, 54.552), (9.0539, 54.6441), (8.9619, 54.6441), (8.9504, 54.8166), (8.8929, 54.8741)),)
DLR_FOREST_STRUCTURE_DE_OUTLINE_SOURCE
module-attribute
¶
DLR_FOREST_STRUCTURE_DE_OUTLINE_SOURCE = {'product': 'dlr_forest_structure_agbd', 'epoch': 2023, 'overview_level': 64, 'support_block': 4, 'closing_iterations': 6, 'simplification_degrees': 0.02, 'measured_on': '2026-08-24'}
DLR_OUTLINE_REFERENCE_SITES
module-attribute
¶
DLR_OUTLINE_REFERENCE_SITES = (('Hohes Holz forest', 52.09, 11.226, True), ('Boerde farmland', 52.02, 11.36, True), ('Magdeburg city', 52.13, 11.62, True), ('Darss peninsula', 54.43, 12.53, True), ('Munich', 48.14, 11.58, True), ('Hamburg', 53.55, 9.99, True), ('Jasmund National Park (no product data)', 54.52, 13.63, False), ('Ruegen (no product data)', 54.42, 13.42, False), ('Szczecin', 53.43, 14.55, False), ('Rzepin forest', 52.35, 14.83, False), ('Strasbourg', 48.58, 7.75, False), ('Salzburg', 47.8, 13.04, False), ('Baltic Sea', 54.6, 12.5, False))
is_point_in_outline ¶
Return whether a point falls inside a set of rings, by ray casting.
A standalone copy of the catalogue's own test so a freshly derived outline can be checked before it is pasted in, without building a catalogue entry for it.
derive_dlr_forest_structure_outline ¶
Re-measure the forest-structure outline from a published epoch.
Needs network access and the geo extra. href skips catalogue discovery
and is what the tests use to run the derivation against a local fixture.
derive_product_outline ¶
derive_product_outline(href, *, overview_level=64, support_block=4, closing_iterations=6, simplification_degrees=0.02, minimum_ring_vertices=4)
Measure a product's data extent from its own raster.
The four steps of the module docstring, in order. The heavy imports are local because this runs once per re-measurement and importing the catalogue must stay cheap.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
href
|
str
|
Raster to measure. Any source rasterio can open, including a local file. |
required |
overview_level
|
int
|
Internal overview to read. Nearest-neighbour decimation is assumed, so a cell is valid exactly when the full-resolution pixel it samples is valid. |
64
|
support_block
|
int
|
Block side reduced with "any valid cell" into the support grid. |
4
|
closing_iterations
|
int
|
Iterations of a 3 by 3 morphological closing, which bridges the parts of the extent that carry no phenomenon rather than no data. |
6
|
simplification_degrees
|
float
|
Douglas-Peucker tolerance applied to the traced contour. |
0.02
|
format_outline_literal ¶
Return a freshly derived outline as the Python literal to paste in.
neptoon.external.remote_sensing.providers.worldcover¶
worldcover ¶
ESA WorldCover 10 m land cover from public object storage.
The simplest provider in the category: no account, no token, no catalogue query. Tiles are 3 by 3 degrees, directly addressable by name, and served with byte ranges, so a footprint window out of an 85 MB tile measures 0.042 s.
Informative only. The class and its shares are written, reported and documented; they enter no computation anywhere in Neptoon. That is enforced by the variable specification rather than described in prose, and it matters because a land-cover map has a classification accuracy and a fixed epoch: a map of 2021 describing a field that was afforested in 2023 is wrong in a way no quality flag would catch, and letting it touch a soil-moisture result would propagate that silently.
Two shapes of output, because a class is not a number. The dominant class answers "what is this site", and the per-class shares answer "how mixed is it", which is the more useful of the two for a footprint that is rarely uniform. Neither is an average: the mean of tree cover (10) and cropland (40) is 25, which reads as shrubland.
Tile identifiers name the south-west corner, on a three-degree grid, N51E009 covers 51 to 54 north and 9 to 12 east. That is the opposite convention from the global biomass product in the same catalogue, whose identifiers name the north-west corner, so the two are derived by separate functions that each say which corner they mean.
WORLDCOVER_BASE_URL
module-attribute
¶
WORLDCOVER_CLASS_LABELS
module-attribute
¶
WORLDCOVER_CLASS_LABELS = {10: 'tree_cover', 20: 'shrubland', 30: 'grassland', 40: 'cropland', 50: 'built_up', 60: 'bare_or_sparse_vegetation', 70: 'snow_and_ice', 80: 'permanent_water_bodies', 90: 'herbaceous_wetland', 95: 'mangroves', 100: 'moss_and_lichen'}
WORLDCOVER_RELEASES
module-attribute
¶
WORLDCOVER_RELEASES = {'esa_worldcover_v200': WorldCoverRelease(version='v200', year=2021), 'esa_worldcover_v100': WorldCoverRelease(version='v100', year=2020)}
WorldCoverRelease
dataclass
¶
get_worldcover_release ¶
Return the release serving one catalogued product.
worldcover_tile_for_lonlat ¶
Return the tile identifier covering one WGS84 point.
Identifiers name the south-west corner on a three-degree grid, so both
edges floor: 52.09 N / 11.226 E lies in N51E009, whose bounds are
(9, 51, 12, 54).
worldcover_tile_bounds ¶
Return (left, bottom, right, top) WGS84 bounds of a tile identifier.
worldcover_map_url ¶
worldcover_map_url(*, tile, release, base_url=WORLDCOVER_BASE_URL)
Return the anonymous HTTPS URL of one map tile.
build_worldcover_raster_source ¶
build_worldcover_raster_source(*, product, release, tile, base_url=WORLDCOVER_BASE_URL, uri=None)
Return the raster source for one tile, carrying the catalogued validity.
fetch_worldcover_land_cover ¶
fetch_worldcover_land_cover(request, *, base_url=WORLDCOVER_BASE_URL, weighting=None, uri=None)
Fetch the dominant land-cover class and the footprint class shares.
describe_open_water ¶
describe_open_water(class_fractions, *, latitude, longitude, threshold=WATER_FRACTION_FINDING_THRESHOLD)
Report open water in the footprint as a site-suitability problem.
This is the one cross-product check in the design, and it is deliberately small: land cover observes water directly, so the provider that reads it says so. Nothing else happens, no column, no error, no other product consulted.
What it is not about. Arithmetically water behaves like farmland under a forest-only biomass product: both carry no woody biomass, so the weighted biomass mean over a shoreline footprint is not wrong. The problem sits upstream of any reference column. Open water is an enormous hydrogen pool, so a CRNS footprint containing it has a compromised soil-moisture retrieval, and that outranks anything a biomass or vegetation column says about the site. The finding therefore talks about the retrieval rather than about biomass.
The share is footprint-weighted, so water beside the sensor counts for far more than the same area at the footprint edge, the right ordering for a hydrogen pool the neutron signal actually sees.
Limitation worth knowing: this fires only where land cover is attached. A
biomass-only configuration reads no land-cover map, and having the biomass
provider fetch one would mean pulling a product the user did not request,
with its own transfer, licence and failure mode. Attaching
land_cover_class at a coastal or lakeside station is therefore worth
doing on purpose.
create_worldcover_fetcher ¶
create_worldcover_fetcher(*, base_url=WORLDCOVER_BASE_URL, weighting=None, uri=None)
Return a fetcher bound to one endpoint, for tests and for a mirror.
register_worldcover_provider ¶
Register the WorldCover fetcher under its catalogue provider key.
describe_class_shares ¶
Return a compact human-readable description of the footprint composition.
neptoon.external.remote_sensing.reporting¶
reporting ¶
Make the remote-sensing choices legible, queryable and reproducible.
Three audiences need the same facts in different shapes:
- the report, where a reader wants a summary and then only the stations that need attention, at thirty stations, prose buries the one that matters;
- the data audit, where a rerun has to be verifiable rather than hoped for, which needs the catalog version alongside the Neptoon version plus a content hash per fetched artefact, because provider version strings have been observed to disagree with the data they name;
- machine-readable output, so the product choice can be queried rather than only read.
Nothing here decides anything. It renders what :mod:.eligibility decided and
:mod:.attachment did.
NOTABLE_SPATIAL_OUTCOMES
module-attribute
¶
NOTABLE_TEMPORAL_OUTCOMES
module-attribute
¶
build_selection_export ¶
Return a JSON-ready record of which product served which variable.
Written so the choice is queryable, not merely readable: with automatic selection a variable can resolve to different products at different stations, and a later reader needs to be able to ask which, without parsing prose.
build_multi_sensor_report ¶
Return a summary table followed by an exceptions section.
The exceptions section is the point. A table of thirty stations is a table; the four that substituted an epoch or sit on a product boundary are the finding, and they must not have to be searched for.
describe_epoch_substitutions ¶
Return which epoch covered which part of the record, for the report.
An empty string when nothing was substituted, so the sentence appears only where it applies. Where it does apply it is not optional: a biomass column whose first two years carry a map from 2017 has to say so next to the numbers, not only in the log.
collect_citations ¶
Return the citations owed for every product used.
Attribution is a licence condition for several of these products rather than a courtesy, so it is discharged automatically instead of relying on the user to remember.
record_remote_sensing_audit ¶
record_remote_sensing_audit(result=None, *, sensor_name=None, correction_summary=None, snow_mask_result=None)
Write the catalog version, artefact provenance and correction use to the audit.
The catalog version travels next to the Neptoon version because the catalog is data that determines results: a coverage envelope, a validity predicate or a policy field changes the numbers. Recording only the code version would leave a rerun looking reproducible while the inputs had moved.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
result
|
RemoteSensingAttachmentResult | None
|
Attachment result whose artefacts are recorded. Omitted when only a correction summary is being written. |
None
|
sensor_name
|
str | None
|
Station the record belongs to. |
None
|
correction_summary
|
RemoteSensingCorrectionSummary | None
|
Written when a satellite product fed a neutron correction. It is a second step of the same audit rather than a second audit: the attachment records which artefacts were read, and this records what was then done with them: product, epochs, aggregation resolution and the range of the correction factor, which is the part a reviewer of the science needs. It arrives separately because the factor range does not exist until the correction has run. |
None
|
snow_mask_result
|
SnowMaskResult | None
|
Written when a gridded snow series flagged part of the soil-moisture product. A third step of the same audit rather than a third audit: a reader of the output has to be able to see which quantity decided the flag, at which threshold, and how much of the record it questions, because the values themselves are unchanged and the flag is the only record that anything was assessed at all. |
None
|
describe_selection_summary ¶
Return a compact one-station summary for the report.
neptoon.external.remote_sensing.scaling¶
scaling ¶
Explicit, reversible adjustment of a gridded reference series.
Gridded soil moisture, satellite retrieval and land-surface reanalysis alike, often reproduces the dynamics well while the absolute value is off, because the assimilation that constrains the dynamics does not constrain the mean. An adjustment therefore has to be possible. The design here makes it explicit, reversible and audited rather than convenient:
- the default performs nothing, because a shipped offset would be a site-specific number applied globally;
- the raw column is kept and the adjustment is written beside it, so the adjustment can be inspected and undone without re-running a fetch;
- the parameters carry their origin, so a number chosen by hand and a number fitted against a comparison period never look alike in a report;
- a factor supplied with an additive method is a validation error, not a silently ignored field, and a negative slope is refused rather than fitted.
The module deliberately sits below both external categories rather than inside
one. ERA5-Land soil moisture reaches Neptoon through external_meteo and
satellite soil moisture through remote_sensing, and both need exactly the
same treatment; two copies of this arithmetic would be two places for the
validation rules to drift apart.
ScalingOrigin
module-attribute
¶
ScalingError ¶
Bases: NeptoonExternalDataError, ValueError
Raised when a scaling policy or a fitted parameter is not usable.
ScalingPolicy
dataclass
¶
Offset/factor adjustment of one reference series.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
method
|
ScalingMethod
|
|
'none'
|
offset
|
float
|
Additive term in the variable's own unit. |
0.0
|
factor
|
float | None
|
Multiplicative term, valid only with |
None
|
origin
|
ScalingOrigin
|
Whether the parameters were chosen manually or derived from a comparison period. Reported, because the two are not equally defensible. |
'manual'
|
Notes
config_path is a class attribute rather than a field so a subclass can
name the configuration block it belongs to and error messages can point at
the key the user actually wrote.
apply ¶
Return the adjusted series, leaving the input untouched.
Missing values stay missing: an adjustment is not an opportunity to invent a value where the provider returned none.
coerce_scaling_policy ¶
coerce_scaling_policy(value, *, policy_type=ScalingPolicy)
Coerce mapping/Pydantic/policy input into a runtime policy object.
The nested validation is the point: a factor supplied with an additive method has to fail wherever the object is built, not only where a YAML file is read.
apply_scaling_to_frame ¶
Write the scaled companion column for one value column.
Returns the name of the column written, or None when the policy performs
nothing and no companion column is warranted. The raw column is never
overwritten: raw and scaled travel together so the adjustment stays
auditable and invertible.
fit_scaling_policy ¶
fit_scaling_policy(reference, target, *, method='offset', policy_type=ScalingPolicy, minimum_sample_size=MINIMUM_FIT_SAMPLE_SIZE)
Derive scaling parameters that map reference onto target.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
reference
|
Series
|
The gridded series to be adjusted. |
required |
target
|
Series
|
The series to adjust it towards, normally CRNS soil moisture over a comparison period. |
required |
method
|
ScalingMethod
|
|
'offset'
|
minimum_sample_size
|
int
|
Refuse to fit below this many paired observations. |
MINIMUM_FIT_SAMPLE_SIZE
|
Raises:
| Type | Description |
|---|---|
ScalingError
|
When too few pairs overlap, or when a linear fit returns a non-positive slope. The rejection is deliberate: a fit over a short noise window returns a negative slope without complaint, and accepting it would put "wetter implies drier" into a published series for a negligible gain in RMSE. |
describe_scaling_for_report ¶
Return the report line for one scaled column.
record_scaling_audit ¶
Write the scaling parameters and their origin to the data audit.
Recorded even when the method is none: "no adjustment was applied" is
itself a fact a later reader needs, and its absence reads as an omission.
replace_scaling_origin ¶
Return the same parameters with a different recorded origin.
neptoon.external.remote_sensing.raster_access¶
raster_access ¶
Read a footprint-sized raster window and reduce it to one number per station.
Everything raster-shaped in the remote-sensing category goes through here, so the
three raster providers differ only in which URL they resolve and what their
values mean. What this module contributes on top of :mod:neptoon.geo is the
CRNS part: the footprint disc, the Schroen weighting, the substitution of an
absent phenomenon, and the content hash that makes a rerun checkable.
Windows, never whole rasters. The German biomass map is 102120 x 86697 cells, about 71 GB as float64, while a 600 m window over HTTP range requests measures 0.03 s. One window is read for the whole set of positions and every position is then weighted inside it, so a rover track costs one read rather than one read per fix.
A geographic grid is not a metric grid. Both raster products in the initial product set are distributed in EPSG:4326 with metrically non-square pixels: about 10.00 m north-south against 6.14 m east-west at 52 degrees north for the biomass map, 9.28 m against 5.70 m for the land-cover map. Distances are therefore measured after transforming the cell centres into the UTM zone of the station, not by scaling degrees.
Do not replace this with :func:~neptoon.geo.crs.reproject_grid. That
function is the right tool for a caller who needs a metric grid, and reaching
for it here looks like the obvious simplification, which is exactly why this
paragraph exists. A footprint aggregate needs metric distances, not a metric
grid. Computing the distances directly means every native cell is counted exactly
once and the values being averaged are the product's own; reprojecting first puts
a nearest-neighbour kernel in front of them, which duplicates some cells, drops
others, and turns the rectangle-versus-footprint distinction into a third thing
again at the corners. The predecessor package resampled onto a metric grid
through cubo before weighting, and that is the main reason its numbers and
these differ at the third digit.
The footprint is a disc, not the window. The window is a rectangle around the
station because that is what a raster read is; the aggregate runs over the cells
within footprint_radius_m. That keeps the configured radius meaning what it
says and makes the valid fraction a property of the footprint rather than of an
implementation detail.
RasterFootprint
dataclass
¶
RasterFootprint(weighted_value, unweighted_value, valid_fraction, source_coverage_fraction=1.0, cell_count=0, valid_cell_count=0, heterogeneity=float('nan'), uncertainty=None, class_fractions=dict(), cell_size_m=(float('nan'), float('nan')), window_cell_count=0)
One station's footprint reduced from a raster window.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
weighted_value
|
float
|
Footprint value: the Schroen-weighted mean over the cells in the disc. |
required |
unweighted_value
|
float
|
The same cells without weighting, so the effect of the weighting stays visible instead of being asserted. |
required |
valid_fraction
|
float
|
Share of footprint cells carrying a real observation. What it means is
decided by the product's |
required |
cell_count
|
int
|
Cells in the footprint disc, and how many of them are valid. |
0
|
valid_cell_count
|
int
|
Cells in the footprint disc, and how many of them are valid. |
0
|
heterogeneity
|
float
|
Weighted standard deviation of the values. A property of the site, never an uncertainty. |
float('nan')
|
uncertainty
|
float | None
|
Propagated from a per-pixel uncertainty band under full spatial
correlation, or |
None
|
class_fractions
|
dict[int, float]
|
Weighted share per class code, for a categorical product. |
dict()
|
cell_size_m
|
tuple[float, float]
|
Measured east-west and north-south cell size in metres at the station. |
(float('nan'), float('nan'))
|
window_cell_count
|
int
|
Cells actually read, which is the rectangle rather than the disc. |
0
|
require_raster_support ¶
Fail with an installation instruction rather than an ImportError.
Every raster provider calls this before doing anything else, so a user without the optional extra is told the one command that fixes it instead of meeting a traceback from a transitive import.
read_footprint_window ¶
read_footprint_window(source, *, latitudes, longitudes, footprint_radius_m, use_cache=True, padding_m=WINDOW_PADDING_M)
Read one window covering every position plus its footprint radius.
Reading the union once rather than per position is what keeps a multi-station or rover request to a single range-request round trip.
calculate_metric_cell_distances ¶
Return each cell's distance to the station in metres, plus the cell size.
A geographic grid has its cell centres transformed into the UTM zone of the station; a projected grid keeps its own metric coordinates and only the station is transformed in. Either way the returned distances are metres and the cell size is measured rather than assumed, which is the whole point for a product whose pixels are 10.00 m in one direction and 6.14 m in the other.
aggregate_footprint ¶
aggregate_footprint(grid, *, latitude, longitude, footprint_radius_m, weighting=None, absent_value=None, uncertainty_grid=None, categorical=False, extraction='footprint')
Reduce one raster window to one value for one station.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
grid
|
GeoGrid
|
Window read around the station. |
required |
latitude
|
float
|
Station position in WGS84 degrees. |
required |
longitude
|
float
|
Station position in WGS84 degrees. |
required |
footprint_radius_m
|
float
|
Radius of the footprint disc, in metres. |
required |
weighting
|
FootprintWeighting | None
|
Shape parameters of the radial weighting. |
None
|
absent_value
|
float | None
|
Value contributed by cells where the phenomenon is genuinely absent.
Supplied only for a product whose |
None
|
uncertainty_grid
|
GeoGrid | None
|
Per-pixel uncertainty band on the same window, where the product ships one. |
None
|
categorical
|
bool
|
Whether the values are class codes, in which case shares replace a mean. |
False
|
extraction
|
ExtractionMode
|
|
'footprint'
|
calculate_bounding_box_edge_size_m ¶
Return the longest geodesic edge of a WGS84 box, in metres.
Ported from the predecessor package, where it replaced a Web-Mercator
computation whose metres are inflated by about sec(latitude), roughly
1.6x at 52 degrees north, which over-fetched every request by that factor.
Used here to state the real metric size of the window that was read, so the
performance guarantee is checkable against a number rather than a hope.
calculate_window_content_hash ¶
Hash the raster window a result actually rests on.
Not the whole artefact: the whole artefact is gigabytes and never transferred in the first place, so hashing it would mean downloading data the design exists to avoid. The window that produced the number is both cheaper and more to the point, it changes whenever the values behind the result change, which is the question "same configuration, six months later" is really asking.
Version strings do not answer that question. This project measured two
reasons why: one collection declares an open-ended temporal extent while its
items stop years earlier, and another ships files named fv5.0 inside a
v5.01 directory.
build_source_metadata ¶
Return the provider-independent facts worth carrying into provenance.
neptoon.external.remote_sensing.units¶
units ¶
Unit alignment between what providers publish and what Neptoon columns hold.
Both biomass products publish megagrams per hectare. Neptoon's biomass column is
kilograms per square metre, which is one of the two unit names the above-ground
biomass correction already understands (kg_m2 and bwe_mm), so the
conversion happens here, once, at the provider boundary. A column that arrives in
the provider's unit and is relabelled downstream is the classic route to a
factor-of-ten error that nothing catches.
The water-equivalent factor is imported rather than restated: the correction layer owns it, and a second copy would be a second thing to keep right.
BIOMASS_UNIT_FACTORS
module-attribute
¶
BIOMASS_UNIT_FACTORS = {'Mg/ha': MEGAGRAMS_PER_HECTARE_TO_KG_PER_M2, 't/ha': MEGAGRAMS_PER_HECTARE_TO_KG_PER_M2, 'kg/m2': 1.0, 'kg_m2': 1.0, 'bwe_mm': 1.0 / BIOMASS_WATER_EQUIVALENT_PER_KG_M2}
get_biomass_conversion_factor ¶
Return the factor converting a published biomass unit to kg/m2.
Raises:
| Type | Description |
|---|---|
ValueError
|
For a unit this module cannot align, rather than passing values through unchanged and letting the mismatch surface as a wrong number. |
convert_biomass_to_kg_per_m2 ¶
Convert published biomass values into Neptoon's kg/m2 biomass column.
convert_kg_per_m2_to_water_equivalent_mm ¶
Convert kg/m2 biomass into biomass water equivalent in mm.
Offered because several correction methods are parameterised in water
equivalent. Neptoon's column stays kg/m2 and the correction layer performs
this conversion itself when configured with biomass_units: bwe_mm; this
function exists so a user reading the reference column directly does not have
to rediscover the factor.
convert_values_to_canonical_unit ¶
Convert a provider/local value into the variable's canonical Neptoon unit.
The function is intentionally conservative. A declaration that cannot be converted is rejected instead of relabelling the raw number with a canonical semantic variable name. Additional unit families can be added here when a provider actually needs them.
neptoon.external_rs_registry¶
external_rs_registry ¶
Shared registry helpers for remote-sensing providers and products.
The YAML configuration layer, the product catalog, the eligibility preflight and
the CLI all need the same provider and product spellings. Keeping the aliases in
one lightweight module avoids the drift that appears as soon as a second place
starts normalising names, and mirrors external_meteo_registry so the two
external categories stay recognisably related.
Configuration names variables semantically (ndvi, soil_moisture) rather
than by provider band. A band-level surface invites requesting the wrong band and
receiving plausible-looking numbers that mean nothing: a MOD13Q1 subset requested
without a band returns blue reflectance, whose fill value differs from NDVI's, so
the mistake survives as data.
SUPPORTED_REMOTE_SENSING_PROVIDERS
module-attribute
¶
SUPPORTED_REMOTE_SENSING_PROVIDERS = frozenset({'modis_ornl', 'dlr', 'esa_worldcover', 'esa_cci_biomass', 'cdse', 'earth_engine', 'local_raster', 'local_timeseries'})
REMOTE_SENSING_PROVIDER_ALIASES
module-attribute
¶
REMOTE_SENSING_PROVIDER_ALIASES = {'modis': 'modis_ornl', 'viirs': 'modis_ornl', 'ornl': 'modis_ornl', 'ornl_daac': 'modis_ornl', 'modis_viirs': 'modis_ornl', 'modis_daac': 'modis_ornl', 'dlr_eoc': 'dlr', 'dlr_forest_structure': 'dlr', 'geoservice_dlr': 'dlr', 'worldcover': 'esa_worldcover', 'esa_world_cover': 'esa_worldcover', 'world_cover': 'esa_worldcover', 'cci_biomass': 'esa_cci_biomass', 'esacci_biomass': 'esa_cci_biomass', 'biomass_cci': 'esa_cci_biomass', 'ceda_biomass': 'esa_cci_biomass', 'copernicus': 'cdse', 'copernicus_dataspace': 'cdse', 'dataspace': 'cdse', 'copernicus_data_space_ecosystem': 'cdse', 'gee': 'earth_engine', 'google_earth_engine': 'earth_engine', 'earthengine': 'earth_engine', 'ee': 'earth_engine', 'local': 'local_raster', 'local_geotiff': 'local_raster', 'local_cog': 'local_raster', 'geotiff': 'local_raster', 'raster_file': 'local_raster', 'local_csv': 'local_timeseries', 'local_series': 'local_timeseries', 'local_time_series': 'local_timeseries'}
RASTER_READING_PROVIDERS
module-attribute
¶
SUPPORTED_REMOTE_SENSING_PRODUCTS
module-attribute
¶
SUPPORTED_REMOTE_SENSING_PRODUCTS = frozenset({'modis_ornl_mod13q1', 'modis_ornl_mcd15a3h', 'dlr_forest_structure_agbd', 'esa_worldcover_v200', 'esa_worldcover_v100', 'esa_cci_biomass_v6', 'copernicus_lcm10', 'copernicus_swi_ts'})
REMOTE_SENSING_PRODUCT_ALIASES
module-attribute
¶
REMOTE_SENSING_PRODUCT_ALIASES = {'mod13q1': 'modis_ornl_mod13q1', 'modis_mod13q1': 'modis_ornl_mod13q1', 'modis_ndvi': 'modis_ornl_mod13q1', 'modis_evi': 'modis_ornl_mod13q1', 'mcd15a3h': 'modis_ornl_mcd15a3h', 'modis_mcd15a3h': 'modis_ornl_mcd15a3h', 'modis_lai': 'modis_ornl_mcd15a3h', 'modis_fpar': 'modis_ornl_mcd15a3h', 'dlr_agbd': 'dlr_forest_structure_agbd', 'forest_structure_de_agbd_p1y': 'dlr_forest_structure_agbd', 'dlr_biomass': 'dlr_forest_structure_agbd', 'esa_worldcover': 'esa_worldcover_v200', 'worldcover': 'esa_worldcover_v200', 'worldcover_2021': 'esa_worldcover_v200', 'worldcover_v200': 'esa_worldcover_v200', 'worldcover_2020': 'esa_worldcover_v100', 'worldcover_v100': 'esa_worldcover_v100', 'esa_cci_biomass': 'esa_cci_biomass_v6', 'cci_biomass_v6': 'esa_cci_biomass_v6', 'esacci_biomass_v6_0': 'esa_cci_biomass_v6', 'lcm10': 'copernicus_lcm10', 'clc_backbone': 'copernicus_lcm10', 'copernicus_land_cover_10m': 'copernicus_lcm10', 'copernicus_swi': 'copernicus_swi_ts', 'swi': 'copernicus_swi_ts', 'swi_ts': 'copernicus_swi_ts', 'cdse_swi': 'copernicus_swi_ts'}
SUPPORTED_REMOTE_SENSING_VARIABLES
module-attribute
¶
SUPPORTED_REMOTE_SENSING_VARIABLES = frozenset({'above_ground_biomass', 'canopy_height', 'ndvi', 'evi', 'lai', 'fpar', 'soil_moisture', 'soil_water_index', 'snow_cover_fraction', 'land_cover_class'})
REMOTE_SENSING_VARIABLE_ALIASES
module-attribute
¶
REMOTE_SENSING_VARIABLE_ALIASES = {'agb': 'above_ground_biomass', 'agbd': 'above_ground_biomass', 'biomass': 'above_ground_biomass', 'above_ground_biomass_density': 'above_ground_biomass', 'rh95': 'canopy_height', 'tree_height': 'canopy_height', 'canopy_top_height': 'canopy_height', 'normalized_difference_vegetation_index': 'ndvi', 'enhanced_vegetation_index': 'evi', 'leaf_area_index': 'lai', 'fraction_of_photosynthetically_active_radiation': 'fpar', 'fapar': 'fpar', 'volumetric_soil_moisture': 'soil_moisture', 'swi': 'soil_water_index', 'soil_wetness_index': 'soil_water_index', 'snow_cover': 'snow_cover_fraction', 'snow_cover_extent': 'snow_cover_fraction', 'land_cover': 'land_cover_class', 'landcover': 'land_cover_class', 'land_cover_classification': 'land_cover_class'}
convert_iso_duration_to_timedelta ¶
Convert a supported ISO-8601 duration into a pandas timedelta.
add_iso_duration ¶
Add an ISO duration while preserving calendar month/year semantics.
subtract_iso_duration ¶
Subtract an ISO duration with calendar-aware month/year handling.
validate_uri_scheme ¶
Validate a user-declared product URI against the scheme allowlist.
normalize_remote_sensing_token ¶
Normalize provider/product/variable tokens before alias lookup.
normalize_remote_sensing_provider ¶
Normalize a provider name and validate that Neptoon supports it.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
provider
|
object
|
Provider name or alias from configuration. |
required |
keep_auto
|
bool
|
Return |
True
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If the provider is unknown. The message names every supported provider, because a typo is the common case and a list is the fix. |
normalize_remote_sensing_product ¶
Normalize a product name, or return None when none was requested.
A configuration may name a variable and leave the product to auto, so an
empty product is a valid, common input rather than an error.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
extra_products
|
Iterable[str]
|
Names declared by the caller rather than shipped here. A user-supplied
product is a first-class catalogue entry, so its name has to be
selectable by |
()
|
normalize_remote_sensing_variable ¶
Normalize a semantic variable name and validate that it is supported.
normalize_remote_sensing_variables ¶
Normalize an ordered collection of variable names without duplicates.