Datasets#

This page provides an overview of how datasets are defined, structured, and handled in the processing chain. The goal is to ensure consistent, analysis-ready data products that can be accessed efficiently and extended over time.

What is a Dataset?#

A dataset is the main unit of storage and analysis. A dataset:

represents a rectangular collection of variables with shared coordinates
is stored in Zarr format
corresponds to one deployment of an instrument or product

Datasets are designed so that they can be accessed directly for analysis via our intake catalog.

Analysis-Ready Datasets#

Our goal is to produce analysis-ready datasets. These datasets should be usable for scientific analysis without additional preprocessing.

An analysis-ready dataset should:

represent the whole observation period
follow CF conventions where possible
include appropriate metadata and units

Dataset Organization#

Datasets follow a hierarchical naming scheme that reflects their observational context.

The hierarchy is:

platform.campaign.name

or, if no campaign is relevant:

platform.name

The name should include relevant configuration (_c1) information and abide by versioning conventions.

Naming rules:

. separates hierarchical levels
_ is used within names

Access#

Datasets can be accessed through the intake catalog, for example:

import intake


cat = intake.open_catalog("https://tcodata.mpimet.mpg.de/catalog.yaml")
cat.BCO.wxt.to_dask()

/builds/tco/bco/docs/.venv/lib/python3.12/site-packages/intake_xarray/base.py:21: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  'dims': dict(self._ds.dims),

<xarray.Dataset> Size: 5GB
Dimensions:                 (time: 44691148, bnd: 2)
Coordinates:
    alt                     float64 8B ...
    lat                     float64 8B ...
    lon                     float64 8B ...
  * time                    (time) datetime64[ns] 358MB 2010-12-16T16:24:00 ....
Dimensions without coordinates: bnd
Data variables: (12/22)
    DIR                     (time) float64 358MB dask.array<chunksize=(4194304,), meta=np.ndarray>
    DIR_offset              (time) float32 179MB dask.array<chunksize=(4194304,), meta=np.ndarray>
    DL                      (time) float64 358MB dask.array<chunksize=(4194304,), meta=np.ndarray>
    DR                      (time) float64 358MB dask.array<chunksize=(4194304,), meta=np.ndarray>
    MNV                     (time) float32 179MB dask.array<chunksize=(4194304,), meta=np.ndarray>
    MXV                     (time) float32 179MB dask.array<chunksize=(4194304,), meta=np.ndarray>
    ...                      ...
    VH                      (time) float32 179MB dask.array<chunksize=(4194304,), meta=np.ndarray>
    VR                      (time) float32 179MB dask.array<chunksize=(4194304,), meta=np.ndarray>
    VS                      (time) float32 179MB dask.array<chunksize=(4194304,), meta=np.ndarray>
    air_temperature_status  (time) int8 45MB dask.array<chunksize=(4194304,), meta=np.ndarray>
    sensor_location         (time) int8 45MB dask.array<chunksize=(4194304,), meta=np.ndarray>
    time_bounds             (time, bnd) datetime64[ns] 715MB dask.array<chunksize=(4194304, 2), meta=np.ndarray>
Attributes:
    Conventions:           CF-1.12
    _logical_cutoff_date:  2026-08-01T00:00:00Z
    bcoproc_version:       0.0.0.post1839.dev0+1641148
    featureType:           timeSeries
    institution:           Max Planck Institute for Meteorology, Hamburg
    license:               CC0-1.0
    location:              The Barbados Cloud Observatory (BCO), Deebles Poin...
    platform:              BCO
    source:                Vaisala WXT-520
    summary:               This dataset contains basic meteorological measure...
    title:                 WXT-2 ground station data from BCO (Level 1)
    tool_versions:         {"Python": "3.11.2 (main, Apr 28 2025, 14:11:48) [...

We provide for every catalog entry a ‘default’ dataset which, for the most part, are growing in time but processed with a constant code revision. However, if, for instance, for publication, there is a need for a static dataset, we can provide that too, accessible via the rev and build parameters in the catalog. In the event in which the stability of the processing is not a concern, rev=main is a growing dataset processed with the most recent version of the code. This dataset is not the default because its processing may crash and is not reproducible. More about versioning can be found in the versions section.

Paramaters can be used as follows:

cat.BCO.wxt(rev='main', build=0).to_dask()

/builds/tco/bco/docs/.venv/lib/python3.12/site-packages/intake_xarray/base.py:21: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  'dims': dict(self._ds.dims),

<xarray.Dataset> Size: 5GB
Dimensions:                 (time: 44649575, bnd: 2)
Coordinates:
    alt                     float64 8B ...
    lat                     float64 8B ...
    lon                     float64 8B ...
  * time                    (time) datetime64[ns] 357MB 2010-12-16T16:24:00 ....
Dimensions without coordinates: bnd
Data variables: (12/22)
    DIR                     (time) float64 357MB dask.array<chunksize=(262144,), meta=np.ndarray>
    DL                      (time) float64 357MB dask.array<chunksize=(262144,), meta=np.ndarray>
    DR                      (time) float64 357MB dask.array<chunksize=(262144,), meta=np.ndarray>
    MNV                     (time) float32 179MB dask.array<chunksize=(262144,), meta=np.ndarray>
    MXV                     (time) float32 179MB dask.array<chunksize=(262144,), meta=np.ndarray>
    P                       (time) float32 179MB dask.array<chunksize=(262144,), meta=np.ndarray>
    ...                      ...
    VR                      (time) float32 179MB dask.array<chunksize=(262144,), meta=np.ndarray>
    VS                      (time) float32 179MB dask.array<chunksize=(262144,), meta=np.ndarray>
    air_temperature_status  (time) int8 45MB dask.array<chunksize=(262144,), meta=np.ndarray>
    sensor_location         (time) int8 45MB dask.array<chunksize=(262144,), meta=np.ndarray>
    time_bounds             (time, bnd) datetime64[ns] 714MB dask.array<chunksize=(262144, 2), meta=np.ndarray>
    wind_direction_status   (time) int8 45MB dask.array<chunksize=(262144,), meta=np.ndarray>
Attributes:
    Conventions:           CF-1.12
    _logical_cutoff_date:  2026-07-22T00:00:00Z
    featureType:           timeSeries
    institution:           Max Planck Institute for Meteorology, Hamburg
    license:               CC0-1.0
    location:              The Barbados Cloud Observatory (BCO), Deebles Poin...
    platform:              BCO
    source:                Vaisala WXT-520
    summary:               This dataset contains basic meteorological measure...
    title:                 WXT-2 ground station data from BCO (Level 1)

Incrementally Growing Datasets#

Datasets are stored as Zarr archives and are extended continuously as new data becomes available. Rather than rewriting datasets, new data is appended as additional chunks. This enables efficient cloud-based storage and scalable analysis. The processing chain is orchestrated by an Airflow server.

Dataset versions#

Versions are used to distinguish reproducible states of processed data. As a consequence, in order to attach a version to a dataset, one needs a revision or branch name of processing code (e.g. a Git commit hash), a start date, an end date, and a build number.

The following example table illustrates how these conventions impact dataset naming:

Object name	Revision/Branch	Start date	End date	Build
`BCO.wxt_2350a90_0`	`#2350a90`	2010-12-16	`None`	0
`BCO.wxt_afe854e_0`	`#afe854e`	2010-12-16	`None`	0
`BCO.wxt_main_0`	`main`	2010-12-16	`None`	0
`BCO.wxt_afe854e_1`	`#afe854e`	2024-08-10	2024-09-30	1

Here, BCO.wxt_2350a90_0 and BCO.wxt_afe854e_0 cover the same observation period but were processed with different software revisions. Using a non-changing code revision ensures that any dataset version can be reproduced in the future, with the exception of BCO.wxt_main_0, which uses the most up-to-date code revision. The main dataset therefore prioritises recency over reproducibility and stability. For this reason, the default datasets we provide are processed with stable code revisions but are still ever-growing.

Note that BCO.wxt_afe854e_1 uses the same code revision as BCO.wxt_afe854e_0 but has an explicit end date and spans the temporal extent of the ORCESTRA campaign. If there is a need for such a static dataset, they can be created upon request. However, they are not part of our standard practices, as the beauty of BCO data is its constant growth.

The dataset management page gives an overview of our internal protocols for re-processing and deleting datasets.

Coordinate Conventions#

Datasets follow CF conventions for coordinate naming where possible.

Primary coordinates include:

Coordinate	Meaning
`time`	UTC timestamp
`alt`	altitude above the geoid (meters)
`range`	line-of-sight distance from instrument (meters)
`lat`	latitude (`degrees_north`)
`lon`	longitude (`degrees_east`)

Primary coordinates must:

be strictly monotonic
contain no missing values

When sensor and data coordinates differ, sensor coordinates are provided using the prefix sensor_, e.g., sensor_alt.

Datasets

Contents