Datasets#

This page provides an overview of how datasets are defined, structured, and handled in the processing chain. The goal is to ensure consistent, analysis-ready data products that can be accessed efficiently and extended over time.

What is a Dataset?#

A dataset is the main unit of storage and analysis. A dataset:

represents a rectangular collection of variables with shared coordinates
is stored in Zarr format
corresponds to one deployment of an instrument or product

Datasets are designed so that they can be accessed directly for analysis via our intake catalog.

Analysis-Ready Datasets#

Our goal is to produce analysis-ready datasets. These datasets should be usable for scientific analysis without additional preprocessing.

An analysis-ready dataset should:

represent the whole observation period
follow CF conventions where possible
include appropriate metadata and units

Dataset Organization#

Datasets follow a hierarchical naming scheme that reflects their observational context.

The hierarchy is:

platform.campaign.name

or, if no campaign is relevant:

platform.name

The name should include relevant configuration (_c1) information and abide by versioning conventions.

Naming rules:

. separates hierarchical levels
_ is used within names

Access#

Datasets can be accessed through the intake catalog, for example:

import intake


cat = intake.open_catalog("https://tcodata.mpimet.mpg.de/catalog.yaml")
cat.BCO.wxt.to_dask()

/builds/tco/bco/docs/.venv/lib/python3.14/site-packages/intake_xarray/base.py:21: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  'dims': dict(self._ds.dims),

<xarray.Dataset> Size: 5GB
Dimensions:                 (time: 45382222, bnd: 2)
Coordinates:
  * time                    (time) datetime64[ns] 363MB 2010-12-16T16:24:00 ....
    alt                     float64 8B ...
    lat                     float64 8B ...
    lon                     float64 8B ...
Dimensions without coordinates: bnd
Data variables: (12/22)
    DIR                     (time) float64 363MB dask.array<chunksize=(4194304,), meta=np.ndarray>
    DIR_offset              (time) float32 182MB dask.array<chunksize=(4194304,), meta=np.ndarray>
    DL                      (time) float64 363MB dask.array<chunksize=(4194304,), meta=np.ndarray>
    DR                      (time) float64 363MB dask.array<chunksize=(4194304,), meta=np.ndarray>
    MNV                     (time) float32 182MB dask.array<chunksize=(4194304,), meta=np.ndarray>
    MXV                     (time) float32 182MB dask.array<chunksize=(4194304,), meta=np.ndarray>
    ...                      ...
    VH                      (time) float32 182MB dask.array<chunksize=(4194304,), meta=np.ndarray>
    VR                      (time) float32 182MB dask.array<chunksize=(4194304,), meta=np.ndarray>
    VS                      (time) float32 182MB dask.array<chunksize=(4194304,), meta=np.ndarray>
    air_temperature_status  (time) int8 45MB dask.array<chunksize=(4194304,), meta=np.ndarray>
    sensor_location         (time) int8 45MB dask.array<chunksize=(4194304,), meta=np.ndarray>
    time_bounds             (time, bnd) datetime64[ns] 726MB dask.array<chunksize=(4194304, 2), meta=np.ndarray>
Attributes:
    Conventions:           CF-1.12
    _logical_cutoff_date:  2026-10-20T00:00:00Z
    bcoproc_version:       0.0.0.post2301.dev0+58a3748
    featureType:           timeSeries
    institution:           Max Planck Institute for Meteorology, Hamburg
    license:               CC-BY-4.0
    location:              The Barbados Cloud Observatory (BCO), Deebles Poin...
    platform:              BCO
    source:                Vaisala WXT-520
    summary:               This dataset contains basic meteorological measure...
    title:                 WXT-2 ground station data from BCO (Level 1)
    tool_versions:         {"Python": "3.14.3 (main, Mar  3 2026, 14:59:53) [...

We provide for every catalog entry a ‘default’ dataset which, for the most part, are growing in time but processed with a constant code revision. However, if, for instance, for publication, there is a need for a static dataset, we can provide that.

More about versioning can be found in the versions section. We also provide time aggregations for analysis-ready datasets, accessible via the time parameter, where raw is the original, non-uniform grid. (More about time aggregations below.)

Paramaters can be used as follows:

cat.BCO.surfacemet(rev=rev, build=build, time='P1D').to_dask()

/builds/tco/bco/docs/.venv/lib/python3.14/site-packages/intake_xarray/base.py:21: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  'dims': dict(self._ds.dims),

<xarray.Dataset> Size: 393kB
Dimensions:         (time: 5783, bnd: 2)
Coordinates:
  * time            (time) datetime64[ns] 46kB 2010-12-16T12:00:00 ... 2026-1...
    alt             float64 8B ...
    lat             float64 8B ...
    lon             float64 8B ...
Dimensions without coordinates: bnd
Data variables:
    p               (time) float32 23kB dask.array<chunksize=(5783,), meta=np.ndarray>
    rain_intensity  (time) float32 23kB dask.array<chunksize=(5783,), meta=np.ndarray>
    raw_count       (time) float64 46kB dask.array<chunksize=(5783,), meta=np.ndarray>
    rh              (time) float32 23kB dask.array<chunksize=(5783,), meta=np.ndarray>
    ta              (time) float32 23kB dask.array<chunksize=(5783,), meta=np.ndarray>
    time_bounds     (time, bnd) datetime64[ns] 93kB dask.array<chunksize=(5783, 2), meta=np.ndarray>
    u               (time) float32 23kB dask.array<chunksize=(5783,), meta=np.ndarray>
    v               (time) float32 23kB dask.array<chunksize=(5783,), meta=np.ndarray>
    ws              (time) float32 23kB dask.array<chunksize=(5783,), meta=np.ndarray>
    ws:max          (time) float32 23kB dask.array<chunksize=(5783,), meta=np.ndarray>
    ws:min          (time) float32 23kB dask.array<chunksize=(5783,), meta=np.ndarray>
Attributes:
    Conventions:      CF-1.12
    _cutoff_date:     2026-10-15T23:55:00.000001000
    bcoproc_version:  0.0.0.post2339.dev0+dc6876f9
    institution:      Max Planck Institute for Meteorology, Hamburg
    license:          CC-BY-4.0
    location:         The Barbados Cloud Observatory (BCO), Deebles Point, Ba...
    platform:         BCO
    source:           https://swift.dkrz.de/v1/dkrz_948e7d4bbfbb445fbff5315fc...
    summary:          This dataset contains basic meteorological measurements...
    title:            WXT-2 ground station data from BCO (Level 2)
    tool_versions:    {"Python": "3.14.3 (main, Mar  3 2026, 14:59:53) [Clang...

Incrementally Growing Datasets#

Datasets are stored as Zarr archives and are extended continuously as new data becomes available. Rather than rewriting datasets, new data is appended as additional chunks. This enables efficient cloud-based storage and scalable analysis. The processing chain is orchestrated by an Airflow server.

Time aggregations#

Analysis-ready datasets are provided alongside their time aggregations: we provide a raw, non-uniform time grid, and subsequently 1-minute, 10-minute, 1-hour, and 1-day aggregate products for various use cases. Discrete bins are currently used to perform averages in the appriopriate spaces, where for the 1- and 10-minute products, an aggregate is produced regardless of the volume of raw data in the bin. For the 1-hour products, an aggregate is produced only if three or more 10-minute bins have data, and for the 1-day products, an aggregate is produced only if 22 or more hourly bins have data.

Dataset versions#

Versions are used to distinguish reproducible states of processed data. As a consequence, in order to attach a version to a dataset, one needs a revision or branch name of processing code (e.g. a Git commit hash), a start date, an end date, and a build number.

The following example table illustrates how these conventions impact dataset naming:

Object name	Revision/Branch	Start date	End date	Build
`BCO.wxt_2350a90_0`	`#2350a90`	2010-12-16	`None`	0
`BCO.wxt_afe854e_0`	`#afe854e`	2010-12-16	`None`	0
`BCO.wxt_main_0`	`main`	2010-12-16	`None`	0
`BCO.wxt_afe854e_1`	`#afe854e`	2024-08-10	2024-09-30	1

Here, BCO.wxt_2350a90_0 and BCO.wxt_afe854e_0 cover the same observation period but were processed with different software revisions. Using a non-changing code revision ensures that any dataset version can be reproduced in the future, with the exception of BCO.wxt_main_0, which uses the most up-to-date code revision. The main dataset therefore prioritises recency over reproducibility and stability. For this reason, the default datasets we provide are processed with stable code revisions but are still ever-growing.

Note that BCO.wxt_afe854e_1 uses the same code revision as BCO.wxt_afe854e_0 but has an explicit end date and spans the temporal extent of the ORCESTRA campaign. If there is a need for such a static dataset, they can be created upon request. However, they are not part of our standard practices, as the beauty of BCO data is its constant growth.

The dataset management page gives an overview of our internal protocols for re-processing and deleting datasets.

Coordinate Conventions#

Datasets follow CF conventions for coordinate naming where possible.

Primary coordinates include:

Coordinate	Meaning
`time`	UTC timestamp
`alt`	altitude above the geoid (meters)
`range`	line-of-sight distance from instrument (meters)
`lat`	latitude (`degrees_north`)
`lon`	longitude (`degrees_east`)

Primary coordinates must:

be strictly monotonic
contain no missing values

When sensor and data coordinates differ, sensor coordinates are provided using the prefix sensor_, e.g., sensor_alt.

Analysis-ready datasets also include a time_bounds variable, as well as a raw_count variable which displays how many time steps in the raw product are used in each aggregation.