Datasets

This page provides an overview of how datasets are defined, structured, and handled in the processing chain. The goal is to ensure consistent, analysis-ready data products that can be accessed efficiently and extended over time.

What is a Dataset?

A dataset is the main unit of storage and analysis. A dataset:

• represents a rectangular collection of variables with shared coordinates

• is stored in Zarr format

• corresponds to one deployment of an instrument or product

Datasets are designed so that they can be accessed directly for analysis via our intake catalog.

Analysis-Ready Datasets

Our goal is to produce analysis-ready datasets. These datasets should be usable for scientific analysis without additional preprocessing.

An analysis-ready dataset should:

• represent the whole observation period

• follow CF conventions where possible

• include appropriate metadata and units

Dataset Organization

Datasets follow a hierarchical naming scheme that reflects their observational context.

The hierarchy is:

platform.campaign.instrument

or, if no campaign is relevant:

platform.instrument

The instrument name should include configuration (_c1) and version (_v1) information.

Examples:

BCO.surfacemet_wxt_v1
BCO.lidar_CORAL_LR_t_c1_v1
METEOR.EUREC4A.lidar_LICHT_LR_t_v1

Naming rules:

• . separates hierarchical levels

• _ is used within names

Datasets can be accessed through the intake catalog, for example:

import intake


cat = intake.open_catalog("https://tcodata.mpimet.mpg.de/catalog.yaml")
cat.BCO.surfacemet_wxt_v1.to_dask()

/builds/tco/bco/docs/.venv/lib/python3.12/site-packages/intake_xarray/base.py:21: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  'dims': dict(self._ds.dims),

<xarray.Dataset> Size: 6GB
Dimensions:                 (time: 43806390, bnd: 2)
Coordinates:
    alt                     float64 8B ...
    lat                     float64 8B ...
    lon                     float64 8B ...
  * time                    (time) datetime64[ns] 350MB 2010-12-16T16:24:00 ....
Dimensions without coordinates: bnd
Data variables: (12/25)
    DIR                     (time) float64 350MB dask.array<chunksize=(262144,), meta=np.ndarray>
    DL                      (time) float64 350MB dask.array<chunksize=(262144,), meta=np.ndarray>
    DR                      (time) float64 350MB dask.array<chunksize=(262144,), meta=np.ndarray>
    H                       (time) float32 175MB dask.array<chunksize=(262144,), meta=np.ndarray>
    HDS                     (time) float32 175MB dask.array<chunksize=(262144,), meta=np.ndarray>
    HI                      (time) float32 175MB dask.array<chunksize=(262144,), meta=np.ndarray>
    ...                      ...
    VR                      (time) float32 175MB dask.array<chunksize=(262144,), meta=np.ndarray>
    VS                      (time) float32 175MB dask.array<chunksize=(262144,), meta=np.ndarray>
    air_temperature_status  (time) int8 44MB dask.array<chunksize=(262144,), meta=np.ndarray>
    sensor_location         (time) int8 44MB dask.array<chunksize=(262144,), meta=np.ndarray>
    time_bounds             (time, bnd) datetime64[ns] 701MB dask.array<chunksize=(262144, 2), meta=np.ndarray>
    wind_direction_status   (time) int8 44MB dask.array<chunksize=(262144,), meta=np.ndarray>
Attributes:
    Conventions:           CF-1.12
    _logical_cutoff_date:  2026-04-14T00:00:00Z
    bcoproc_version:       0.0.0.post1345.dev0+2215fe5
    featureType:           timeSeries
    institution:           Max Planck Institute for Meteorology, Hamburg
    license:               CC0-1.0
    location:              The Barbados Cloud Observatory (BCO), Deebles Poin...
    platform:              BCO
    source:                Vaisala WXT-520
    summary:               This dataset contains basic meteorological measure...
    title:                 WXT-2 ground station data from BCO (Level 1)
    tool_versions:         {"Python": "3.11.2 (main, Apr 28 2025, 14:11:48) [...

xarray.Dataset

• Dimensions:
  • time: 43806390
  • bnd: 2
• Coordinates: (4)
  • alt
    ()
    float64
    ...

    long_name :

    height above mean sea level

    standard_name :

    altitude

    units :

    m

    [1 values with dtype=float64]

  • lat
    ()
    float64
    ...

    long_name :

    north latitude

    units :

    degree_north

    valid_max :

    90.0

    valid_min :

    -90.0

    [1 values with dtype=float64]

  • lon
    ()
    float64
    ...

    long_name :

    east longitude

    units :

    degree_east

    valid_max :

    180.0

    valid_min :

    -180.0

    [1 values with dtype=float64]

  • time
    (time)
    datetime64[ns]
    2010-12-16T16:24:00 ... 2026-04-...

    axis :

    T

    bounds :

    time_bounds

    standard_name :

    time

    units_metadata :

    leap_seconds: utc

    array(['2010-12-16T16:24:00.000000000', '2010-12-16T16:24:10.000000000',
           '2010-12-16T16:24:20.000000000', ..., '2026-04-13T23:59:30.000000000',
           '2026-04-13T23:59:40.000000000', '2026-04-13T23:59:50.000000000'],
          shape=(43806390,), dtype='datetime64[ns]')

• Data variables: (25)
  • DIR
    (time)
    float64
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: circmean

    long_name :

    wind direction

    standard_name :

    wind_from_direction

    units :

    degree

    Array Chunk Bytes 334.22 MiB 2.00 MiB Shape (43806390,) (262144,) Dask graph 168 chunks in 2 graph layers Data type float64 numpy.ndarray

  • DL
    (time)
    float64
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: circleft(DIR)

    long_name :

    extreme left wind direction

    units :

    degree

    Array Chunk Bytes 334.22 MiB 2.00 MiB Shape (43806390,) (262144,) Dask graph 168 chunks in 2 graph layers Data type float64 numpy.ndarray

  • DR
    (time)
    float64
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: circright(DIR)

    long_name :

    extreme right wind direction

    units :

    degree

    Array Chunk Bytes 334.22 MiB 2.00 MiB Shape (43806390,) (262144,) Dask graph 168 chunks in 2 graph layers Data type float64 numpy.ndarray

  • H
    (time)
    float32
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: sum

    long_name :

    sum of hail within 10 sec time bin

    units :

    cm-2

    Array Chunk Bytes 167.11 MiB 1.00 MiB Shape (43806390,) (262144,) Dask graph 168 chunks in 2 graph layers Data type float32 numpy.ndarray

  • HDS
    (time)
    float32
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: sum

    long_name :

    duration of hail in second

    units :

    s

    Array Chunk Bytes 167.11 MiB 1.00 MiB Shape (43806390,) (262144,) Dask graph 168 chunks in 2 graph layers Data type float32 numpy.ndarray

  • HI
    (time)
    float32
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: mean

    long_name :

    mean hail intensity

    units :

    cm-2 h-1

    Array Chunk Bytes 167.11 MiB 1.00 MiB Shape (43806390,) (262144,) Dask graph 168 chunks in 2 graph layers Data type float32 numpy.ndarray

  • MNV
    (time)
    float32
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: minimum

    long_name :

    minimum wind speed

    units :

    m s-1

    Array Chunk Bytes 167.11 MiB 1.00 MiB Shape (43806390,) (262144,) Dask graph 168 chunks in 2 graph layers Data type float32 numpy.ndarray

  • MXV
    (time)
    float32
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: maximum

    long_name :

    maximum wind speed

    units :

    m s-1

    Array Chunk Bytes 167.11 MiB 1.00 MiB Shape (43806390,) (262144,) Dask graph 168 chunks in 2 graph layers Data type float32 numpy.ndarray

  • P
    (time)
    float32
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: mean

    long_name :

    pressure

    standard_name :

    air_pressure

    units :

    hPa

    Array Chunk Bytes 167.11 MiB 1.00 MiB Shape (43806390,) (262144,) Dask graph 168 chunks in 2 graph layers Data type float32 numpy.ndarray

  • R
    (time)
    float32
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: sum

    long_name :

    sum of rain within 10 sec time bin

    standard_name :

    rainfall_amount

    units :

    mm

    Array Chunk Bytes 167.11 MiB 1.00 MiB Shape (43806390,) (262144,) Dask graph 168 chunks in 2 graph layers Data type float32 numpy.ndarray

  • RDS
    (time)
    float32
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: sum

    long_name :

    duration of rain

    units :

    s

    Array Chunk Bytes 167.11 MiB 1.00 MiB Shape (43806390,) (262144,) Dask graph 168 chunks in 2 graph layers Data type float32 numpy.ndarray

  • RH
    (time)
    float32
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: mean

    long_name :

    relative humidity in percent

    standard_name :

    relative_humidity

    units :

    0.01

    Array Chunk Bytes 167.11 MiB 1.00 MiB Shape (43806390,) (262144,) Dask graph 168 chunks in 2 graph layers Data type float32 numpy.ndarray

  • RI
    (time)
    float32
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: mean

    long_name :

    rain intensity, running mean over one minute before timestamp

    units :

    mm h-1

    Array Chunk Bytes 167.11 MiB 1.00 MiB Shape (43806390,) (262144,) Dask graph 168 chunks in 2 graph layers Data type float32 numpy.ndarray

  • SDQ
    (time)
    float64
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: mean

    long_name :

    data quality in %

    units :

    0.01

    Array Chunk Bytes 334.22 MiB 2.00 MiB Shape (43806390,) (262144,) Dask graph 168 chunks in 2 graph layers Data type float64 numpy.ndarray

  • T
    (time)
    float32
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: mean

    long_name :

    temperature

    standard_name :

    air_temperature

    units :

    degree_Celsius

    Array Chunk Bytes 167.11 MiB 1.00 MiB Shape (43806390,) (262144,) Dask graph 168 chunks in 2 graph layers Data type float32 numpy.ndarray

  • TH
    (time)
    float32
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: mean

    long_name :

    heater temperature

    units :

    degree_Celsius

    Array Chunk Bytes 167.11 MiB 1.00 MiB Shape (43806390,) (262144,) Dask graph 168 chunks in 2 graph layers Data type float32 numpy.ndarray

  • TI
    (time)
    float32
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: mean

    long_name :

    internal temperature

    units :

    degree_Celsius

    Array Chunk Bytes 167.11 MiB 1.00 MiB Shape (43806390,) (262144,) Dask graph 168 chunks in 2 graph layers Data type float32 numpy.ndarray

  • VEL
    (time)
    float32
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: mean

    long_name :

    wind speed

    standard_name :

    wind_speed

    units :

    m s-1

    Array Chunk Bytes 167.11 MiB 1.00 MiB Shape (43806390,) (262144,) Dask graph 168 chunks in 2 graph layers Data type float32 numpy.ndarray

  • VH
    (time)
    float32
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: mean

    long_name :

    heater voltage

    units :

    V

    Array Chunk Bytes 167.11 MiB 1.00 MiB Shape (43806390,) (262144,) Dask graph 168 chunks in 2 graph layers Data type float32 numpy.ndarray

  • VR
    (time)
    float32
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: mean

    comment :

    nominally 3.5V

    long_name :

    reference voltage

    units :

    V

    Array Chunk Bytes 167.11 MiB 1.00 MiB Shape (43806390,) (262144,) Dask graph 168 chunks in 2 graph layers Data type float32 numpy.ndarray

  • VS
    (time)
    float32
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: mean

    long_name :

    supply voltage

    units :

    V

    Array Chunk Bytes 167.11 MiB 1.00 MiB Shape (43806390,) (262144,) Dask graph 168 chunks in 2 graph layers Data type float32 numpy.ndarray

  • air_temperature_status
    (time)
    int8
    dask.array<chunksize=(262144,), meta=np.ndarray>

    flag_meanings :

    ok warn broken

    flag_values :

    [0, 1, 2]

    long_name :

    air temperature measurement status flag

    Array Chunk Bytes 41.78 MiB 256.00 kiB Shape (43806390,) (262144,) Dask graph 168 chunks in 2 graph layers Data type int8 numpy.ndarray

  • sensor_location
    (time)
    int8
    dask.array<chunksize=(262144,), meta=np.ndarray>

    description :

    important for fine location-sensitive variables like winds

    flag_meanings :

    earli office licht coral

    flag_values :

    [1, 2, 3, 4]

    long_name :

    site shipping container where instrument is installed

    Array Chunk Bytes 41.78 MiB 256.00 kiB Shape (43806390,) (262144,) Dask graph 168 chunks in 2 graph layers Data type int8 numpy.ndarray

  • time_bounds
    (time, bnd)
    datetime64[ns]
    dask.array<chunksize=(262144, 2), meta=np.ndarray>

    Array Chunk Bytes 668.43 MiB 4.00 MiB Shape (43806390, 2) (262144, 2) Dask graph 168 chunks in 2 graph layers Data type datetime64[ns] numpy.ndarray

  • wind_direction_status
    (time)
    int8
    dask.array<chunksize=(262144,), meta=np.ndarray>

    description :

    shift indicates deviation from expected climatology of installation location; winds also sensitive to sensor_location

    flag_meanings :

    ok shift broken

    flag_values :

    [0, 1, 2]

    long_name :

    wind direction status flag

    Array Chunk Bytes 41.78 MiB 256.00 kiB Shape (43806390,) (262144,) Dask graph 168 chunks in 2 graph layers Data type int8 numpy.ndarray

• Indexes: (1)
  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2010-12-16 16:24:00', '2010-12-16 16:24:10',
                   '2010-12-16 16:24:20', '2010-12-16 16:24:30',
                   '2010-12-16 16:27:50', '2010-12-16 16:28:00',
                   '2010-12-16 16:28:10', '2010-12-16 16:28:20',
                   '2010-12-16 16:28:30', '2010-12-16 16:30:20',
                   ...
                   '2026-04-13 23:58:20', '2026-04-13 23:58:30',
                   '2026-04-13 23:58:40', '2026-04-13 23:58:50',
                   '2026-04-13 23:59:00', '2026-04-13 23:59:10',
                   '2026-04-13 23:59:20', '2026-04-13 23:59:30',
                   '2026-04-13 23:59:40', '2026-04-13 23:59:50'],
                  dtype='datetime64[ns]', name='time', length=43806390, freq=None))

• Attributes: (12)

  Conventions :

  CF-1.12

  _logical_cutoff_date :

  2026-04-14T00:00:00Z

  bcoproc_version :

  0.0.0.post1345.dev0+2215fe5

  featureType :

  timeSeries

  institution :

  Max Planck Institute for Meteorology, Hamburg

  license :

  CC0-1.0

  location :

  The Barbados Cloud Observatory (BCO), Deebles Point, Barbados, West Indies

  platform :

  BCO

  source :

  Vaisala WXT-520

  summary :

  This dataset contains basic meteorological measurements from a multi-sensor weather instrument is permanently installed on top of the office container at the BCO.

  title :

  WXT-2 ground station data from BCO (Level 1)

  tool_versions :

  {"Python": "3.11.2 (main, Apr 28 2025, 14:11:48) [GCC 12.2.0]", "ruff": "0.13.0", "toolz": "1.0.0", "fsspec": "2025.3.2", "zipp": "3.21.0", "idna": "3.10", "bcoproc": "0.0.0.post1345.dev0+2215fe5", "cftime": "1.6.4.post1", "frozenlist": "1.6.0", "python-dateutil": "2.9.0.post0", "partd": "1.4.2", "dask": "2025.4.1", "aiosignal": "1.3.2", "pluggy": "1.5.0", "cloudpickle": "3.1.1", "iniconfig": "2.1.0", "tzdata": "2025.2", "datarecords": "0.0.0.post1345.dev0+2215fe5", "multidict": "6.4.3", "pytz": "2025.2", "xarray": "2025.4.0", "yarl": "1.20.0", "propcache": "0.3.1", "six": "1.17.0", "aiohttp": "3.11.18", "pendulum": "3.1.0", "numpy": "2.2.5", "bcologs": "0.0.0.post1345.dev0+2215fe5", "certifi": "2025.4.26", "pyyaml": "6.0.2", "pytest": "8.3.5", "importlib_metadata": "8.7.0", "aiohappyeyeballs": "2.6.1", "locket": "1.0.0", "attrs": "25.3.0", "netcdf4": "1.7.2", "packaging": "25.0", "pandas": "2.2.3", "click": "8.1.8"}

Coordinate Conventions

Datasets follow CF conventions for coordinate naming where possible.

Primary coordinates include:

Coordinate	Meaning
time	UTC timestamp
alt	altitude above the geoid (meters)
range	line-of-sight distance from instrument (meters)
lat	latitude (degrees_north)
lon	longitude (degrees_east)

Primary coordinates must:

• be strictly monotonic

• contain no missing values

When sensor and data coordinates differ, sensor coordinates are provided using the prefix sensor_, e.g., sensor_alt.

Incrementally Growing Datasets

Datasets are stored as Zarr archives and are extended continuously as new data becomes available. Rather than rewriting datasets, new data is appended as additional chunks. This enables efficient cloud-based storage and scalable analysis. The processing chaing orchestrated by an Airflow server.