# Dataset management

This article serves as an internal guideline on how we intend to manage the addition, reprocessing and, if needed, removal of datasets.

We keep a list of actively maintained datasets in [`datarecords.datasets`](https://gitlab.dkrz.de/tco/bco/-/blob/main/airflow/dags/datarecords/datasets.py).
This list represents a desired state.
We use workflow automation to automatically produce and remove datasets in our public store, should the desired state deviate from the actual state.
Thus, we only update the desired state manually and the system will adjust automatically.

## Managing datasets

A `DSCollection` is essentially a dataset in multiple versions. Within a `DSCollection`, `instances` is a list of `DS` objects which describe the individual datasets (dataset versions) themselves.

For all `DSCollection`s, there should exist one and only one `default` to which we point end users.

`DS` instances, once created, stay as they are, except:

1) Changing a dataset from ever-growing in time to fixed in time is acceptable because nothing about the processing, object ID, or metadata changes.
```diff
- DS(end=None)
+ DS(end=datetime)
```

2) Switching `forever` to `True` means that the dataset should **never** be deleted.
```diff
- DS(forever=False)
+ DS(forever=True)
```

3) `default` datasets can be switched if and only if the property of one `default` per `DSCollection` is maintained.
```diff
- instances=[DS(rev_or_branch="old", ..., default=True), DS(rev_or_branch="new", ..., default=False)]
+ instances=[DS(rev_or_branch="old", ..., default=False), DS(rev_or_branch="new", ..., default=True)]
```
Switching the `default` from one to another `DS` should only be done **once the new dataset has caught up processing**, i.e. first add a new `DS`, then let the processing catch up, then switch the `default`.
Otherwise, users would immediately be directed to a broken dataset.

## Deleting datasets

The following framework describes when and how to delete datasets.

1) If an error is discovered in a code revision used to process the data, and if `forever=False`, we can delete the dataset.
    If `default=True`, delete the dataset if and only if there exists another dataset spanning the entire temporal extent of the dataset in question which should be the new `default`.
    Switch to `default=True` for that dataset.

2) The same policy governs deleting a dataset in case of improved processing. However, if no errors exist in a legacy dataset for which processing has been altered and improved, **keep the dataset** unless storage is an issue.

```{admonition} Errors with dynamic code revisions
:class: important
In the case of (1), if the dataset in question is processed with incrementally updating code, the following steps should be taken to ensure safe deletion and continued development:
 - fix the error on the branch used to process the data
 - add a new instance of `DS(rev_or_branch="branch_in_question", ..., build=old_build+1)`
 - wait until re-processing for `DS(build=old_build+1)` has caught up to that of `DS(build=old_build)`
 - remove `DS(build=old_build)`
```