Dataset management

Dataset management#

This article serves as an internal guideline on how we intend to manage the addition, reprocessing and, if needed, removal of datasets.

We keep a list of actively maintained datasets in datarecords.datasets. This list represents a desired state. We use workflow automation to automatically produce and remove datasets in our public store, should the desired state deviate from the actual state. Thus, we only update the desired state manually and the system will adjust automatically.

Managing datasets#

A DSCollection is essentially a dataset in multiple versions. Within a DSCollection, instances is a list of DS objects which describe the individual datasets (dataset versions) themselves.

For all DSCollections, there should exist one and only one default to which we point end users.

DS instances, once created, stay as they are, except:

  1. Changing a dataset from ever-growing in time to fixed in time is acceptable because nothing about the processing, object ID, or metadata changes.

- DS(end=None)
+ DS(end=datetime)

  1. Switching forever to True means that the dataset should never be deleted.

- DS(forever=False)
+ DS(forever=True)

  1. default datasets can be switched if and only if the property of one default per DSCollection is maintained.

- instances=[DS(rev_or_branch="old", ..., default=True), DS(rev_or_branch="new", ..., default=False)]
+ instances=[DS(rev_or_branch="old", ..., default=False), DS(rev_or_branch="new", ..., default=True)]

Switching the default from one to another DS should only be done once the new dataset has caught up processing, i.e. first add a new DS, then let the processing catch up, then switch the default. Otherwise, users would immediately be directed to a broken dataset.

Deleting datasets#

The following framework describes when and how to delete datasets.

  1. If an error is discovered in a code revision used to process the data, and if forever=False, we can delete the dataset. If default=True, delete the dataset if and only if there exists another dataset spanning the entire temporal extent of the dataset in question which should be the new default. Switch to default=True for that dataset.

  2. The same policy governs deleting a dataset in case of improved processing. However, if no errors exist in a legacy dataset for which processing has been altered and improved, keep the dataset unless storage is an issue.

Errors with dynamic code revisions

In the case of (1), if the dataset in question is processed with incrementally updating code, the following steps should be taken to ensure safe deletion and continued development:

  • fix the error on the branch used to process the data

  • add a new instance of DS(rev_or_branch="branch_in_question", ..., build=old_build+1)

  • wait until re-processing for DS(build=old_build+1) has caught up to that of DS(build=old_build)

  • remove DS(build=old_build)