ESM Catalog Background Information

A simplified view of ESM catalogs is that they consist of the paths of ESM output files, metadata about these files, e.g., names of data variables in the files and date ranges covered, and metadata about how the data files can be aggregated together.

More generally, data files can reside in the cloud, in which case URIs are used instead of paths, and data files can be in a format where their content is spread across multiple files, e.g., zarr. In the following, ESM output is referred to as assets, to recognize these generalizations.

Metadata about the assets referred to by an ESM catalog (paths or URIs, data variable names, date ranges, etc.) is stored in memory in a pandas DataFrame object, and on disk in a comma-separated values (CSV) file.

The primary data structure in intake-esm to support ESM catalogs is the esm_datastore class. Loosely speaking, this class consists of an ESM Catalog Specification and functions that operate on class objects. The ESM Catalog Specification consists of a dictionary of asset metadata that is available, i.e., columns in the above-mentioned CSV file, metadata about how the assets can be aggregated together, and some other metadata, such as a description of the catalog. The metadata regarding aggregation is stored in an aggregation control object.