CoMet Toolkit

Introduction

CoMet Toolkit (Community Metrology Toolkit) is an open-source software project that provides Python tools for the easy handling and processing of dataset error-covariance information. The toolkit aims to abstract away the complexity dealing with measurement uncertainties. ​

CoMet Toolkit code is hosted on github with packages installable via Python Package Index (i.e., pip).

Technical Background

QA4EO ensures credible and reliable interpretation of environmental observations from satellites and in-situ measurements by requiring that associated uncertainty information is provided. It is also key to understand error-covariances in the data (e.g., separate handling of random and systematic uncertainties).

The approaches defined within QA4EO enable the Earth observation (EO) community to develop quantitative characterisation of uncertainty in EO data. However, practically implementing these methods is not trivial and can be time consuming. To facilitate this, the CoMet Toolkit was developed to provide a means to store and propagate uncertainty and error-correlation information. These tools allow the user to rely on quality-assured code, rather than having to reinvent the wheel, and lower the barrier to entry for users new to handling uncertainties.

Effects Tables are a useful way to record and report the information required to fully parameterise an error-covariance effect. However, to use this information in a processing chain, it must be provided digitally. The CoMet Toolkit defines a mechanism for this, with a metadata standard that enables the creation of Digital Effects Tables in NetCDF files. In this way, uncertainty information can be written, read, and processed in a way that machine-readable and preserved.

The CoMet Toolkit currently consists of three core modules:

• comet_maths is the module that has mathematical algorithms, linear algebra calculations, and random (correlated) sample generation, for use throughout the CoMet Toolkit.
• Obsarray is an extension to xarray for handling uncertainty quantified observation data, and storing uncertainty and covariance information in NetCDF files.
• Punpy is a tool for 'Propagation of UNcertainties in Python'. It propagates uncertainties on input quantities through any measurement function defined in python to uncertainties on the measurand, taking into account error-correlation information.

punpy interfaces with obsarray to make uncertainty propagation as efficient and easy to use as possible. The digital effects tables produced with obsarray can be propagated through measurement functions using punpy, without the need for providing additional information. The data has thus been encoded with all relevant error-covariance information, though users don’t need to interact with it. Together these tools enable both the experienced and inexperienced user to efficiently include uncertainties throughout their data processing, and thus make them more reliable and interpretable. Optional keywords provide the user with the flexibility to deal with all kinds of complex use cases.

For further info, refer to the CoMet Toolkit website as well as the punpy and obsarray documentation.