Freva - Data search and analysis framework¶

by: Brian Lewis, Christopher Kadow, Etor Lucio Eceiza, Andrej Fast, Martin Bergemann and many others

Common Problem I: Finding Data

Common Problem II: Use code of others

Common Problem III. Remember your own code

How can we search and access various datasets efficiently?
How can be streamline user data analysis tools reusable and reproducible?

Yet another solution - Freva framework¶

Freva comes with a three layer architecture

Example I: Searching and reading CMIP6 data¶

Python call:¶

import freva
import xarray as xr
files = freva.databrowser(
    project="cmip6",
    experiment="historical",
    model="mpi*",
    ensemble="r1i1*",
    variable="tas",
    time_frequency="mon"
)
dset = xr.open_mfdataset(files, combine="by_coords", parallel=True)

CLI call:¶

⋊  freva databrowser project=cmip6 experiment=historical model='mip*' ensemble='r1i1*' time_frequency=mon variable=tas

Web UI

Inspecting metadata

Example II: Data analysis tool / plugins¶

Calculate climate change signal and uncertainty in Düren¶

Where is the data for the ensemble?
How to select the municipality area?
...

Inspecting the results

Setting up data analysis tools¶

Freva plugin API is code agnostic
Yet: User defined plugins are defined and within an configured within an python API wrapper:

from evaluation_system.api import plugin
from evaluation_system.api.parameters import File, ParameterDictionary, SolrField
class PlottingTool(plugin.PluginAbstract):
"""Plugin that plots content of selected data."""

# Plugin setup
    tool_developer = {"name": "Jane Doe", "email": "Jane.Doe@example.com"}
    __category__ = "support"
    __short_description__ = "Plot data"
    __version__ = (2022, 1, 1)
    __parameters__ = ParameterDictionary(
        File(
            name="in_file",
            file_extension="nc",
            mandatory=True,
            help="Select the input netCDF file.",
        ),
        SolrField(
            name="variable",
            mandatory=False,
            facet="variable",
            help=("Select the variables(s) to be plotted"),
            max_items=1,
        )
    )

# Workflow definition of the actual tool
 def runTool(self, setup: dict[str, str] = {}) -> None:
        """This method calls the actual plugin."""
        self.call(
          f"python src/plot_map.py {setup['in_file']} -v {setup['variable']}"
        )

⋊  freva plugin -l
Animator: Animate data on lon/lat grids
ClimateChangeProfile: Create climate change signals.
ClimDexCalc: Calculate the ETCCDI climate extreme indices on the
             basis of daily temperature and precipitation data using
             the ClimDex software provided by PCIC.
Climpact: Process climate model data for input of impact model
CWT: Calculate Circulation Weather Type by mean sea level pressure.
EnsemblePlotter: Create and Plot maps of multi-model ensemble means and
                 standard deviations
MoviePlotter: Plots 2D lon/lat movies in GIF format
Papermill: Parametrize a given notebook
PlottingTool: Plot data

⋊  freva plugin plottingtool --doc
PlottingTool (v2022.1.1): Plot data
Options:
in_file  (default: <undefined>) [mandatory]
         Select the input netCDF file.
variable (default: <undefined>)
         Select the variables(s) to be plotted

Example III: Users adding their own data¶

import freva
import xarray as xr

user_data = freva.UserData()
user_data.add("eur-11b", "/tmp/my_awesome_data/outfile_?.nc",
              institute="clex", model="UM-RA2T",
              experiment="Bias-correct")
dset = xr.open_mfdataset(freva.databrowser(experiment="bias*", experiment="clex"))

CLI call:¶

⋊  freva-user-data add eur-11b /tmp/my_awesome_data/outfile_?.nc --institute clex --model UM-RA2T --experiment Bias-correct

A couple of additional features:¶

Users can search their own plugin history and other users results
Web UI suggests similar results on plugin config
Full integration of workload managers

What's next?¶

Freva has to be come a RestAPI
- Clients are easy to implement since we only need a couple of REST calls

We teach freva the Common Workflow Language (cwl) via toil:

cwlVersion: v1.0
class: CommandLineTool
hints:
 SoftwareRequirement:
    packages:
    - package: cdo
baseCommand: cdo

We add a system for ordering/pre-processing data
We add a simple interface for intake