The databrowser command line interface#

This section introduces the usage of the freva-client databrowser sub command. Please see the Installation and configuration section on how to install and configure the command line interface.

After successful installation you can use the freva-client databrowser sub command

freva-client databrowser --help

Results

[1m                                                                                [0m
[1m [0m[1;33mUsage: [0m[1mfreva-client databrowser [OPTIONS] COMMAND [ARGS]...[0m[1m                   [0m[1m [0m
[1m                                                                                [0m
 Data search related commands                                                   
                                                                                
[2m╭─[0m[2m Options [0m[2m───────────────────────────────────────────────────────────────────[0m[2m─╮[0m
[2m│[0m [1;36m-[0m[1;36m-help[0m          Show this message and exit.                                  [2m│[0m
[2m╰──────────────────────────────────────────────────────────────────────────────╯[0m
[2m╭─[0m[2m Commands [0m[2m──────────────────────────────────────────────────────────────────[0m[2m─╮[0m
[2m│[0m [1;36mdata-count       [0m[1;36m [0m Count the databrowser search results                      [2m│[0m
[2m│[0m [1;36mdata-overview    [0m[1;36m [0m Get an overview over what is available in the             [2m│[0m
[2m│[0m [1;36m                  [0m databrowser.                                              [2m│[0m
[2m│[0m [1;36mdata-search      [0m[1;36m [0m Search the databrowser for datasets.                      [2m│[0m
[2m│[0m [1;36mintake-catalogue [0m[1;36m [0m Create an intake catalogue from the search.               [2m│[0m
[2m│[0m [1;36mmetadata-search  [0m[1;36m [0m Search databrowser for metadata (facets).                 [2m│[0m
[2m╰──────────────────────────────────────────────────────────────────────────────╯[0m

Searching for data locations#

Getting the locations of the data is probably the most common use case of the databrowser application. You can search for data locations by applying the data-search sub-command:

freva-client databrowser data-search --help

Results

[1m                                                                                [0m
[1m [0m[1;33mUsage: [0m[1mfreva-client databrowser data-search [OPTIONS] [SEARCH_KEYS]...[0m[1m        [0m[1m [0m
[1m                                                                                [0m
 Search the databrowser for datasets.                                           
                                                                                
[2m╭─[0m[2m Arguments [0m[2m─────────────────────────────────────────────────────────────────[0m[2m─╮[0m
[2m│[0m   search_keys      [1;2;33m[[0m[1;33mSEARCH_KEYS]...[0m  Refine your data search with this       [2m│[0m
[2m│[0m                                      `key=value` pair search parameters. The [2m│[0m
[2m│[0m                                      parameters could be, depending on the   [2m│[0m
[2m│[0m                                      DRS standard, flavour product, project  [2m│[0m
[2m│[0m                                      model etc.                              [2m│[0m
[2m│[0m                                      [2m[default: None]                        [0m [2m│[0m
[2m╰──────────────────────────────────────────────────────────────────────────────╯[0m
[2m╭─[0m[2m Options [0m[2m───────────────────────────────────────────────────────────────────[0m[2m─╮[0m
[2m│[0m [1;36m-[0m[1;36m-facet[0m                   [1;33mTEXT                   [0m  If you are not sure about [2m│[0m
[2m│[0m                                                    the correct search key's  [2m│[0m
[2m│[0m                                                    you can use the           [2m│[0m
[2m│[0m                                                    ``[1;36m-[0m[1;36m-facet[0m`` flag to       [2m│[0m
[2m│[0m                                                    search of any matching    [2m│[0m
[2m│[0m                                                    entries. For example      [2m│[0m
[2m│[0m                                                    [1;36m-[0m[1;36m-facet[0m 'era5' would      [2m│[0m
[2m│[0m                                                    allow you to search for   [2m│[0m
[2m│[0m                                                    any entries containing    [2m│[0m
[2m│[0m                                                    era5, regardless of       [2m│[0m
[2m│[0m                                                    project, product etc.     [2m│[0m
[2m│[0m                                                    [2m[default: None]          [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-uniq[0m[1;36m-key[0m       [1;32m-u[0m       [1;2;33m[[0m[1;33mfile[0m[1;2;33m|[0m[1;33muri[0m[1;2;33m][0m[1;33m             [0m  The type of search        [2m│[0m
[2m│[0m                                                    result, which can be      [2m│[0m
[2m│[0m                                                    either “file” or “uri”.   [2m│[0m
[2m│[0m                                                    This parameter determines [2m│[0m
[2m│[0m                                                    whether the search will   [2m│[0m
[2m│[0m                                                    be based on file paths or [2m│[0m
[2m│[0m                                                    Uniform Resource          [2m│[0m
[2m│[0m                                                    Identifiers               [2m│[0m
[2m│[0m                                                    [2m[default: file]          [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-flavour[0m        [1;32m-f[0m       [1;2;33m[[0m[1;33mfreva[0m[1;2;33m|[0m[1;33mcmip6[0m[1;2;33m|[0m[1;33mcmip5[0m[1;2;33m|[0m[1;33mcord[0m  The Data Reference Syntax [2m│[0m
[2m│[0m                           [1;33mex[0m[1;2;33m|[0m[1;33mnextgems[0m[1;2;33m][0m[1;33m           [0m  (DRS) standard specifying [2m│[0m
[2m│[0m                                                    the type of climate       [2m│[0m
[2m│[0m                                                    datasets to query.        [2m│[0m
[2m│[0m                                                    [2m[default: freva]         [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-time[0m[1;36m-select[0m    [1;32m-ts[0m      [1;2;33m[[0m[1;33mstrict[0m[1;2;33m|[0m[1;33mflexible[0m[1;2;33m|[0m[1;33mfile[0m[1;2;33m][0m[1;33m [0m  Operator that specifies   [2m│[0m
[2m│[0m                                                    how the time period is    [2m│[0m
[2m│[0m                                                    selected. Choose from     [2m│[0m
[2m│[0m                                                    flexible (default),       [2m│[0m
[2m│[0m                                                    strict or file.           [2m│[0m
[2m│[0m                                                    ``strict`` returns only   [2m│[0m
[2m│[0m                                                    those files that have the [2m│[0m
[2m│[0m                                                    *entire* time period      [2m│[0m
[2m│[0m                                                    covered. The time search  [2m│[0m
[2m│[0m                                                    ``2000 to 2012`` will not [2m│[0m
[2m│[0m                                                    select files containing   [2m│[0m
[2m│[0m                                                    data from 2010 to 2020    [2m│[0m
[2m│[0m                                                    with the ``strict``       [2m│[0m
[2m│[0m                                                    method. ``flexible`` will [2m│[0m
[2m│[0m                                                    select those files as     [2m│[0m
[2m│[0m                                                    ``flexible`` returns      [2m│[0m
[2m│[0m                                                    those files that have     [2m│[0m
[2m│[0m                                                    either start or end       [2m│[0m
[2m│[0m                                                    period covered. ``file``  [2m│[0m
[2m│[0m                                                    will only return files    [2m│[0m
[2m│[0m                                                    where the entire time     [2m│[0m
[2m│[0m                                                    period is contained       [2m│[0m
[2m│[0m                                                    within *one single* file. [2m│[0m
[2m│[0m                                                    [2m[default: flexible]      [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-zarr[0m                    [1;33m                       [0m  Create zarr stream files. [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-access[0m[1;36m-token[0m            [1;33mTEXT                   [0m  Use this access token for [2m│[0m
[2m│[0m                                                    authentication when       [2m│[0m
[2m│[0m                                                    creating a zarr stream    [2m│[0m
[2m│[0m                                                    files.                    [2m│[0m
[2m│[0m                                                    [2m[default: None]          [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-time[0m           [1;32m-t[0m       [1;33mTEXT                   [0m  Special search facet to   [2m│[0m
[2m│[0m                                                    refine/subset search      [2m│[0m
[2m│[0m                                                    results by time. This can [2m│[0m
[2m│[0m                                                    be a string               [2m│[0m
[2m│[0m                                                    representation of a time  [2m│[0m
[2m│[0m                                                    range or a single time    [2m│[0m
[2m│[0m                                                    step. The time steps have [2m│[0m
[2m│[0m                                                    to follow ISO-8601. Valid [2m│[0m
[2m│[0m                                                    strings are               [2m│[0m
[2m│[0m                                                    ``%Y-%m-%dT%H:%M`` to     [2m│[0m
[2m│[0m                                                    ``%Y-%m-%dT%H:%M`` for    [2m│[0m
[2m│[0m                                                    time ranges and           [2m│[0m
[2m│[0m                                                    ``%Y-%m-%dT%H:%M``.       [2m│[0m
[2m│[0m                                                    **Note**: You don't have  [2m│[0m
[2m│[0m                                                    to give the full string   [2m│[0m
[2m│[0m                                                    format to subset time     [2m│[0m
[2m│[0m                                                    steps ``%Y``, ``%Y-%m``   [2m│[0m
[2m│[0m                                                    etc are also valid.       [2m│[0m
[2m│[0m                                                    [2m[default: None]          [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-json[0m           [1;32m-j[0m       [1;33m                       [0m  Parse output in json      [2m│[0m
[2m│[0m                                                    format.                   [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-host[0m                    [1;33mTEXT                   [0m  Set the hostname of the   [2m│[0m
[2m│[0m                                                    databrowser, if not set   [2m│[0m
[2m│[0m                                                    (default) the hostname is [2m│[0m
[2m│[0m                                                    read from a config file   [2m│[0m
[2m│[0m                                                    [2m[default: None]          [0m [2m│[0m
[2m│[0m                  [1;32m-v[0m       [1;33mINTEGER                [0m  Increase verbosity        [2m│[0m
[2m│[0m                                                    [2m[default: 0]      [0m        [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-multi[0m[1;36m-version[0m           [1;33m                       [0m  Select all versions and   [2m│[0m
[2m│[0m                                                    not just the latest       [2m│[0m
[2m│[0m                                                    version (default).        [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-version[0m        [1;32m-V[0m       [1;33m                       [0m  Show version an exit      [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-help[0m                    [1;33m                       [0m  Show this message and     [2m│[0m
[2m│[0m                                                    exit.                     [2m│[0m
[2m╰──────────────────────────────────────────────────────────────────────────────╯[0m

The command expects a list of key=value pairs. The order of the pairs doesn’t really matter. Most important is that you don’t need to split the search according to the type of data you are searching for. You can search for files within observations, reanalysis and model data at the same time. Also important is that all queries are case insensitive. You can also search for attributes themselves instead of file paths. For example you can search for the list of variables available that satisfies a certain constraint (e.g. sampled 6hr, from a certain model, etc).

freva-client databrowser data-search project=observations variable=pr model=cp*

Results

https://swift.dkrz.de/v1/dkrz_a32dc0e8-2299-4239-a47d-6bf45c8b0160/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020000-201609022330.zarr
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022300-201609022330.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022200-201609022230.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022100-201609022130.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022000-201609022030.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021900-201609021930.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021800-201609021830.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021700-201609021730.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021600-201609021630.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021500-201609021530.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021400-201609021430.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021300-201609021330.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021200-201609021230.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021100-201609021130.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021000-201609021030.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020900-201609020930.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020800-201609020830.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020700-201609020730.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020600-201609020630.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020500-201609020530.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020400-201609020430.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020300-201609020330.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020200-201609020230.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020100-201609020130.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020000-201609020030.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022300-201609022330.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022200-201609022230.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022100-201609022130.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022000-201609022030.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021900-201609021930.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021800-201609021830.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021700-201609021730.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021600-201609021630.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021500-201609021530.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021400-201609021430.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021300-201609021330.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021200-201609021230.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021100-201609021130.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021000-201609021030.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020900-201609020930.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020800-201609020830.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020700-201609020730.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020600-201609020630.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020500-201609020530.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020400-201609020430.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020300-201609020330.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020200-201609020230.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020100-201609020130.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020000-201609020030.nc

There are many more options for defining a value for a given key:

Attribute syntax	Meaning
`attribute=value`	Search for files containing exactly that attribute
`attribute='val\*'`	Search for files containing a value for attribute that starts with the prefix val
`attribute='*lue'`	Search for files containing a value for attribute that ends with the suffix lue
`attribute='alu\'`	Search for files containing a value for attribute that has alu somewhere
`attribute='/.alu./'`	Search for files containing a value for attribute that matches the given regular expression (yes! you might use any regular expression to find what you want.)
`attribute=value1 attribute=value2` OR: `attribute={value1,value2}`	Search for files containing either value1 OR value2 for the given attribute (note that’s the same attribute twice!)
`attribute1=value1 attribute2=value2`	Search for files containing value1 for attribute1 AND value2 for attribute2
`attribute_not_=value`	Search for files NOT containing value
`attribute_not_=value1 attribute_not_=value2`	Search for files containing neither value1 nor value2

Note

When using * remember that your shell might give it a different meaning (normally it will try to match files with that name) to turn that off you can use backslash (key=*) or use quotes (key=’*’).

Streaming files via zarr#

Instead of getting the file locations on disk or tape, you can instruct the system to register zarr streams. Which means that instead of opening the data directly you can open it via zarr from anywhere. To do so simply add the --zarr flag.

Note

Before you can use the --zarr flag you will have to create an access token and use that token to log on to the system see also the Authentication chapter for more details on token creation.

token=$(freva-client auth -u janedoe|jq -r .access_token)
freva-client databrowser data-search dataset=cmip6-fs --zarr --access-token $token

Results

Give password for server authentication:  *****
http://localhost:7777/api/freva-data-portal/zarr/c48f9024-f61e-5682-9651-7b9a21d81048.zarr
http://localhost:7777/api/freva-data-portal/zarr/ff8b9acf-b652-5ce3-a26c-58f9bc4884a3.zarr

Special cases: Searching for times#

For example you want to know how many files we have between a certain time range To do so you can use the time search key to subset time steps and whole time ranges:

freva-client databrowser data-search project=observations -t '2016-09-02T22:15 to 2016-10'

Results

https://swift.dkrz.de/v1/dkrz_a32dc0e8-2299-4239-a47d-6bf45c8b0160/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020000-201609022330.zarr
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022300-201609022330.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022200-201609022230.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/model/regional/cordex/output/EUR-11/CLMcom/MPI-M-MPI-ESM-LR/historical/r0i0p0/CLMcom-CCLM4-8-17/v1/fx/orog/v20140515/orog_EUR-11_MPI-M-MPI-ESM-LR_historical_r1i1p1_CLMcom-CCLM4-8-17_v1_fx.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022300-201609022330.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022200-201609022230.nc

The default method for selecting time periods is flexible, which means all files are selected that cover at least start or end date. The strict method implies that the entire search time period has to be covered by the files. Using the strict method in the example above would only yield on file because the first file contains time steps prior to the start of the time period:

freva-client databrowser data-search project=observations -t '2016-09-02T22:15 to 2016-10' -ts strict

Results

/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022300-201609022330.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022300-201609022330.nc

Giving single time steps is also possible:

freva-client databrowser data-search project=observations -t 2016-09-02T22:10

Results

https://swift.dkrz.de/v1/dkrz_a32dc0e8-2299-4239-a47d-6bf45c8b0160/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020000-201609022330.zarr
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/random/test.json
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022300-201609022330.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022200-201609022230.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/model/regional/cordex/output/EUR-11/CLMcom/MPI-M-MPI-ESM-LR/historical/r0i0p0/CLMcom-CCLM4-8-17/v1/fx/orog/v20140515/orog_EUR-11_MPI-M-MPI-ESM-LR_historical_r1i1p1_CLMcom-CCLM4-8-17_v1_fx.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022300-201609022330.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022200-201609022230.nc
/arch/bb1203/freva_test/model/global/cmip6/CMIP6/CMIP/MPI-M/MPI-ESM1-2-LR/amip/r2i1p1f1/Amon/ua/gn/v20190815/ua_mon_MPI-ESM1-2-LR_amip_r2i1p1f1_197901-199812.nc

Note

The time format has to follow the ISO-8601 standard. Time ranges are indicated by the to keyword such as 2000 to 2100 or 2000-01 to 2100-12 and alike. Single time steps are given without the to keyword.

Creating intake-esm catalouges#

The intake-catalogue sub command allows you to create an intake-esm catalogue <https://intake-esm.readthedocs.io/en/stable/>_ from the current search. This can be useful to share the catalogue with others or merge datasets.

freva-client databrowser intake-catalogue --help

Results

[1m                                                                                [0m
[1m [0m[1;33mUsage: [0m[1mfreva-client databrowser intake-catalogue [OPTIONS] [SEARCH_KEYS]...[0m[1m   [0m[1m [0m
[1m                                                                                [0m
 Create an intake catalogue from the search.                                    
                                                                                
[2m╭─[0m[2m Arguments [0m[2m─────────────────────────────────────────────────────────────────[0m[2m─╮[0m
[2m│[0m   search_keys      [1;2;33m[[0m[1;33mSEARCH_KEYS]...[0m  Refine your data search with this       [2m│[0m
[2m│[0m                                      `key=value` pair search parameters. The [2m│[0m
[2m│[0m                                      parameters could be, depending on the   [2m│[0m
[2m│[0m                                      DRS standard, flavour product, project  [2m│[0m
[2m│[0m                                      model etc.                              [2m│[0m
[2m│[0m                                      [2m[default: None]                        [0m [2m│[0m
[2m╰──────────────────────────────────────────────────────────────────────────────╯[0m
[2m╭─[0m[2m Options [0m[2m───────────────────────────────────────────────────────────────────[0m[2m─╮[0m
[2m│[0m [1;36m-[0m[1;36m-facet[0m                   [1;33mTEXT                   [0m  If you are not sure about [2m│[0m
[2m│[0m                                                    the correct search key's  [2m│[0m
[2m│[0m                                                    you can use the           [2m│[0m
[2m│[0m                                                    ``[1;36m-[0m[1;36m-facet[0m`` flag to       [2m│[0m
[2m│[0m                                                    search of any matching    [2m│[0m
[2m│[0m                                                    entries. For example      [2m│[0m
[2m│[0m                                                    [1;36m-[0m[1;36m-facet[0m 'era5' would      [2m│[0m
[2m│[0m                                                    allow you to search for   [2m│[0m
[2m│[0m                                                    any entries containing    [2m│[0m
[2m│[0m                                                    era5, regardless of       [2m│[0m
[2m│[0m                                                    project, product etc.     [2m│[0m
[2m│[0m                                                    [2m[default: None]          [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-uniq[0m[1;36m-key[0m       [1;32m-u[0m       [1;2;33m[[0m[1;33mfile[0m[1;2;33m|[0m[1;33muri[0m[1;2;33m][0m[1;33m             [0m  The type of search        [2m│[0m
[2m│[0m                                                    result, which can be      [2m│[0m
[2m│[0m                                                    either “file” or “uri”.   [2m│[0m
[2m│[0m                                                    This parameter determines [2m│[0m
[2m│[0m                                                    whether the search will   [2m│[0m
[2m│[0m                                                    be based on file paths or [2m│[0m
[2m│[0m                                                    Uniform Resource          [2m│[0m
[2m│[0m                                                    Identifiers               [2m│[0m
[2m│[0m                                                    [2m[default: file]          [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-flavour[0m        [1;32m-f[0m       [1;2;33m[[0m[1;33mfreva[0m[1;2;33m|[0m[1;33mcmip6[0m[1;2;33m|[0m[1;33mcmip5[0m[1;2;33m|[0m[1;33mcord[0m  The Data Reference Syntax [2m│[0m
[2m│[0m                           [1;33mex[0m[1;2;33m|[0m[1;33mnextgems[0m[1;2;33m][0m[1;33m           [0m  (DRS) standard specifying [2m│[0m
[2m│[0m                                                    the type of climate       [2m│[0m
[2m│[0m                                                    datasets to query.        [2m│[0m
[2m│[0m                                                    [2m[default: freva]         [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-time[0m[1;36m-select[0m    [1;32m-ts[0m      [1;2;33m[[0m[1;33mstrict[0m[1;2;33m|[0m[1;33mflexible[0m[1;2;33m|[0m[1;33mfile[0m[1;2;33m][0m[1;33m [0m  Operator that specifies   [2m│[0m
[2m│[0m                                                    how the time period is    [2m│[0m
[2m│[0m                                                    selected. Choose from     [2m│[0m
[2m│[0m                                                    flexible (default),       [2m│[0m
[2m│[0m                                                    strict or file.           [2m│[0m
[2m│[0m                                                    ``strict`` returns only   [2m│[0m
[2m│[0m                                                    those files that have the [2m│[0m
[2m│[0m                                                    *entire* time period      [2m│[0m
[2m│[0m                                                    covered. The time search  [2m│[0m
[2m│[0m                                                    ``2000 to 2012`` will not [2m│[0m
[2m│[0m                                                    select files containing   [2m│[0m
[2m│[0m                                                    data from 2010 to 2020    [2m│[0m
[2m│[0m                                                    with the ``strict``       [2m│[0m
[2m│[0m                                                    method. ``flexible`` will [2m│[0m
[2m│[0m                                                    select those files as     [2m│[0m
[2m│[0m                                                    ``flexible`` returns      [2m│[0m
[2m│[0m                                                    those files that have     [2m│[0m
[2m│[0m                                                    either start or end       [2m│[0m
[2m│[0m                                                    period covered. ``file``  [2m│[0m
[2m│[0m                                                    will only return files    [2m│[0m
[2m│[0m                                                    where the entire time     [2m│[0m
[2m│[0m                                                    period is contained       [2m│[0m
[2m│[0m                                                    within *one single* file. [2m│[0m
[2m│[0m                                                    [2m[default: flexible]      [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-time[0m           [1;32m-t[0m       [1;33mTEXT                   [0m  Special search facet to   [2m│[0m
[2m│[0m                                                    refine/subset search      [2m│[0m
[2m│[0m                                                    results by time. This can [2m│[0m
[2m│[0m                                                    be a string               [2m│[0m
[2m│[0m                                                    representation of a time  [2m│[0m
[2m│[0m                                                    range or a single time    [2m│[0m
[2m│[0m                                                    step. The time steps have [2m│[0m
[2m│[0m                                                    to follow ISO-8601. Valid [2m│[0m
[2m│[0m                                                    strings are               [2m│[0m
[2m│[0m                                                    ``%Y-%m-%dT%H:%M`` to     [2m│[0m
[2m│[0m                                                    ``%Y-%m-%dT%H:%M`` for    [2m│[0m
[2m│[0m                                                    time ranges and           [2m│[0m
[2m│[0m                                                    ``%Y-%m-%dT%H:%M``.       [2m│[0m
[2m│[0m                                                    **Note**: You don't have  [2m│[0m
[2m│[0m                                                    to give the full string   [2m│[0m
[2m│[0m                                                    format to subset time     [2m│[0m
[2m│[0m                                                    steps ``%Y``, ``%Y-%m``   [2m│[0m
[2m│[0m                                                    etc are also valid.       [2m│[0m
[2m│[0m                                                    [2m[default: None]          [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-zarr[0m                    [1;33m                       [0m  Create zarr stream files, [2m│[0m
[2m│[0m                                                    as catalogue targets.     [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-access[0m[1;36m-token[0m            [1;33mTEXT                   [0m  Use this access token for [2m│[0m
[2m│[0m                                                    authentication when       [2m│[0m
[2m│[0m                                                    creating a zarr based     [2m│[0m
[2m│[0m                                                    intake catalogue.         [2m│[0m
[2m│[0m                                                    [2m[default: None]          [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-filename[0m       [1;32m-f[0m       [1;33mPATH                   [0m  Path to the file where    [2m│[0m
[2m│[0m                                                    the catalogue, should be  [2m│[0m
[2m│[0m                                                    written to. if None given [2m│[0m
[2m│[0m                                                    (default) the catalogue   [2m│[0m
[2m│[0m                                                    is parsed to stdout.      [2m│[0m
[2m│[0m                                                    [2m[default: None]          [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-host[0m                    [1;33mTEXT                   [0m  Set the hostname of the   [2m│[0m
[2m│[0m                                                    databrowser, if not set   [2m│[0m
[2m│[0m                                                    (default) the hostname is [2m│[0m
[2m│[0m                                                    read from a config file   [2m│[0m
[2m│[0m                                                    [2m[default: None]          [0m [2m│[0m
[2m│[0m                  [1;32m-v[0m       [1;33mINTEGER                [0m  Increase verbosity        [2m│[0m
[2m│[0m                                                    [2m[default: 0]      [0m        [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-multi[0m[1;36m-version[0m           [1;33m                       [0m  Select all versions and   [2m│[0m
[2m│[0m                                                    not just the latest       [2m│[0m
[2m│[0m                                                    version (default).        [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-version[0m        [1;32m-V[0m       [1;33m                       [0m  Show version an exit      [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-help[0m                    [1;33m                       [0m  Show this message and     [2m│[0m
[2m│[0m                                                    exit.                     [2m│[0m
[2m╰──────────────────────────────────────────────────────────────────────────────╯[0m

You can either set the --filename flag to save the catalogue to a .json file or pipe the catalogue to stdout (default). Just like for the data-search sub command you can instruct the system to create zarr file streams to access the data via zarr.

Query the number of occurrences#

In some cases it might be useful to know how many files are found in the databrowser for certain search constraints. In such cases you can use the data-count sub command to count the number of found files instead of getting the files themselves.

freva-client databrowser data-count --help

Results

[1m                                                                                [0m
[1m [0m[1;33mUsage: [0m[1mfreva-client databrowser data-count [OPTIONS] [SEARCH_KEYS]...[0m[1m         [0m[1m [0m
[1m                                                                                [0m
 Count the databrowser search results                                           
                                                                                
[2m╭─[0m[2m Arguments [0m[2m─────────────────────────────────────────────────────────────────[0m[2m─╮[0m
[2m│[0m   search_keys      [1;2;33m[[0m[1;33mSEARCH_KEYS]...[0m  Refine your data search with this       [2m│[0m
[2m│[0m                                      `key=value` pair search parameters. The [2m│[0m
[2m│[0m                                      parameters could be, depending on the   [2m│[0m
[2m│[0m                                      DRS standard, flavour product, project  [2m│[0m
[2m│[0m                                      model etc.                              [2m│[0m
[2m│[0m                                      [2m[default: None]                        [0m [2m│[0m
[2m╰──────────────────────────────────────────────────────────────────────────────╯[0m
[2m╭─[0m[2m Options [0m[2m───────────────────────────────────────────────────────────────────[0m[2m─╮[0m
[2m│[0m [1;36m-[0m[1;36m-facet[0m                     [1;33mTEXT                   [0m  If you are not sure     [2m│[0m
[2m│[0m                                                      about the correct       [2m│[0m
[2m│[0m                                                      search key's you can    [2m│[0m
[2m│[0m                                                      use the ``[1;36m-[0m[1;36m-facet[0m``     [2m│[0m
[2m│[0m                                                      flag to search of any   [2m│[0m
[2m│[0m                                                      matching entries. For   [2m│[0m
[2m│[0m                                                      example [1;36m-[0m[1;36m-facet[0m 'era5'  [2m│[0m
[2m│[0m                                                      would allow you to      [2m│[0m
[2m│[0m                                                      search for any entries  [2m│[0m
[2m│[0m                                                      containing era5,        [2m│[0m
[2m│[0m                                                      regardless of project,  [2m│[0m
[2m│[0m                                                      product etc.            [2m│[0m
[2m│[0m                                                      [2m[default: None]        [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-detail[0m           [1;32m-d[0m       [1;33m                       [0m  Separate the count by   [2m│[0m
[2m│[0m                                                      search facets.          [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-flavour[0m          [1;32m-f[0m       [1;2;33m[[0m[1;33mfreva[0m[1;2;33m|[0m[1;33mcmip6[0m[1;2;33m|[0m[1;33mcmip5[0m[1;2;33m|[0m[1;33mcord[0m  The Data Reference      [2m│[0m
[2m│[0m                             [1;33mex[0m[1;2;33m|[0m[1;33mnextgems[0m[1;2;33m][0m[1;33m           [0m  Syntax (DRS) standard   [2m│[0m
[2m│[0m                                                      specifying the type of  [2m│[0m
[2m│[0m                                                      climate datasets to     [2m│[0m
[2m│[0m                                                      query.                  [2m│[0m
[2m│[0m                                                      [2m[default: freva]       [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-time[0m[1;36m-select[0m      [1;32m-ts[0m      [1;2;33m[[0m[1;33mstrict[0m[1;2;33m|[0m[1;33mflexible[0m[1;2;33m|[0m[1;33mfile[0m[1;2;33m][0m[1;33m [0m  Operator that specifies [2m│[0m
[2m│[0m                                                      how the time period is  [2m│[0m
[2m│[0m                                                      selected. Choose from   [2m│[0m
[2m│[0m                                                      flexible (default),     [2m│[0m
[2m│[0m                                                      strict or file.         [2m│[0m
[2m│[0m                                                      ``strict`` returns only [2m│[0m
[2m│[0m                                                      those files that have   [2m│[0m
[2m│[0m                                                      the *entire* time       [2m│[0m
[2m│[0m                                                      period covered. The     [2m│[0m
[2m│[0m                                                      time search ``2000 to   [2m│[0m
[2m│[0m                                                      2012`` will not select  [2m│[0m
[2m│[0m                                                      files containing data   [2m│[0m
[2m│[0m                                                      from 2010 to 2020 with  [2m│[0m
[2m│[0m                                                      the ``strict`` method.  [2m│[0m
[2m│[0m                                                      ``flexible`` will       [2m│[0m
[2m│[0m                                                      select those files as   [2m│[0m
[2m│[0m                                                      ``flexible`` returns    [2m│[0m
[2m│[0m                                                      those files that have   [2m│[0m
[2m│[0m                                                      either start or end     [2m│[0m
[2m│[0m                                                      period covered.         [2m│[0m
[2m│[0m                                                      ``file`` will only      [2m│[0m
[2m│[0m                                                      return files where the  [2m│[0m
[2m│[0m                                                      entire time period is   [2m│[0m
[2m│[0m                                                      contained within *one   [2m│[0m
[2m│[0m                                                      single* file.           [2m│[0m
[2m│[0m                                                      [2m[default: flexible]    [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-time[0m             [1;32m-t[0m       [1;33mTEXT                   [0m  Special search facet to [2m│[0m
[2m│[0m                                                      refine/subset search    [2m│[0m
[2m│[0m                                                      results by time. This   [2m│[0m
[2m│[0m                                                      can be a string         [2m│[0m
[2m│[0m                                                      representation of a     [2m│[0m
[2m│[0m                                                      time range or a single  [2m│[0m
[2m│[0m                                                      time step. The time     [2m│[0m
[2m│[0m                                                      steps have to follow    [2m│[0m
[2m│[0m                                                      ISO-8601. Valid strings [2m│[0m
[2m│[0m                                                      are ``%Y-%m-%dT%H:%M``  [2m│[0m
[2m│[0m                                                      to ``%Y-%m-%dT%H:%M``   [2m│[0m
[2m│[0m                                                      for time ranges and     [2m│[0m
[2m│[0m                                                      ``%Y-%m-%dT%H:%M``.     [2m│[0m
[2m│[0m                                                      **Note**: You don't     [2m│[0m
[2m│[0m                                                      have to give the full   [2m│[0m
[2m│[0m                                                      string format to subset [2m│[0m
[2m│[0m                                                      time steps ``%Y``,      [2m│[0m
[2m│[0m                                                      ``%Y-%m`` etc are also  [2m│[0m
[2m│[0m                                                      valid.                  [2m│[0m
[2m│[0m                                                      [2m[default: None]        [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-extended[0m[1;36m-search[0m  [1;32m-e[0m       [1;33m                       [0m  Retrieve information on [2m│[0m
[2m│[0m                                                      additional search keys. [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-multi[0m[1;36m-version[0m             [1;33m                       [0m  Select all versions and [2m│[0m
[2m│[0m                                                      not just the latest     [2m│[0m
[2m│[0m                                                      version (default).      [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-host[0m                      [1;33mTEXT                   [0m  Set the hostname of the [2m│[0m
[2m│[0m                                                      databrowser, if not set [2m│[0m
[2m│[0m                                                      (default) the hostname  [2m│[0m
[2m│[0m                                                      is read from a config   [2m│[0m
[2m│[0m                                                      file                    [2m│[0m
[2m│[0m                                                      [2m[default: None]        [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-json[0m             [1;32m-j[0m       [1;33m                       [0m  Parse output in json    [2m│[0m
[2m│[0m                                                      format.                 [2m│[0m
[2m│[0m                    [1;32m-v[0m       [1;33mINTEGER                [0m  Increase verbosity      [2m│[0m
[2m│[0m                                                      [2m[default: 0]      [0m      [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-version[0m          [1;32m-V[0m       [1;33m                       [0m  Show version an exit    [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-help[0m                      [1;33m                       [0m  Show this message and   [2m│[0m
[2m│[0m                                                      exit.                   [2m│[0m
[2m╰──────────────────────────────────────────────────────────────────────────────╯[0m

By default the data-count sub command will display the total number of items matching your search query. For example:

freva-client databrowser data-count project=observations

Results

If you want to group the number of occurrences by search categories (facets) use the -d or --detail flag:

freva-client databrowser data-count -d project=observations

Results

ensemble: r1i1p1[52]
experiment: cmorph[49], oc5[3]
institute: cpc[49], noaa[3]
model: cpc[25], cpc-cmorph[24], nodc[3]
product: grid[49], reanalysis[3]
project: observations[52]
realm: atmos[49], ocean[3]
time_aggregation: mean[52]
time_frequency: 1min[49], mon[3]
variable: hc700[3], pr[49]

Retrieving the available metadata#

Sometime it might be useful to know about possible values that search attributes. For this you use the metadata-search sub command:

freva-client databrowser metadata-search --help

Results

[1m                                                                                [0m
[1m [0m[1;33mUsage: [0m[1mfreva-client databrowser metadata-search [OPTIONS] [SEARCH_KEYS]...[0m[1m    [0m[1m [0m
[1m                                                                                [0m
 Search databrowser for metadata (facets).                                      
                                                                                
[2m╭─[0m[2m Arguments [0m[2m─────────────────────────────────────────────────────────────────[0m[2m─╮[0m
[2m│[0m   search_keys      [1;2;33m[[0m[1;33mSEARCH_KEYS]...[0m  Refine your data search with this       [2m│[0m
[2m│[0m                                      `key=value` pair search parameters. The [2m│[0m
[2m│[0m                                      parameters could be, depending on the   [2m│[0m
[2m│[0m                                      DRS standard, flavour product, project  [2m│[0m
[2m│[0m                                      model etc.                              [2m│[0m
[2m│[0m                                      [2m[default: None]                        [0m [2m│[0m
[2m╰──────────────────────────────────────────────────────────────────────────────╯[0m
[2m╭─[0m[2m Options [0m[2m───────────────────────────────────────────────────────────────────[0m[2m─╮[0m
[2m│[0m [1;36m-[0m[1;36m-facet[0m                     [1;33mTEXT                   [0m  If you are not sure     [2m│[0m
[2m│[0m                                                      about the correct       [2m│[0m
[2m│[0m                                                      search key's you can    [2m│[0m
[2m│[0m                                                      use the ``[1;36m-[0m[1;36m-facet[0m``     [2m│[0m
[2m│[0m                                                      flag to search of any   [2m│[0m
[2m│[0m                                                      matching entries. For   [2m│[0m
[2m│[0m                                                      example [1;36m-[0m[1;36m-facet[0m 'era5'  [2m│[0m
[2m│[0m                                                      would allow you to      [2m│[0m
[2m│[0m                                                      search for any entries  [2m│[0m
[2m│[0m                                                      containing era5,        [2m│[0m
[2m│[0m                                                      regardless of project,  [2m│[0m
[2m│[0m                                                      product etc.            [2m│[0m
[2m│[0m                                                      [2m[default: None]        [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-flavour[0m          [1;32m-f[0m       [1;2;33m[[0m[1;33mfreva[0m[1;2;33m|[0m[1;33mcmip6[0m[1;2;33m|[0m[1;33mcmip5[0m[1;2;33m|[0m[1;33mcord[0m  The Data Reference      [2m│[0m
[2m│[0m                             [1;33mex[0m[1;2;33m|[0m[1;33mnextgems[0m[1;2;33m][0m[1;33m           [0m  Syntax (DRS) standard   [2m│[0m
[2m│[0m                                                      specifying the type of  [2m│[0m
[2m│[0m                                                      climate datasets to     [2m│[0m
[2m│[0m                                                      query.                  [2m│[0m
[2m│[0m                                                      [2m[default: freva]       [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-time[0m[1;36m-select[0m      [1;32m-ts[0m      [1;2;33m[[0m[1;33mstrict[0m[1;2;33m|[0m[1;33mflexible[0m[1;2;33m|[0m[1;33mfile[0m[1;2;33m][0m[1;33m [0m  Operator that specifies [2m│[0m
[2m│[0m                                                      how the time period is  [2m│[0m
[2m│[0m                                                      selected. Choose from   [2m│[0m
[2m│[0m                                                      flexible (default),     [2m│[0m
[2m│[0m                                                      strict or file.         [2m│[0m
[2m│[0m                                                      ``strict`` returns only [2m│[0m
[2m│[0m                                                      those files that have   [2m│[0m
[2m│[0m                                                      the *entire* time       [2m│[0m
[2m│[0m                                                      period covered. The     [2m│[0m
[2m│[0m                                                      time search ``2000 to   [2m│[0m
[2m│[0m                                                      2012`` will not select  [2m│[0m
[2m│[0m                                                      files containing data   [2m│[0m
[2m│[0m                                                      from 2010 to 2020 with  [2m│[0m
[2m│[0m                                                      the ``strict`` method.  [2m│[0m
[2m│[0m                                                      ``flexible`` will       [2m│[0m
[2m│[0m                                                      select those files as   [2m│[0m
[2m│[0m                                                      ``flexible`` returns    [2m│[0m
[2m│[0m                                                      those files that have   [2m│[0m
[2m│[0m                                                      either start or end     [2m│[0m
[2m│[0m                                                      period covered.         [2m│[0m
[2m│[0m                                                      ``file`` will only      [2m│[0m
[2m│[0m                                                      return files where the  [2m│[0m
[2m│[0m                                                      entire time period is   [2m│[0m
[2m│[0m                                                      contained within *one   [2m│[0m
[2m│[0m                                                      single* file.           [2m│[0m
[2m│[0m                                                      [2m[default: flexible]    [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-time[0m             [1;32m-t[0m       [1;33mTEXT                   [0m  Special search facet to [2m│[0m
[2m│[0m                                                      refine/subset search    [2m│[0m
[2m│[0m                                                      results by time. This   [2m│[0m
[2m│[0m                                                      can be a string         [2m│[0m
[2m│[0m                                                      representation of a     [2m│[0m
[2m│[0m                                                      time range or a single  [2m│[0m
[2m│[0m                                                      time step. The time     [2m│[0m
[2m│[0m                                                      steps have to follow    [2m│[0m
[2m│[0m                                                      ISO-8601. Valid strings [2m│[0m
[2m│[0m                                                      are ``%Y-%m-%dT%H:%M``  [2m│[0m
[2m│[0m                                                      to ``%Y-%m-%dT%H:%M``   [2m│[0m
[2m│[0m                                                      for time ranges and     [2m│[0m
[2m│[0m                                                      ``%Y-%m-%dT%H:%M``.     [2m│[0m
[2m│[0m                                                      **Note**: You don't     [2m│[0m
[2m│[0m                                                      have to give the full   [2m│[0m
[2m│[0m                                                      string format to subset [2m│[0m
[2m│[0m                                                      time steps ``%Y``,      [2m│[0m
[2m│[0m                                                      ``%Y-%m`` etc are also  [2m│[0m
[2m│[0m                                                      valid.                  [2m│[0m
[2m│[0m                                                      [2m[default: None]        [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-extended[0m[1;36m-search[0m  [1;32m-e[0m       [1;33m                       [0m  Retrieve information on [2m│[0m
[2m│[0m                                                      additional search keys. [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-multi[0m[1;36m-version[0m             [1;33m                       [0m  Select all versions and [2m│[0m
[2m│[0m                                                      not just the latest     [2m│[0m
[2m│[0m                                                      version (default).      [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-host[0m                      [1;33mTEXT                   [0m  Set the hostname of the [2m│[0m
[2m│[0m                                                      databrowser, if not set [2m│[0m
[2m│[0m                                                      (default) the hostname  [2m│[0m
[2m│[0m                                                      is read from a config   [2m│[0m
[2m│[0m                                                      file                    [2m│[0m
[2m│[0m                                                      [2m[default: None]        [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-json[0m             [1;32m-j[0m       [1;33m                       [0m  Parse output in json    [2m│[0m
[2m│[0m                                                      format.                 [2m│[0m
[2m│[0m                    [1;32m-v[0m       [1;33mINTEGER                [0m  Increase verbosity      [2m│[0m
[2m│[0m                                                      [2m[default: 0]      [0m      [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-version[0m          [1;32m-V[0m       [1;33m                       [0m  Show version an exit    [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-help[0m                      [1;33m                       [0m  Show this message and   [2m│[0m
[2m│[0m                                                      exit.                   [2m│[0m
[2m╰──────────────────────────────────────────────────────────────────────────────╯[0m

Just like with any other databrowser command you can apply different search constraints when acquiring metadata

freva-client databrowser metadata-search project=observations

Results

ensemble: r1i1p1
experiment: cmorph, oc5
institute: cpc, noaa
model: cpc, cpc-cmorph, nodc
product: grid, reanalysis
project: observations
realm: atmos, ocean
time_aggregation: mean
time_frequency: 1min, mon
variable: hc700, pr

By default the command will display only the most commonly used metadata keys. To retrieve all metadata you can use the -e or --extended-search flag.

freva-client databrowser metadata-search -e project=observations

Results

cmor_table: 30min, omon
dataset: obs-fs, obs-hsm, obs-swfit, reana-fs, reana-hsm, reana-swfit
driving_model: 
ensemble: r1i1p1
experiment: cmorph, oc5
format: nc, zarr
fs_type: posix
grid_id: 
grid_label: gn
institute: cpc, noaa
level_type: 2d
model: cpc, cpc-cmorph, nodc
product: grid, reanalysis
project: observations
rcm_name: 
rcm_version: 
realm: atmos, ocean
time_aggregation: mean
time_frequency: 1min, mon
user: 
variable: hc700, pr

Sometimes you don’t exactly know the exact names of the search keys and want retrieve all file objects that match a certain category. For example for getting all ocean reanalysis datasets you can apply the --facet flag:

freva-client databrowser metadata-search -e realm=ocean --facet 'rean*'

Results

ensemble: r1i1p1
experiment: oc5
institute: noaa
model: nodc
product: reanalysis
project: observations
realm: ocean
time_aggregation: mean
time_frequency: mon
variable: hc700

Expert tip: Getting metadata for certain files#

In some cases it might be useful to retrieve metadata for certain file or object store locations. For example if we wanted to retrieve the metadata of those files on tape:

freva-client databrowser metadata-search -e file="/arch/*"

Results

cmor_table: 1day, 30min, 3hr, amon, inst-or-acc, omon
dataset: cmip6-hsm, cordex-hsm, nextgems, obs-hsm, reana-hsm
driving_model: mpi-m-mpi-esm-lr, ncc-noresm1-m
ensemble: r1i1p1, r1i1p1f1
experiment: amip, cmorph, historical, hqys, oc5, rcp85
format: nc
fs_type: posix
grid_id: tco2559
grid_label: gn
institute: clmcom, cpc, csiro-arccss, ecmwf-awi, gerics, noaa
level_type: 2d
model: access-cm2, cpc, ifs-fesom, mpi-m-mpi-esm-lr-clmcom-cclm4-8-17-v1, ncc-noresm1-m-gerics-remo2015-v1, nodc
product: cmip, eur-11, grid, nextgems_cycle2, reanalysis
project: cmip6, cordex, nextgems, observations
rcm_name: clmcom-cclm4-8-17, gerics-remo2015
rcm_version: v1
realm: atm, atmos, ocean
time_aggregation: mean
time_frequency: 1day, 1hr, 1min, 3hr, mon
user: 
variable: 100si, 100u, 100v, 10u, 10v, 2d, 2t, asn, blh, cape, cdir, chnk, ci, cin, cp, dsrp, e, es, ewss, fal, fdir, fsr, hc700, hcc, lcc, litota1, litoti, lsm, lsp, lwcs, mcc, msl, nsss, par, pev, pr, ro, rsn, sd, sf, skt, slhf, smlt, sp, sro, sshf, ssr, ssrc, ssrd, ssro, sst, stl1, stl2, stl3, stl4, str, strc, strd, swvl1, swvl2, swvl3, swvl4, tas, tcc, tciw, tclw, tcrw, tcslw, tcsw, tcw, tcwv, tisr, tp, tsn, tsr, tsrc, ttr, ttrc, ua, uvb, vike, vipie, vipile, vithe, z

Parsing the command output#

You might have already noticed that each command has a --json flag. Enabling this flag lets you parse the output of each command to JSON.

Following on from the example above we can parse the output of the reverse search to the command line json processor jq:

freva-client databrowser metadata-search -e file="/arch/*" --json

Results

{"cmor_table": ["1day", "30min", "3hr", "amon", "inst-or-acc", "omon"], "dataset": ["cmip6-hsm", "cordex-hsm", "nextgems", "obs-hsm", "reana-hsm"], "driving_model": ["mpi-m-mpi-esm-lr", "ncc-noresm1-m"], "ensemble": ["r1i1p1", "r1i1p1f1"], "experiment": ["amip", "cmorph", "historical", "hqys", "oc5", "rcp85"], "format": ["nc"], "fs_type": ["posix"], "grid_id": ["tco2559"], "grid_label": ["gn"], "institute": ["clmcom", "cpc", "csiro-arccss", "ecmwf-awi", "gerics", "noaa"], "level_type": ["2d"], "model": ["access-cm2", "cpc", "ifs-fesom", "mpi-m-mpi-esm-lr-clmcom-cclm4-8-17-v1", "ncc-noresm1-m-gerics-remo2015-v1", "nodc"], "product": ["cmip", "eur-11", "grid", "nextgems_cycle2", "reanalysis"], "project": ["cmip6", "cordex", "nextgems", "observations"], "rcm_name": ["clmcom-cclm4-8-17", "gerics-remo2015"], "rcm_version": ["v1"], "realm": ["atm", "atmos", "ocean"], "time_aggregation": ["mean"], "time_frequency": ["1day", "1hr", "1min", "3hr", "mon"], "user": [], "variable": ["100si", "100u", "100v", "10u", "10v", "2d", "2t", "asn", "blh", "cape", "cdir", "chnk", "ci", "cin", "cp", "dsrp", "e", "es", "ewss", "fal", "fdir", "fsr", "hc700", "hcc", "lcc", "litota1", "litoti", "lsm", "lsp", "lwcs", "mcc", "msl", "nsss", "par", "pev", "pr", "ro", "rsn", "sd", "sf", "skt", "slhf", "smlt", "sp", "sro", "sshf", "ssr", "ssrc", "ssrd", "ssro", "sst", "stl1", "stl2", "stl3", "stl4", "str", "strc", "strd", "swvl1", "swvl2", "swvl3", "swvl4", "tas", "tcc", "tciw", "tclw", "tcrw", "tcslw", "tcsw", "tcw", "tcwv", "tisr", "tp", "tsn", "tsr", "tsrc", "ttr", "ttrc", "ua", "uvb", "vike", "vipie", "vipile", "vithe", "z"]}

By using the pipe operator | the JSON output of the freva-client commands can be piped and processed by jq:

freva-client databrowser metadata-search -e file="/arch/*" --json | jq -r .ensemble[0]

Results

r1i1p1

The above example will select only the first entry of the ensembles that are associated with files on the tape archive.