The databrowser command line interface#
This section introduces the usage of the freva-client databrowser sub command.
Please see the Installation and configuration section on how to install and
configure the command line interface.
After successful installation you can use the freva-client databrowser sub
command
freva-client databrowser --help
Results
Usage: freva-client databrowser [OPTIONS] COMMAND [ARGS]...
Data search related commands
Options:
--help Show this message and exit.
Commands:
data-count Count the databrowser search results
data-overview Get an overview over what is available in the...
data-search Search the databrowser for datasets.
intake-catalogue Create an intake catalogue from the search.
metadata-search Search databrowser for metadata (facets).
user-data Add or delete user data.
Searching for data locations#
Getting the locations of the data is probably the most common use case of the
databrowser application. You can search for data locations by applying the
data-search sub-command:
freva-client databrowser data-search --help
Results
Usage: freva-client databrowser data-search [OPTIONS] [SEARCH_KEYS]...
Search the databrowser for datasets.
Arguments:
[SEARCH_KEYS]... Refine your data search with this `key=value` pair search
parameters. The parameters could be, depending on the DRS
standard, flavour product, project model etc.
Options:
--facet TEXT If you are not sure about the correct search
key's you can use the ``--facet`` flag to
search of any matching entries. For example
--facet 'era5' would allow you to search for
any entries containing era5, regardless of
project, product etc.
-u, --uniq-key [file|uri] The type of search result, which can be
either “file” or “uri”. This parameter
determines whether the search will be based
on file paths or Uniform Resource
Identifiers [default: file]
-f, --flavour [freva|cmip6|cmip5|cordex|nextgems|user]
The Data Reference Syntax (DRS) standard
specifying the type of climate datasets to
query. [default: freva]
-ts, --time-select [strict|flexible|file]
Operator that specifies how the time period
is selected. Choose from flexible (default),
strict or file. ``strict`` returns only
those files that have the *entire* time
period covered. The time search ``2000 to
2012`` will not select files containing data
from 2010 to 2020 with the ``strict``
method. ``flexible`` will select those files
as ``flexible`` returns those files that
have either start or end period covered.
``file`` will only return files where the
entire time period is contained within *one
single* file. [default: flexible]
--zarr Create zarr stream files.
--access-token TEXT Use this access token for authentication
when creating a zarr stream files.
-t, --time TEXT Special search facet to refine/subset search
results by time. This can be a string
representation of a time range or a single
time step. The time steps have to follow
ISO-8601. Valid strings are
``%Y-%m-%dT%H:%M`` to ``%Y-%m-%dT%H:%M`` for
time ranges and ``%Y-%m-%dT%H:%M``.
**Note**: You don't have to give the full
string format to subset time steps ``%Y``,
``%Y-%m`` etc are also valid.
-j, --json Parse output in json format.
--host TEXT Set the hostname of the databrowser, if not
set (default) the hostname is read from a
config file
-v Increase verbosity [default: 0]
--multi-version Select all versions and not just the latest
version (default).
-V, --version Show version an exit
--help Show this message and exit.
The command expects a list of key=value pairs. The order of the pairs doesn’t really matter. Most important is that you don’t need to split the search according to the type of data you are searching for. You can search for files within observations, reanalysis and model data at the same time. Also important is that all queries are case insensitive. You can also search for attributes themselves instead of file paths. For example you can search for the list of variables available that satisfies a certain constraint (e.g. sampled 6hr, from a certain model, etc).
freva-client databrowser data-search project=observations variable=pr model=cp*
Results
https://swift.dkrz.de/v1/dkrz_a32dc0e8-2299-4239-a47d-6bf45c8b0160/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020000-201609022330.zarr
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022300-201609022330.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022200-201609022230.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022100-201609022130.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022000-201609022030.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021900-201609021930.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021800-201609021830.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021700-201609021730.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021600-201609021630.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021500-201609021530.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021400-201609021430.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021300-201609021330.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021200-201609021230.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021100-201609021130.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021000-201609021030.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020900-201609020930.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020800-201609020830.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020700-201609020730.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020600-201609020630.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020500-201609020530.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020400-201609020430.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020300-201609020330.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020200-201609020230.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020100-201609020130.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020000-201609020030.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022300-201609022330.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022200-201609022230.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022100-201609022130.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022000-201609022030.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021900-201609021930.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021800-201609021830.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021700-201609021730.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021600-201609021630.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021500-201609021530.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021400-201609021430.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021300-201609021330.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021200-201609021230.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021100-201609021130.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021000-201609021030.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020900-201609020930.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020800-201609020830.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020700-201609020730.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020600-201609020630.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020500-201609020530.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020400-201609020430.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020300-201609020330.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020200-201609020230.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020100-201609020130.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020000-201609020030.nc
There are many more options for defining a value for a given key:
Attribute syntax |
Meaning |
|---|---|
|
Search for files containing exactly that attribute |
|
Search for files containing a value for attribute that starts with the prefix val |
|
Search for files containing a value for attribute that ends with the suffix lue |
|
Search for files containing a value for attribute that has alu somewhere |
|
Search for files containing a value for attribute that matches the given regular expression (yes! you might use any regular expression to find what you want.) |
OR:
|
Search for files containing either value1 OR value2 for the given attribute (note that’s the same attribute twice!) |
|
Search for files containing value1 for attribute1 AND value2 for attribute2 |
|
Search for files NOT containing value |
|
Search for files containing neither value1 nor value2 |
Note
When using * remember that your shell might give it a different meaning (normally it will try to match files with that name) to turn that off you can use backslash (key=*) or use quotes (key=’*’).
Streaming files via zarr#
Instead of getting the file locations on disk or tape, you can instruct the
system to register zarr streams. Which means that instead of opening the
data directly you can open it via zarr from anywhere. To do so simply add
the --zarr flag.
Note
Before you can use the --zarr flag you will have
to create an access token and use that token to log on to the system
see also the Authentication chapter for more details on token creation.
token=$(freva-client auth -u janedoe|jq -r .access_token)
freva-client databrowser data-search dataset=cmip6-fs --zarr --access-token $token
Results
Give password for server authentication: *****
http://localhost:7777/api/freva-data-portal/zarr/c48f9024-f61e-5682-9651-7b9a21d81048.zarr
http://localhost:7777/api/freva-data-portal/zarr/ff8b9acf-b652-5ce3-a26c-58f9bc4884a3.zarr
Special cases: Searching for times#
For example you want to know how many files we have between a certain time range To do so you can use the time search key to subset time steps and whole time ranges:
freva-client databrowser data-search project=observations -t '2016-09-02T22:15 to 2016-10'
Results
https://swift.dkrz.de/v1/dkrz_a32dc0e8-2299-4239-a47d-6bf45c8b0160/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020000-201609022330.zarr
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022300-201609022330.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022200-201609022230.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/model/regional/cordex/output/EUR-11/CLMcom/MPI-M-MPI-ESM-LR/historical/r0i0p0/CLMcom-CCLM4-8-17/v1/fx/orog/v20140515/orog_EUR-11_MPI-M-MPI-ESM-LR_historical_r1i1p1_CLMcom-CCLM4-8-17_v1_fx.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022300-201609022330.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022200-201609022230.nc
The default method for selecting time periods is flexible, which means
all files are selected that cover at least start or end date. The
strict method implies that the entire search time period has to be
covered by the files. Using the strict method in the example above would
only yield on file because the first file contains time steps prior to the
start of the time period:
freva-client databrowser data-search project=observations -t '2016-09-02T22:15 to 2016-10' -ts strict
Results
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022300-201609022330.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022300-201609022330.nc
Giving single time steps is also possible:
freva-client databrowser data-search project=observations -t 2016-09-02T22:10
Results
https://swift.dkrz.de/v1/dkrz_a32dc0e8-2299-4239-a47d-6bf45c8b0160/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020000-201609022330.zarr
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/random/test.json
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022300-201609022330.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022200-201609022230.nc
/home/runner/work/freva-nextgen/freva-nextgen/freva-rest/src/databrowser_api/mock/data/model/regional/cordex/output/EUR-11/CLMcom/MPI-M-MPI-ESM-LR/historical/r0i0p0/CLMcom-CCLM4-8-17/v1/fx/orog/v20140515/orog_EUR-11_MPI-M-MPI-ESM-LR_historical_r1i1p1_CLMcom-CCLM4-8-17_v1_fx.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022300-201609022330.nc
/arch/bb1203/freva_test/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022200-201609022230.nc
/arch/bb1203/freva_test/model/global/cmip6/CMIP6/CMIP/MPI-M/MPI-ESM1-2-LR/amip/r2i1p1f1/Amon/ua/gn/v20190815/ua_mon_MPI-ESM1-2-LR_amip_r2i1p1f1_197901-199812.nc
Note
The time format has to follow the
ISO-8601 standard. Time ranges
are indicated by the to keyword such as 2000 to 2100 or
2000-01 to 2100-12 and alike. Single time steps are given without the
to keyword.
Creating intake-esm catalouges#
The intake-catalogue sub command allows you to create an
intake-esm catalogue <https://intake-esm.readthedocs.io/en/stable/>_ from
the current search. This can be useful to share the catalogue with others
or merge datasets.
freva-client databrowser intake-catalogue --help
Results
Usage: freva-client databrowser intake-catalogue [OPTIONS] [SEARCH_KEYS]...
Create an intake catalogue from the search.
Arguments:
[SEARCH_KEYS]... Refine your data search with this `key=value` pair search
parameters. The parameters could be, depending on the DRS
standard, flavour product, project model etc.
Options:
--facet TEXT If you are not sure about the correct search
key's you can use the ``--facet`` flag to
search of any matching entries. For example
--facet 'era5' would allow you to search for
any entries containing era5, regardless of
project, product etc.
-u, --uniq-key [file|uri] The type of search result, which can be
either “file” or “uri”. This parameter
determines whether the search will be based
on file paths or Uniform Resource
Identifiers [default: file]
-f, --flavour [freva|cmip6|cmip5|cordex|nextgems|user]
The Data Reference Syntax (DRS) standard
specifying the type of climate datasets to
query. [default: freva]
-ts, --time-select [strict|flexible|file]
Operator that specifies how the time period
is selected. Choose from flexible (default),
strict or file. ``strict`` returns only
those files that have the *entire* time
period covered. The time search ``2000 to
2012`` will not select files containing data
from 2010 to 2020 with the ``strict``
method. ``flexible`` will select those files
as ``flexible`` returns those files that
have either start or end period covered.
``file`` will only return files where the
entire time period is contained within *one
single* file. [default: flexible]
-t, --time TEXT Special search facet to refine/subset search
results by time. This can be a string
representation of a time range or a single
time step. The time steps have to follow
ISO-8601. Valid strings are
``%Y-%m-%dT%H:%M`` to ``%Y-%m-%dT%H:%M`` for
time ranges and ``%Y-%m-%dT%H:%M``.
**Note**: You don't have to give the full
string format to subset time steps ``%Y``,
``%Y-%m`` etc are also valid.
--zarr Create zarr stream files, as catalogue
targets.
--access-token TEXT Use this access token for authentication
when creating a zarr based intake catalogue.
-f, --filename PATH Path to the file where the catalogue, should
be written to. if None given (default) the
catalogue is parsed to stdout.
--host TEXT Set the hostname of the databrowser, if not
set (default) the hostname is read from a
config file
-v Increase verbosity [default: 0]
--multi-version Select all versions and not just the latest
version (default).
-V, --version Show version an exit
--help Show this message and exit.
You can either set the --filename flag to save the catalogue to a .json
file or pipe the catalogue to stdout (default). Just like for the data-search
sub command you can instruct the system to create zarr file streams to access
the data via zarr.
Query the number of occurrences#
In some cases it might be useful to know how many files are found in the
databrowser for certain search constraints. In such cases you can use the
data-count sub command to count the number of found files instead of getting
the files themselves.
freva-client databrowser data-count --help
Results
Usage: freva-client databrowser data-count [OPTIONS] [SEARCH_KEYS]...
Count the databrowser search results
Arguments:
[SEARCH_KEYS]... Refine your data search with this `key=value` pair search
parameters. The parameters could be, depending on the DRS
standard, flavour product, project model etc.
Options:
--facet TEXT If you are not sure about the correct search
key's you can use the ``--facet`` flag to
search of any matching entries. For example
--facet 'era5' would allow you to search for
any entries containing era5, regardless of
project, product etc.
-d, --detail Separate the count by search facets.
-f, --flavour [freva|cmip6|cmip5|cordex|nextgems|user]
The Data Reference Syntax (DRS) standard
specifying the type of climate datasets to
query. [default: freva]
-ts, --time-select [strict|flexible|file]
Operator that specifies how the time period
is selected. Choose from flexible (default),
strict or file. ``strict`` returns only
those files that have the *entire* time
period covered. The time search ``2000 to
2012`` will not select files containing data
from 2010 to 2020 with the ``strict``
method. ``flexible`` will select those files
as ``flexible`` returns those files that
have either start or end period covered.
``file`` will only return files where the
entire time period is contained within *one
single* file. [default: flexible]
-t, --time TEXT Special search facet to refine/subset search
results by time. This can be a string
representation of a time range or a single
time step. The time steps have to follow
ISO-8601. Valid strings are
``%Y-%m-%dT%H:%M`` to ``%Y-%m-%dT%H:%M`` for
time ranges and ``%Y-%m-%dT%H:%M``.
**Note**: You don't have to give the full
string format to subset time steps ``%Y``,
``%Y-%m`` etc are also valid.
-e, --extended-search Retrieve information on additional search
keys.
--multi-version Select all versions and not just the latest
version (default).
--host TEXT Set the hostname of the databrowser, if not
set (default) the hostname is read from a
config file
-j, --json Parse output in json format.
-v Increase verbosity [default: 0]
-V, --version Show version an exit
--help Show this message and exit.
By default the data-count sub command will display the total number of items
matching your search query. For example:
freva-client databrowser data-count project=observations
Results
52
If you want to group the number of occurrences by search categories (facets)
use the -d or --detail flag:
freva-client databrowser data-count -d project=observations
Results
ensemble: r1i1p1[52]
experiment: cmorph[49], oc5[3]
institute: cpc[49], noaa[3]
model: cpc[25], cpc-cmorph[24], nodc[3]
product: grid[49], reanalysis[3]
project: observations[52]
realm: atmos[49], ocean[3]
time_aggregation: mean[52]
time_frequency: 1min[49], mon[3]
variable: hc700[3], pr[49]
Retrieving the available metadata#
Sometime it might be useful to know about possible values that search attributes.
For this you use the metadata-search sub command:
freva-client databrowser metadata-search --help
Results
Usage: freva-client databrowser metadata-search [OPTIONS] [SEARCH_KEYS]...
Search databrowser for metadata (facets).
Arguments:
[SEARCH_KEYS]... Refine your data search with this `key=value` pair search
parameters. The parameters could be, depending on the DRS
standard, flavour product, project model etc.
Options:
--facet TEXT If you are not sure about the correct search
key's you can use the ``--facet`` flag to
search of any matching entries. For example
--facet 'era5' would allow you to search for
any entries containing era5, regardless of
project, product etc.
-f, --flavour [freva|cmip6|cmip5|cordex|nextgems|user]
The Data Reference Syntax (DRS) standard
specifying the type of climate datasets to
query. [default: freva]
-ts, --time-select [strict|flexible|file]
Operator that specifies how the time period
is selected. Choose from flexible (default),
strict or file. ``strict`` returns only
those files that have the *entire* time
period covered. The time search ``2000 to
2012`` will not select files containing data
from 2010 to 2020 with the ``strict``
method. ``flexible`` will select those files
as ``flexible`` returns those files that
have either start or end period covered.
``file`` will only return files where the
entire time period is contained within *one
single* file. [default: flexible]
-t, --time TEXT Special search facet to refine/subset search
results by time. This can be a string
representation of a time range or a single
time step. The time steps have to follow
ISO-8601. Valid strings are
``%Y-%m-%dT%H:%M`` to ``%Y-%m-%dT%H:%M`` for
time ranges and ``%Y-%m-%dT%H:%M``.
**Note**: You don't have to give the full
string format to subset time steps ``%Y``,
``%Y-%m`` etc are also valid.
-e, --extended-search Retrieve information on additional search
keys.
--multi-version Select all versions and not just the latest
version (default).
--host TEXT Set the hostname of the databrowser, if not
set (default) the hostname is read from a
config file
-j, --json Parse output in json format.
-v Increase verbosity [default: 0]
-V, --version Show version an exit
--help Show this message and exit.
Just like with any other databrowser command you can apply different search constraints when acquiring metadata
freva-client databrowser metadata-search project=observations
Results
ensemble: r1i1p1
experiment: cmorph, oc5
institute: cpc, noaa
model: cpc, cpc-cmorph, nodc
product: grid, reanalysis
project: observations
realm: atmos, ocean
time_aggregation: mean
time_frequency: 1min, mon
variable: hc700, pr
By default the command will display only the most commonly used metadata
keys. To retrieve all metadata you can use the -e or --extended-search
flag.
freva-client databrowser metadata-search -e project=observations
Results
cmor_table: 30min, omon
dataset: obs-fs, obs-hsm, obs-swfit, reana-fs, reana-hsm, reana-swfit
driving_model:
ensemble: r1i1p1
experiment: cmorph, oc5
format: nc, zarr
fs_type: posix
grid_id:
grid_label: gn
institute: cpc, noaa
level_type: 2d
model: cpc, cpc-cmorph, nodc
product: grid, reanalysis
project: observations
rcm_name:
rcm_version:
realm: atmos, ocean
time_aggregation: mean
time_frequency: 1min, mon
user:
variable: hc700, pr
Sometimes you don’t exactly know the exact names of the search keys and
want retrieve all file objects that match a certain category. For example
for getting all ocean reanalysis datasets you can apply the --facet flag:
freva-client databrowser metadata-search -e realm=ocean --facet 'rean*'
Results
ensemble: r1i1p1
experiment: oc5
institute: noaa
model: nodc
product: reanalysis
project: observations
realm: ocean
time_aggregation: mean
time_frequency: mon
variable: hc700
Expert tip: Getting metadata for certain files#
In some cases it might be useful to retrieve metadata for certain file or object store locations. For example if we wanted to retrieve the metadata of those files on tape:
freva-client databrowser metadata-search -e file="/arch/*"
Results
cmor_table: 1day, 30min, 3hr, amon, inst-or-acc, omon
dataset: cmip6-hsm, cordex-hsm, nextgems, obs-hsm, reana-hsm
driving_model: mpi-m-mpi-esm-lr, ncc-noresm1-m
ensemble: r1i1p1, r1i1p1f1
experiment: amip, cmorph, historical, hqys, oc5, rcp85
format: nc
fs_type: posix
grid_id: tco2559
grid_label: gn
institute: clmcom, cpc, csiro-arccss, ecmwf-awi, gerics, noaa
level_type: 2d
model: access-cm2, cpc, ifs-fesom, mpi-m-mpi-esm-lr-clmcom-cclm4-8-17-v1, ncc-noresm1-m-gerics-remo2015-v1, nodc
product: cmip, eur-11, grid, nextgems_cycle2, reanalysis
project: cmip6, cordex, nextgems, observations
rcm_name: clmcom-cclm4-8-17, gerics-remo2015
rcm_version: v1
realm: atm, atmos, ocean
time_aggregation: mean
time_frequency: 1day, 1hr, 1min, 3hr, mon
user:
variable: 100si, 100u, 100v, 10u, 10v, 2d, 2t, asn, blh, cape, cdir, chnk, ci, cin, cp, dsrp, e, es, ewss, fal, fdir, fsr, hc700, hcc, lcc, litota1, litoti, lsm, lsp, lwcs, mcc, msl, nsss, par, pev, pr, ro, rsn, sd, sf, skt, slhf, smlt, sp, sro, sshf, ssr, ssrc, ssrd, ssro, sst, stl1, stl2, stl3, stl4, str, strc, strd, swvl1, swvl2, swvl3, swvl4, tas, tcc, tciw, tclw, tcrw, tcslw, tcsw, tcw, tcwv, tisr, tp, tsn, tsr, tsrc, ttr, ttrc, ua, uvb, vike, vipie, vipile, vithe, z
Parsing the command output#
You might have already noticed that each command has a --json flag.
Enabling this flag lets you parse the output of each command to JSON.
Following on from the example above we can parse the output of the reverse search to the command line json processor jq:
freva-client databrowser metadata-search -e file="/arch/*" --json
Results
{"cmor_table": ["1day", "30min", "3hr", "amon", "inst-or-acc", "omon"], "dataset": ["cmip6-hsm", "cordex-hsm", "nextgems", "obs-hsm", "reana-hsm"], "driving_model": ["mpi-m-mpi-esm-lr", "ncc-noresm1-m"], "ensemble": ["r1i1p1", "r1i1p1f1"], "experiment": ["amip", "cmorph", "historical", "hqys", "oc5", "rcp85"], "format": ["nc"], "fs_type": ["posix"], "grid_id": ["tco2559"], "grid_label": ["gn"], "institute": ["clmcom", "cpc", "csiro-arccss", "ecmwf-awi", "gerics", "noaa"], "level_type": ["2d"], "model": ["access-cm2", "cpc", "ifs-fesom", "mpi-m-mpi-esm-lr-clmcom-cclm4-8-17-v1", "ncc-noresm1-m-gerics-remo2015-v1", "nodc"], "product": ["cmip", "eur-11", "grid", "nextgems_cycle2", "reanalysis"], "project": ["cmip6", "cordex", "nextgems", "observations"], "rcm_name": ["clmcom-cclm4-8-17", "gerics-remo2015"], "rcm_version": ["v1"], "realm": ["atm", "atmos", "ocean"], "time_aggregation": ["mean"], "time_frequency": ["1day", "1hr", "1min", "3hr", "mon"], "user": [], "variable": ["100si", "100u", "100v", "10u", "10v", "2d", "2t", "asn", "blh", "cape", "cdir", "chnk", "ci", "cin", "cp", "dsrp", "e", "es", "ewss", "fal", "fdir", "fsr", "hc700", "hcc", "lcc", "litota1", "litoti", "lsm", "lsp", "lwcs", "mcc", "msl", "nsss", "par", "pev", "pr", "ro", "rsn", "sd", "sf", "skt", "slhf", "smlt", "sp", "sro", "sshf", "ssr", "ssrc", "ssrd", "ssro", "sst", "stl1", "stl2", "stl3", "stl4", "str", "strc", "strd", "swvl1", "swvl2", "swvl3", "swvl4", "tas", "tcc", "tciw", "tclw", "tcrw", "tcslw", "tcsw", "tcw", "tcwv", "tisr", "tp", "tsn", "tsr", "tsrc", "ttr", "ttrc", "ua", "uvb", "vike", "vipie", "vipile", "vithe", "z"]}
By using the pipe operator | the JSON output of the freva-client
commands can be piped and processed by jq:
freva-client databrowser metadata-search -e file="/arch/*" --json | jq -r .ensemble[0]
Results
r1i1p1
The above example will select only the first entry of the ensembles that are associated with files on the tape archive.
Adding and Deleting User Data#
You can manage your personal datasets within the databrowser by adding or deleting user-specific data. This functionality allows you to include your own data files into the databrowser, making them accessible for analysis and retrieval alongside other datasets.
Before using the user-data commands, you need to create an access token and authenticate with the system. Please refer to the Authentication chapter for more details on token creation and authentication.
Adding User Data#
To add your data to the databrowser, use the user-data add command. You’ll need to provide the paths to your data files, and any metadata you’d like to associate with your data.
token=$(freva-client auth -u janedoe | jq -r .access_token)
freva-client databrowser user-data add \
--path freva-rest/src/databrowser_api/mock/ \
--facet project=cordex \
--facet experiment=rcp85 \
--facet model=mpi-m-mpi-esm-lr-clmcom-cclm4-8-17-v1 \
--facet variable=tas \
--access-token $token
Results
STDOUT: [2024-11-02T21:44:17] ERROR freva-client: Failed to open data file
/home/runner/work/freva-nextgen/freva-nextgen/doc
s/../freva-rest/src/databrowser_api/mock/data/mod
el/obs/reanalysis/reanalysis/NOAA/NODC/OC5/mon/oc
ean/Omon/r1i1p1/v20200101/hc700/hc700_mon_NODC_OC
5_r1i1p1_201201-201212.nc: Failed to decode
variable 'time': unable to decode time units
'months since 1955-01-01 00:00:00' with "calendar
'standard'". Try opening your dataset with
decode_times=False or installing cftime if it is
not installed.
WARNING freva-client: Error getting metadata: {}
This command adds the specified data files to the databrowser and tags them with the provided metadata. These search filters help in indexing and searching your data within the system.
Deleting User Data#
If you need to remove your data from the databrowser, use the user-data delete command. Provide your the search keys (facets) that identify the user data you wish to delete.
token=$(freva-client auth -u janedoe | jq -r .access_token)
freva-client databrowser user-data delete \
--search-key project=cordex \
--search-key experiment=rcp85 \
--access-token $token
Results
This command deletes all data entries that match the specified search keys from the databrowser.