The Freva command line interface#
This notebook introduces the usage of the Freva command line interface - cli. The tutorial assumes that you have already access to Freva either because you’ve setup an instance yourself or one has been setup by the Freva admin team. Hence it is assumed that you know how to access Freva. If this is not the case please contact one of your Freva admins for help.
A general usage overview of the available Freva sub-commands is available via the –help option:
[1]:
freva --help
usage: freva [-h] [-V] {databrowser,esgf,plugin,user-data,history} ...
Free EVAluation system framework (freva)
positional arguments:
{databrowser,esgf,plugin,user-data,history}
Available sub-commands:
databrowser Find data in the system.
esgf Search/Download ESGF the data catalogue.
plugin Apply data analysis plugin.
user-data Update users project data.
history Read the plugin application history.
options:
-h, --help show this help message and exit
-V, --version show program's version number and exit
To get help for the individual sub-commands use: freva <sub-command> --help
The most common sub-commands are databrowser and plugin. You can get more help on the actual commands using the sub-commands --help option, for example getting help on the databrowser usage:
[2]:
freva databrowser --help
usage: freva databrowser [-h] [--multiversion] [--batch-size BATCH_SIZE]
[--count] [--facet FACET] [--facet-limit FACET_LIMIT]
[--time-select {flexible,strict,file}] [--debug]
[facets ...]
Find data in the system.
positional arguments:
facets Search facet(s). (default: None)
options:
-h, --help show this help message and exit
--multiversion Select not only the latest version. (default: False)
--batch-size BATCH_SIZE
Number of files to retrieve. (default: 5000)
--count Show the number of files for each search result.
(default: False)
--facet FACET Retrieve values of given facet instead of files.
(default: None)
--facet-limit FACET_LIMIT
Limit the number of output facets. (default:
9223372036854775807)
--time-select {flexible,strict,file}
Operator that specifies how the time period is
selected. (default: flexible)
--debug, -v, -d, --verbose
Use verbose output. (default: False)
Note: Instead of using sub-commands you have also the option to use commands. For example the command freva-databrowser is equivalent to freva databrowser, freva-user-data to freva user-data etc.
Searching for data: the freva-databrowser command#
All files available on in the project are scanned and indexed via a data search server. This allows you to query the server which responds almost immediately. To search for data you can either use the freva-databrowser command or the freva databrowser sub-command. Let’s inspect the help menu of the databrowser sub-command:
[3]:
freva-databrowser --help
usage: freva-databrowser [-h] [--multiversion] [--batch-size BATCH_SIZE]
[--count] [--facet FACET] [--facet-limit FACET_LIMIT]
[--time-select {flexible,strict,file}] [--debug] [-V]
[facets ...]
Find data in the system.
positional arguments:
facets Search facet(s). (default: None)
options:
-h, --help show this help message and exit
--multiversion Select not only the latest version. (default: False)
--batch-size BATCH_SIZE
Number of files to retrieve. (default: 5000)
--count Show the number of files for each search result.
(default: False)
--facet FACET Retrieve values of given facet instead of files.
(default: None)
--facet-limit FACET_LIMIT
Limit the number of output facets. (default:
9223372036854775807)
--time-select {flexible,strict,file}
Operator that specifies how the time period is
selected. (default: flexible)
--debug, -v, -d, --verbose
Use verbose output. (default: False)
-V, --version show program's version number and exit
The databrowser expects a list of key=value pairs. The order of the pairs doesn’t really matter. Most important is that you don’t need to split the search according to the type of data you are searching for. You can search for any files, for example observations, reanalysis and model data at the same time. Also important is that all queries are case insensitive. You can also search for attributes themselves instead of file paths. For example you can search for the list of variables available
that satisfies a certain constraint (e.g. sampled 6hr, from a certain model, etc).
[4]:
freva-databrowser project=observations variable=pr model='cp*'
/home/runner/work/freva/freva/.docker/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022300-201609022330.nc
/home/runner/work/freva/freva/.docker/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022200-201609022230.nc
/home/runner/work/freva/freva/.docker/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022100-201609022130.nc
/home/runner/work/freva/freva/.docker/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022000-201609022030.nc
/home/runner/work/freva/freva/.docker/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021900-201609021930.nc
/home/runner/work/freva/freva/.docker/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021800-201609021830.nc
/home/runner/work/freva/freva/.docker/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021700-201609021730.nc
/home/runner/work/freva/freva/.docker/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021600-201609021630.nc
/home/runner/work/freva/freva/.docker/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021500-201609021530.nc
/home/runner/work/freva/freva/.docker/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021400-201609021430.nc
/home/runner/work/freva/freva/.docker/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021300-201609021330.nc
/home/runner/work/freva/freva/.docker/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021200-201609021230.nc
/home/runner/work/freva/freva/.docker/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021100-201609021130.nc
/home/runner/work/freva/freva/.docker/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609021000-201609021030.nc
/home/runner/work/freva/freva/.docker/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020900-201609020930.nc
/home/runner/work/freva/freva/.docker/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020800-201609020830.nc
/home/runner/work/freva/freva/.docker/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020700-201609020730.nc
/home/runner/work/freva/freva/.docker/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020600-201609020630.nc
/home/runner/work/freva/freva/.docker/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020500-201609020530.nc
/home/runner/work/freva/freva/.docker/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020400-201609020430.nc
/home/runner/work/freva/freva/.docker/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020300-201609020330.nc
/home/runner/work/freva/freva/.docker/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020200-201609020230.nc
/home/runner/work/freva/freva/.docker/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020100-201609020130.nc
/home/runner/work/freva/freva/.docker/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609020000-201609020030.nc
In some cases it might be useful to know how many files are found in the databrowser for certain search constraints. In such cases you can use the count flag to count the number of found files instead of getting the files themselves.
[5]:
freva-databrowser project=observations --count
24
Sometimes it might be useful to subset the data you’re interested in by time. To do so you can use the time search key to subset time steps and whole time ranges. For example let’s get the for certain time range:
[6]:
freva-databrowser project=observations time='2016-09-02T22:15 to 2016-10'
/home/runner/work/freva/freva/.docker/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022300-201609022330.nc
/home/runner/work/freva/freva/.docker/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022200-201609022230.nc
The default method for selecting time periods is flexible, which means all files are selected that cover at least start or end date. The strict method implies that the entire search time period has to be covered by the files. Using the strict method in the example above would only yield on file because the first file contains time steps prior to the start of the time period:
[7]:
freva-databrowser project=observations time='2016-09-02T22:15 to 2016-10' --time-select strict
/home/runner/work/freva/freva/.docker/data/observations/grid/CPC/CPC/cmorph/30min/atmos/30min/r1i1p1/v20210618/pr/pr_30min_CPC_cmorph_r1i1p1_201609022300-201609022330.nc
You might as well want to know about possible values that an attribute can take after a certain search is done. For this you use the --facet flag (facets are the attributes used to search for and sub set the data). For example to see all facets that are available in the observations project:
[8]:
freva-databrowser project=observations --all-facets
usage: freva-databrowser [-h] [--multiversion] [--batch-size BATCH_SIZE]
[--count] [--facet FACET] [--facet-limit FACET_LIMIT]
[--time-select {flexible,strict,file}] [--debug] [-V]
[facets ...]
freva-databrowser: error: Unknown option: --all-facets
[9]:
freva-databrowser --facet time_frequency --facet variable project=observations
time_frequency: 30min
variable: pr
You can also retrieve information on how many facets are found by the databrowser by giving the count flag
[10]:
freva-databrowser --facet time_frequency --facet variable project=observations --count
time_frequency: 30min (24)
variable: pr (24)
Running data analysis plugins: the freva-plugin command#
Already defined data analysis tools can be started with the freva-plugin command or the freva plugin sub-command. Let’s inspect the help menu of the plugin command:
[11]:
freva-plugin --help
usage: freva-plugin [-h] [--repo-version] [--caption CAPTION] [--save]
[--save-config SAVE_CONFIG] [--show-config] [--batchmode]
[--unique-output {true,false,True,False}] [--debug]
[--list-tools] [--doc] [--json] [--wait] [-V]
[plugin_name] [tool-options ...]
Apply data analysis plugin.
positional arguments:
plugin_name Plugin name. (default: None)
tool-options Tool options (default: None)
options:
-h, --help show this help message and exit
--repo-version Show the version number from the repository. (default:
False)
--caption CAPTION Set a caption for the results (default: )
--save Save the plugin configuration to default destination.
(default: False)
--save-config SAVE_CONFIG
Save the plugin configuration. (default: None)
--show-config Show the resulting configuration. (default: False)
--batchmode Create a Batch job and submit it to the scheduling
system. (default: False)
--unique-output {true,false,True,False}, --unique_output {true,false,True,False}
Append a Freva run id to the output/cache folder(s).
(default: true)
--debug, -v, -d, --verbose
Use verbose output. (default: False)
--list-tools, --list, -l
Only list the available tools. (default: False)
--doc, --plugin-doc Display plugin documentation (default: False)
--json, -j Display a json representation of the result, this can
beuseful if you want to build shell based pipelines
and wantparse the output with help of `jq`. (default:
False)
--wait, -w Wait for the plugin to finish, this has only an effect
for batch mode execution. (default: False)
-V, --version show program's version number and exit
As the help menu suggests you can list all available tools using the -l option (or --list, --list-tools):
[12]:
freva-plugin -l
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Tool ┃ Description ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Animator │ Animate data on lon/lat grids │
│ DummyPlugin │ A dummy plugin │
│ DummyPluginFolders │ A dummy plugin with outputdir folder │
└────────────────────┴──────────────────────────────────────┘
This means currently we have two plugins available (animator and dummyplugin). The general syntax is freva-plugin <plugin-name> [options] for example to inspect the documentation of a certain plugin you can use the --doc option. Here we concentrate on the Animator plugin. A simple plugin that creates animations of geospatial data. The basic usage of that command can be retrieved by:
[13]:
freva-plugin --doc animator
Animator (v2022.7.15): Create animations (in gif or mp4 format) This tool
creates plots of solr facets and an animation.
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Option ┃ Description ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ input_file │ NetCDF input file(s), you can choose multiple │
│ │ files separated by a , or use a global pattern for │
│ │ multiple files chose this option only if you don't │
│ │ want Freva to find files by seach facets (default: │
│ │ <null>) │
│ variable │ Variable name (only applicable if you didn't │
│ │ choose an input file) (default: <null>) │
│ project │ Project name (only applicable if you didn't choose │
│ │ an input file) (default: <null>) │
│ product │ Product name (only applicable if you didn't choose │
│ │ an input file) (default: <null>) │
│ experiment │ Experiment name (only applicable if you didn't │
│ │ choose an input file) (default: <null>) │
│ institute │ Institute name (only applicable if you didn't │
│ │ choose an input file) (default: <null>) │
│ model │ Model name (only applicable if you didn't choose │
│ │ an input file) (default: <null>) │
│ time_frequency │ Time frequency name (only applicable if you didn't │
│ │ choose an input file) (default: <null>) │
│ ensemble │ Ensemble name (only applicable if you didn't │
│ │ choose an input file) (default: <null>) │
│ start │ Define the first time step to be plotted, leave │
│ │ blank if taken from data (default: <null>) │
│ end │ Define the last time step to be plotted, leave │
│ │ blank if taken from data (default: <null>) │
│ time_mean │ Select a time interval if time │
│ │ averaging/min/max/sum should be applied along the │
│ │ time axis. This can be D for daily, M for monthly │
│ │ 6H for 6 hours etc. Leave blank if the time axis │
│ │ should not be resampled (default) or set to │
│ │ <em>all</em> if you want to collapse the time │
│ │ axis. (default: <null>) │
│ time_method │ If resampling of the time axis is chosen (default: │
│ │ no) set the method: mean, max or min. <b>Note:</b> │
│ │ This has only an effect if the above parameter for │
│ │ <em>time_mean</em> is set. (default: mean) │
│ lonlatbox │ Set the extend of a rectangular lonlatbox │
│ │ (left_lon, right_lon, lower_lat, upper_lat) │
│ │ (default: <null>) │
│ output_unit │ Set the output unit of the variable - leave blank │
│ │ for no conversion. This can be useful if the unit │
│ │ of the input files should be converted, for │
│ │ example for precipitation. Note: Although many │
│ │ conversions are supported, by using the `pint` │
│ │ conversion library. (default: <null>) │
│ vmin │ Set the minimum plotting range (leave blank to │
│ │ calculate from data 1st decile) (default: <null>) │
│ vmax │ Set the maximum plotting range (leave blank to │
│ │ calculate the 9th decile) (default: <null>) │
│ cmap │ Set the colormap, more information on colormaps is │
│ │ available on the <a │
│ │ href="https://matplotlib.org/stable/tutorials/colo │
│ │ rs/colormaps.html" target=_blank>matplotlib │
│ │ website</a>. (default: RdYlBu_r) │
│ linecolor │ Color of the coast lines in the map (default: k) │
│ projection │ Set the global map projection. Note: this should │
│ │ the name of the cartopy projection method (e.g │
│ │ PlatteCarree for Cylindrical Projection). Pleas │
│ │ refer to <a │
│ │ href="https://scitools.org.uk/cartopy/docs/latest/ │
│ │ crs/projections.html"target=_blank>cartopy │
│ │ website</a> for details. (default: PlateCarree) │
│ proj_centre │ Set center longitude of the global map projection. │
│ │ (default: 50) │
│ pic_size │ Set the size of the picture (in pixel) (default: │
│ │ 1360,900) │
│ plot_title │ Set plot title (default: ) │
│ cbar_label │ Overwrite default colorbar label by this value │
│ │ (default: ) │
│ suffix │ Filetype of the animation (default: mp4) │
│ fps │ Set the frames per seceonds of the output │
│ │ animation. (default: 5) │
│ extra_scheduler_options │ Set additional options for the job submission to │
│ │ the workload manager (, separated). Note: │
│ │ batchmode and web only. (default: --qos=test, │
│ │ --array=20) │
└─────────────────────────┴────────────────────────────────────────────────────┘
The parameters are also given as key=values pairs. But not all of the above parameters are mandatory. Let’s use one project search key and animate its content.
[14]:
freva plugin animator project=observations variable=pr cmap=Blues fps=5 output_unit=mm/h vmin=0 vmax=5 suffix=gif
/tmp/animator/run.py:91: SyntaxWarning: invalid escape sequence '\d'
for i in re.findall("[-\d]+", unit[n]):
Setp 1: Collecting all files
Step 2: Opening the netcdf-files, collecting metadata
Step 3: Converting Units
Step 4: Loading the dataset, getting min/max values
Step 5: Creating the animation
/tmp/animator/run.py:91: SyntaxWarning: invalid escape sequence '\d'
for i in re.findall("[-\d]+", unit[n]):
Created animation /tmp/eval_conf/work/runner/freva-ces/output/animator/2129/pr_20160902T0000-20160902T2330.gif in 46 seconds
This plugin will run in so called interactive mode. That means that it will run on the login node and block your shell until the command is completed. This can be problematic if you have jobs that might take time to finish. An alternative is setting the -–batchmode flag. This flag tells the plugin to submit a job to the computing queue. The computing nodes are the core of any high performance computing system. Let’s submit the previous plugin job to the computing queue:
[15]:
freva plugin animator project=observations variable=pr cmap=Blues fps=5 output_unit=mm/h vmin=0 vmax=5 suffix=gif --batchmode
Scheduled job with history id: 2130
You can view the job's status with the command squeue
Your job's progress will be shown with the command
tail -f /tmp/share/slurm/animator/Animator-4876.out
Inspecting previous analysis jobs: the freva-history command#
Sometimes it can be useful to access the status of past plugin applications. The freva-history command or freva history sub-command can do that:
[16]:
freva-history --limit 1 --plugin dummyplugin
[05:11:45] INFO freva - INFO - history of {plugin}, limit=1, since=None,
until=None, entry_ids=None
2115) dummyplugin [2024-11-14 05:05:22] finished {"number": null, "the_number": 5, "something": "test", "other": 1.4...
[17]:
freva-history --entry-ids 1 --full-text
[05:11:46] INFO freva - INFO - history, limit=10, since=None, until=None,
entry_ids=['1']
To re-run a command of a past configuration you can use the –return-command option to get the command that was used:
[18]:
freva-history --entry-ids 136 --return-command
Managing your own datasets: the freva-user-data command#
Freva offers the possibility to share custom datasets with other users by making it searchable via freva-databrowser. With help of the freva-user-data command users can add their own data to the central data location, (re)-index or delete data in the databrowser.
Note: Any data that has been added by users will be assigned a special project name: project=user-$USER. Let’s inspect the help menu of the freva-user-data or freva user-data command:
[19]:
freva-user-data --help
usage: freva-user-data [-h] [-V] {index,add,delete} ...
Update users project data.
positional arguments:
{index,add,delete} Available sub-commands:
index Index existing user project data to the databrowser.
add Add new user project data to the databrowser.
delete Delete existing user project data from the databrowser.
options:
-h, --help show this help message and exit
-V, --version show program's version number and exit
Add new data to the databrowser#
To be able to add data to the databrowser the file names must follow a strict standard and the files must reside in a specific location. The add sub command takes care about the correct file naming and location. No pre requirements other than the file has to be a valid netCDF or grib file are assumed. In other words this sub command places the user data with the correct naming structure to the correct location.
[20]:
freva-user-data add --help
usage: freva-user-data add [-h] [--how {copy,move,symlink,link}] [--override]
[--experiment EXPERIMENT] [--institute INSTITUTE]
[--model MODEL] [--variable VARIABLE]
[--time-frequency TIME_FREQUENCY]
[--ensemble ENSEMBLE] [--cmor-table CMOR_TABLE]
[--realm REALM] [--debug]
product paths [paths ...]
Add new user project data to the databrowser.
positional arguments:
product Product search key the newly added data can be found.
paths Filename(s) or Directories that are going to be added
to thedatabrowser.
options:
-h, --help show this help message and exit
--how {copy,move,symlink,link}
Method of how the data is added into the central freva
user directory. (default: copy)
--override, --overwrite
Replace existing files in the user data structure.
(default: False)
--experiment EXPERIMENT
Set the <experiment> information if they can't be
found in the meta data. (default: None)
--institute INSTITUTE
Set the <institute> information if they can't be found
in the meta data. (default: None)
--model MODEL Set the <model> information if they can't be found in
the meta data. (default: None)
--variable VARIABLE Set the <variable> information if they can't be found
in the meta data. (default: None)
--time-frequency TIME_FREQUENCY, --time_frequency TIME_FREQUENCY
Set the <time_frequency> information if they can't be
found in the meta data. (default: None)
--ensemble ENSEMBLE Set the <ensemble> information if they can't be found
in the meta data. (default: None)
--cmor-table CMOR_TABLE, --cmor_table CMOR_TABLE
Set the <cmor-table> information if they can't be
found in the meta data. (default: None)
--realm REALM Set the <realm> information if they can't be found in
the metadata. (default: None)
--debug, -v, -d, --verbose
Use verbose output. (default: False)
Suppose you’ve gotten data from somewhere and want to add this data into the databrowser to make it accessible to others. In this specific example we assume that you have stored your original data in the /tmp/my_awesome_data folder. E.g /tmp/my_awesome_data/outfile_0.nc...tmp/my_awesome_data/outfile_9.nc The routine will try to gather all necessary metadata from the files. You’ll have to provide additional metadata if mandatory keywords are missing. To make the routine work in this
example we have to provide the institute, model and experiment keywords:
[21]:
freva-user-data add eur-11b /tmp/my_awesome_data/outfile_?.nc \
--institute clex --model UM-RA2T --experiment Bias-correct
freva-databrowser experiment=bias-correct
Status: crawling ...ok
/tmp/user_data/user-runner/eur-11b/clex/UM-RA2T/Bias-correct/hr/user_data/hr/r0i0p0/v20241114/tas/tas_hr_UM-RA2T_Bias-correct_r0i0p0_197001041800-197001050300.nc
/tmp/user_data/user-runner/eur-11b/clex/UM-RA2T/Bias-correct/hr/user_data/hr/r0i0p0/v20241114/tas/tas_hr_UM-RA2T_Bias-correct_r0i0p0_197001040800-197001041700.nc
/tmp/user_data/user-runner/eur-11b/clex/UM-RA2T/Bias-correct/hr/user_data/hr/r0i0p0/v20241114/tas/tas_hr_UM-RA2T_Bias-correct_r0i0p0_197001032200-197001040700.nc
/tmp/user_data/user-runner/eur-11b/clex/UM-RA2T/Bias-correct/hr/user_data/hr/r0i0p0/v20241114/tas/tas_hr_UM-RA2T_Bias-correct_r0i0p0_197001031200-197001032100.nc
/tmp/user_data/user-runner/eur-11b/clex/UM-RA2T/Bias-correct/hr/user_data/hr/r0i0p0/v20241114/tas/tas_hr_UM-RA2T_Bias-correct_r0i0p0_197001030200-197001031100.nc
/tmp/user_data/user-runner/eur-11b/clex/UM-RA2T/Bias-correct/hr/user_data/hr/r0i0p0/v20241114/tas/tas_hr_UM-RA2T_Bias-correct_r0i0p0_197001021600-197001030100.nc
/tmp/user_data/user-runner/eur-11b/clex/UM-RA2T/Bias-correct/hr/user_data/hr/r0i0p0/v20241114/tas/tas_hr_UM-RA2T_Bias-correct_r0i0p0_197001020600-197001021500.nc
/tmp/user_data/user-runner/eur-11b/clex/UM-RA2T/Bias-correct/hr/user_data/hr/r0i0p0/v20241114/tas/tas_hr_UM-RA2T_Bias-correct_r0i0p0_197001012000-197001020500.nc
/tmp/user_data/user-runner/eur-11b/clex/UM-RA2T/Bias-correct/hr/user_data/hr/r0i0p0/v20241114/tas/tas_hr_UM-RA2T_Bias-correct_r0i0p0_197001011000-197001011900.nc
/tmp/user_data/user-runner/eur-11b/clex/UM-RA2T/Bias-correct/hr/user_data/hr/r0i0p0/v20241114/tas/tas_hr_UM-RA2T_Bias-correct_r0i0p0_197001010000-197001010900.nc
Remove your data from the databrowser#
The delete sub command removes entries from the databrowser and if necessary existing files from the central user data location.
[22]:
freva-user-data delete --help
usage: freva-user-data delete [-h] [--delete-from-fs] [--debug]
paths [paths ...]
Delete existing user project data from the databrowser.
positional arguments:
paths The user directory(s) that needs to be crawled.
options:
-h, --help show this help message and exit
--delete-from-fs, --delete_from_fs, --delete
Do not only delete the files from the databrowser but
also from file system. (default: False)
--debug, -v, -d, --verbose
Use verbose output. (default: False)
Any data in the central user directory that belongs to the user can be deleted from the databrowser and also from the central data location:
[23]:
freva-user-data delete /tmp/user_data/user-$USER
freva-databrowser experiment=bias-correct
(Re)-Index existing data to the databrowser#
The index subcommand can be used to update the databrowser for existing user data. For example, if data has been removed from the databrowser it can be re-added:
[24]:
freva-user-data index --help
usage: freva-user-data index [-h] [--data-type {fs}] [--continue-on-errors]
[--debug]
[crawl_dir ...]
Index existing user project data to the databrowser.
positional arguments:
crawl_dir The user directory(s) that needs to be crawled.
(default: None)
options:
-h, --help show this help message and exit
--data-type {fs}, --dtype {fs}
The data type of the data. (default: fs)
--continue-on-errors, --continue, -c
Continue indexing on error. (default: False)
--debug, -v, -d, --verbose
Use verbose output. (default: False)
[25]:
freva-user-data delete /tmp/user_data/user-$USER
freva-databrowser experiment=bias-correct