Applying data analysis Plugins#
Already defined data analysis tools can be started with the freva.run_plugin method. Besides the run_plugin method two more utility methods (list_plugins and plugin_doc) are available to get an overview over existing plugins and the documentation of each plugins.
First let’s get an overview of the plugins that are available in the system:
[1]:
import freva
[2]:
freva.list_plugins()
[2]:
['dummyplugin', 'dummypluginfolders', 'animator']
The help menu of each plugin can be inspected by calling the plugin_doc method. For example the help of the animator plugin
[3]:
print(freva.plugin_doc("animator"))
Animator (v2022.7.15): Create animations (in gif or mp4 format) This tool creates plots of solr facets and an animation.
Options:
input_file (default: <null>)
NetCDF input file(s), you can choose multiple files
separated by a , or use a global pattern for multiple
files chose this option only if you don't want Freva to
find files by seach facets
variable (default: <null> [mandatory])
Variable name (only applicable if you didn't choose an
input file)
project (default: <null>)
Project name (only applicable if you didn't choose an
input file)
product (default: <null>)
Product name (only applicable if you didn't choose an
input file)
experiment (default: <null>)
Experiment name (only applicable if you didn't choose an
input file)
institute (default: <null>)
Institute name (only applicable if you didn't choose an
input file)
model (default: <null>)
Model name (only applicable if you didn't choose an
input file)
time_frequency (default: <null>)
Time frequency name (only applicable if you didn't
choose an input file)
ensemble (default: <null>)
Ensemble name (only applicable if you didn't choose an
input file)
start (default: <null>)
Define the first time step to be plotted, leave blank if
taken from data
end (default: <null>)
Define the last time step to be plotted, leave blank if
taken from data
time_mean (default: <null>)
Select a time interval if time averaging/min/max/sum
should be applied along the time axis. This can be D for
daily, M for monthly 6H for 6 hours etc. Leave blank if
the time axis should not be resampled (default) or set
to <em>all</em> if you want to collapse the time axis.
time_method (default: mean)
If resampling of the time axis is chosen (default: no)
set the method: mean, max or min. <b>Note:</b> This has
only an effect if the above parameter for
<em>time_mean</em> is set.
lonlatbox (default: <null>)
Set the extend of a rectangular lonlatbox (left_lon,
right_lon, lower_lat, upper_lat)
output_unit (default: <null>)
Set the output unit of the variable - leave blank for no
conversion. This can be useful if the unit of the input
files should be converted, for example for
precipitation. Note: Although many conversions are
supported, by using the `pint` conversion library.
vmin (default: <null>)
Set the minimum plotting range (leave blank to calculate
from data 1st decile)
vmax (default: <null>)
Set the maximum plotting range (leave blank to calculate
the 9th decile)
cmap (default: RdYlBu_r)
Set the colormap, more information on colormaps is
available on the <a href="https://matplotlib.org/stable/
tutorials/colors/colormaps.html"
target=_blank>matplotlib website</a>.
linecolor (default: k)
Color of the coast lines in the map
projection (default: PlateCarree)
Set the global map projection. Note: this should the
name of the cartopy projection method (e.g PlatteCarree
for Cylindrical Projection). Pleas refer to <a href="htt
ps://scitools.org.uk/cartopy/docs/latest/crs/projections
.html"target=_blank>cartopy website</a> for details.
proj_centre (default: 50)
Set center longitude of the global map projection.
pic_size (default: 1360,900)
Set the size of the picture (in pixel)
plot_title (default: )
Set plot title
cbar_label (default: )
Overwrite default colorbar label by this value
suffix (default: mp4)
Filetype of the animation
fps (default: 5)
Set the frames per seceonds of the output animation.
extra_scheduler_options (default: --qos=test, --array=20)
Set additional options for the job submission to the
workload manager (, separated). Note: batchmode and web
only.
Let’s apply the plugin an create an animation of some observational data.
Note: The plugin user interface allows us to also use key=value pair search queries to search for data in the system, just like the databrowser.
[4]:
freva.run_plugin("animator", variable="pr", project="obs*")
/tmp/animator/run.py:91: SyntaxWarning: invalid escape sequence '\d'
for i in re.findall("[-\d]+", unit[n]):
Setp 1: Collecting all files
Step 2: Opening the netcdf-files, collecting metadata
Step 3: Converting Units
Step 4: Loading the dataset, getting min/max values
Step 5: Creating the animation
/tmp/animator/run.py:91: SyntaxWarning: invalid escape sequence '\d'
for i in re.findall("[-\d]+", unit[n]):
Created animation /tmp/eval_conf/work/runner/freva-ces/output/animator/2128/pr_20160902T0000-20160902T2330.mp4 in 42 seconds
[4]:
PluginStatus('animator', config={'input_file': None, 'variable': 'pr', 'project': 'obs*', 'product': None, 'experiment': None, 'institute': None, 'model': None, 'time_frequency': None, 'ensemble': None, 'start': None, 'end': None, 'time_mean': None, 'time_method': 'mean', 'lonlatbox': None, 'output_unit': None, 'vmin': None, 'vmax': None, 'cmap': 'RdYlBu_r', 'linecolor': 'k', 'projection': 'PlateCarree', 'proj_centre': 50, 'pic_size': '1360,900', 'plot_title': '', 'cbar_label': '', 'suffix': 'mp4', 'fps': 5}, status=finished)
Search the history of previously applied plugins#
To access the configurations of previously applied freva plugins freva offers the history method which displays plugin entries with a one-line compact description. The first number you see is the entry id, which you might use to select single entries for further details. Again, let’s inspect the help first:
[5]:
help(freva.history)
Help on function history in module freva._history:
history(*args: 'str', limit: 'int' = 10, plugin: 'Optional[str]' = None, since: 'Optional[str]' = None, until: 'Optional[str]' = None, entry_ids: 'Union[int, list[int], None]' = None, full_text: 'bool' = False, return_results: 'bool' = False, return_command: 'bool' = False, _return_dict: 'bool' = True, user_name: 'Optional[str]' = None) -> 'Union[list[Any], dict[str, Any]]'
Get access to the configurations of previously applied freva plugins.
The `.history` method displays the entries with a one-line compact description.
The first number you see is the entry id, which you might use to select
single entries.
Parameters
----------
limit: int, default: 10
Limit the number of entries to be displayed.
plugin: str, default: None
Display only entries from a given plugin name.
since: str, datetime.datetime, default: None
Retrieve entries older than date, see hint on date format below.
until: str, datetime.datetime, default: None
Retrieve entries younger than date, see hint on date format below.
entry_ids: list, default: None
Select entries whose ids are in "ids".
full_text: bool, default: False
Show the complete configuration.
return_results: bool, default: False
Also return the plugin results.
return_command: bool, default: False
Return the commands instead of history objects.
user_name: str, default: None
Select entries belonging to another user (e.g., user_name="<username>")
or all users (all or *).
_return_dict: bool, default: True
Return a dictionary representation, this is only for internal use.
Returns
-------
list:
freva plugin history
Example
-------
Get the last three history entries:
.. execute_code::
import freva
hist = freva.history(limit=3)
print(type(hist), len(hist))
print(hist[-1].keys())
config = hist[-1]['configuration']
print(config)
.. hint:: Date Format
Dates are given in the ISO-8601 format and can be "YYYY-MM-DDTHH:mm:ss.n"
or any less accurate subset. These are all valid: "2012-02-01T10:08:32.1233431",
"2012-02-01T10:08:32", "2012-02-01T10:08", "2012-02-01T10", "2012-02-01",
"2012-02", "2012". Missing values are assumed to be the minimal allowed value.
For example: "2012" = "2012-01-01T00:00:00.0"
The configuration of last entry of the animator plugin history can be accessed by:
[6]:
animator_history = freva.history("animator")[-1]
animator_history
[6]:
{'status_dict': {2: 'broken',
0: 'finished',
1: 'finished_no_output',
5: 'not_scheduled',
3: 'running',
4: 'scheduled'},
'id': 2118,
'timestamp': '2024-11-14T05:07:09',
'tool': 'animator',
'version': '(2022, 7, 15)',
'version_details_id': 124,
'configuration': {'input_file': None,
'variable': 'pr',
'project': 'observations',
'product': None,
'experiment': None,
'institute': None,
'model': None,
'time_frequency': None,
'ensemble': None,
'start': None,
'end': None,
'time_mean': None,
'time_method': 'mean',
'lonlatbox': None,
'output_unit': None,
'vmin': None,
'vmax': None,
'cmap': 'RdYlBu_r',
'linecolor': 'k',
'projection': 'PlateCarree',
'proj_centre': 50,
'pic_size': '1360,900',
'plot_title': '',
'cbar_label': '',
'suffix': 'mp4',
'fps': 5},
'slurm_output': '/tmp/share/slurm/animator/Animator-3366.out',
'host': 'fv-az1022-431',
'uid_id': 'runner',
'status': 0,
'flag': 0,
'caption': '',
'result': {}}
[7]:
animator_history["configuration"]
[7]:
{'input_file': None,
'variable': 'pr',
'project': 'observations',
'product': None,
'experiment': None,
'institute': None,
'model': None,
'time_frequency': None,
'ensemble': None,
'start': None,
'end': None,
'time_mean': None,
'time_method': 'mean',
'lonlatbox': None,
'output_unit': None,
'vmin': None,
'vmax': None,
'cmap': 'RdYlBu_r',
'linecolor': 'k',
'projection': 'PlateCarree',
'proj_centre': 50,
'pic_size': '1360,900',
'plot_title': '',
'cbar_label': '',
'suffix': 'mp4',
'fps': 5}
The same goes for the status of the tool
[8]:
animator_history["status_dict"][animator_history["status"]]
[8]:
'finished'