Applying data analysis Plugins#

Already defined data analysis tools can be started with the freva.run_plugin method. Besides the run_plugin method two more utility methods (list_plugins and plugin_doc) are available to get an overview over existing plugins and the documentation of each plugins.

First let’s get an overview of the plugins that are available in the system:

[1]:
import freva
[2]:
freva.list_plugins()
[2]:
['dummyplugin', 'dummypluginfolders', 'animator']

The help menu of each plugin can be inspected by calling the plugin_doc method. For example the help of the animator plugin

[3]:
print(freva.plugin_doc("animator"))
Animator (v2022.7.15): Create animations (in gif or mp4 format) This tool creates plots of solr facets and an animation.
Options:
input_file              (default: <null>)
                        NetCDF input file(s), you can choose multiple files
                        separated by a , or use a global pattern for multiple
                        files chose this option only if you don't want Freva to
                        find files by seach facets
variable                (default: <null> [mandatory])
                        Variable name (only applicable if you didn't choose an
                        input file)
project                 (default: <null>)
                        Project name (only applicable if you didn't choose an
                        input file)
product                 (default: <null>)
                        Product name (only applicable if you didn't choose an
                        input file)
experiment              (default: <null>)
                        Experiment name (only applicable if you didn't choose an
                        input file)
institute               (default: <null>)
                        Institute name (only applicable if you didn't choose an
                        input file)
model                   (default: <null>)
                        Model name (only applicable if you didn't choose an
                        input file)
time_frequency          (default: <null>)
                        Time frequency name (only applicable if you didn't
                        choose an input file)
ensemble                (default: <null>)
                        Ensemble name (only applicable if you didn't choose an
                        input file)
start                   (default: <null>)
                        Define the first time step to be plotted, leave blank if
                        taken from data
end                     (default: <null>)
                        Define the last time step to be plotted, leave blank if
                        taken from data
time_mean               (default: <null>)
                        Select a time interval if time averaging/min/max/sum
                        should be applied along the time axis. This can be D for
                        daily, M for monthly 6H for 6 hours etc. Leave blank if
                        the time axis should not be resampled (default) or set
                        to <em>all</em> if you want to collapse the time axis.
time_method             (default: mean)
                        If resampling of the time axis is chosen (default: no)
                        set the method: mean, max or min. <b>Note:</b> This has
                        only an effect if the above parameter for
                        <em>time_mean</em> is set.
lonlatbox               (default: <null>)
                        Set the extend of a rectangular lonlatbox (left_lon,
                        right_lon, lower_lat, upper_lat)
output_unit             (default: <null>)
                        Set the output unit of the variable - leave blank for no
                        conversion. This can be useful if the unit of the input
                        files should be converted, for example for
                        precipitation. Note: Although many conversions are
                        supported, by using the `pint` conversion library.
vmin                    (default: <null>)
                        Set the minimum plotting range (leave blank to calculate
                        from data 1st decile)
vmax                    (default: <null>)
                        Set the maximum plotting range (leave blank to calculate
                        the 9th decile)
cmap                    (default: RdYlBu_r)
                        Set the colormap, more information on colormaps is
                        available on the <a href="https://matplotlib.org/stable/
                        tutorials/colors/colormaps.html"
                        target=_blank>matplotlib website</a>.
linecolor               (default: k)
                        Color of the coast lines in the map
projection              (default: PlateCarree)
                        Set the global map projection. Note: this should the
                        name of the cartopy projection method (e.g PlatteCarree
                        for Cylindrical Projection). Pleas refer to <a href="htt
                        ps://scitools.org.uk/cartopy/docs/latest/crs/projections
                        .html"target=_blank>cartopy website</a> for details.
proj_centre             (default: 50)
                        Set center longitude of the global map projection.
pic_size                (default: 1360,900)
                        Set the size of the picture (in pixel)
plot_title              (default: )
                        Set plot title
cbar_label              (default: )
                        Overwrite default colorbar label by this value
suffix                  (default: mp4)
                        Filetype of the animation
fps                     (default: 5)
                        Set the frames per seceonds of the output animation.
extra_scheduler_options (default: --qos=test, --array=20)
                        Set additional options for the job submission to the
                        workload manager (, separated). Note: batchmode and web
                        only.

Let’s apply the plugin an create an animation of some observational data.

Note: The plugin user interface allows us to also use key=value pair search queries to search for data in the system, just like the databrowser.

[4]:
freva.run_plugin("animator", variable="pr", project="obs*")
/tmp/animator/run.py:91: SyntaxWarning: invalid escape sequence '\d'
  for i in re.findall("[-\d]+", unit[n]):
Setp 1: Collecting all files
Step 2: Opening the netcdf-files, collecting metadata
Step 3: Converting Units
Step 4: Loading the dataset, getting min/max values
Step 5: Creating the animation
/tmp/animator/run.py:91: SyntaxWarning: invalid escape sequence '\d'
  for i in re.findall("[-\d]+", unit[n]):
Created animation /tmp/eval_conf/work/runner/freva-ces/output/animator/2128/pr_20160902T0000-20160902T2330.mp4 in 41 seconds
[4]:
PluginStatus('animator', config={'input_file': None, 'variable': 'pr', 'project': 'obs*', 'product': None, 'experiment': None, 'institute': None, 'model': None, 'time_frequency': None, 'ensemble': None, 'start': None, 'end': None, 'time_mean': None, 'time_method': 'mean', 'lonlatbox': None, 'output_unit': None, 'vmin': None, 'vmax': None, 'cmap': 'RdYlBu_r', 'linecolor': 'k', 'projection': 'PlateCarree', 'proj_centre': 50, 'pic_size': '1360,900', 'plot_title': '', 'cbar_label': '', 'suffix': 'mp4', 'fps': 5}, status=finished)

Search the history of previously applied plugins#

To access the configurations of previously applied freva plugins freva offers the history method which displays plugin entries with a one-line compact description. The first number you see is the entry id, which you might use to select single entries for further details. Again, let’s inspect the help first:

[5]:
help(freva.history)
Help on function history in module freva._history:

history(*args: 'str', limit: 'int' = 10, plugin: 'Optional[str]' = None, since: 'Optional[str]' = None, until: 'Optional[str]' = None, entry_ids: 'Union[int, list[int], None]' = None, full_text: 'bool' = False, return_results: 'bool' = False, return_command: 'bool' = False, _return_dict: 'bool' = True, user_name: 'Optional[str]' = None) -> 'Union[list[Any], dict[str, Any]]'
    Get access to the configurations of previously applied freva plugins.

    The `.history` method displays the entries with a one-line compact description.
    The first number you see is the entry id, which you might use to select
    single entries.

    Parameters
    ----------
    limit: int, default: 10
      Limit the number of entries to be displayed.
    plugin: str, default: None
      Display only entries from a given plugin name.
    since: str, datetime.datetime, default: None
      Retrieve entries older than date, see hint on date format below.
    until: str, datetime.datetime, default: None
      Retrieve entries younger than date, see hint on date format below.
    entry_ids: list, default: None
       Select entries whose ids are in "ids".
    full_text: bool, default: False
      Show the complete configuration.
    return_results: bool, default: False
      Also return the plugin results.
    return_command: bool, default: False
      Return the commands instead of history objects.
    user_name: str, default: None
      Select entries belonging to another user (e.g., user_name="<username>")
      or all users (all or *).
    _return_dict: bool, default: True
      Return a dictionary representation, this is only for internal use.

    Returns
    -------
      list:
        freva plugin history

    Example
    -------

    Get the last three history entries:

    .. execute_code::

        import freva
        hist = freva.history(limit=3)
        print(type(hist), len(hist))
        print(hist[-1].keys())
        config = hist[-1]['configuration']
        print(config)


    .. hint:: Date Format
        Dates are given in the ISO-8601 format and can be "YYYY-MM-DDTHH:mm:ss.n"
        or any less accurate subset. These are all valid: "2012-02-01T10:08:32.1233431",
        "2012-02-01T10:08:32", "2012-02-01T10:08", "2012-02-01T10", "2012-02-01",
        "2012-02", "2012". Missing values are assumed to be the minimal allowed value.
        For example: "2012" = "2012-01-01T00:00:00.0"

The configuration of last entry of the animator plugin history can be accessed by:

[6]:
animator_history = freva.history("animator")[-1]
animator_history
[6]:
{'status_dict': {2: 'broken',
  0: 'finished',
  1: 'finished_no_output',
  5: 'not_scheduled',
  3: 'running',
  4: 'scheduled'},
 'id': 2118,
 'timestamp': '2024-10-25T04:17:31',
 'tool': 'animator',
 'version': '(2022, 7, 15)',
 'version_details_id': 124,
 'configuration': {'input_file': None,
  'variable': 'pr',
  'project': 'observations',
  'product': None,
  'experiment': None,
  'institute': None,
  'model': None,
  'time_frequency': None,
  'ensemble': None,
  'start': None,
  'end': None,
  'time_mean': None,
  'time_method': 'mean',
  'lonlatbox': None,
  'output_unit': None,
  'vmin': None,
  'vmax': None,
  'cmap': 'RdYlBu_r',
  'linecolor': 'k',
  'projection': 'PlateCarree',
  'proj_centre': 50,
  'pic_size': '1360,900',
  'plot_title': '',
  'cbar_label': '',
  'suffix': 'mp4',
  'fps': 5},
 'slurm_output': '/tmp/share/slurm/animator/Animator-3228.out',
 'host': 'fv-az1245-298',
 'uid_id': 'runner',
 'status': 0,
 'flag': 0,
 'caption': '',
 'result': {}}
[7]:
animator_history["configuration"]
[7]:
{'input_file': None,
 'variable': 'pr',
 'project': 'observations',
 'product': None,
 'experiment': None,
 'institute': None,
 'model': None,
 'time_frequency': None,
 'ensemble': None,
 'start': None,
 'end': None,
 'time_mean': None,
 'time_method': 'mean',
 'lonlatbox': None,
 'output_unit': None,
 'vmin': None,
 'vmax': None,
 'cmap': 'RdYlBu_r',
 'linecolor': 'k',
 'projection': 'PlateCarree',
 'proj_centre': 50,
 'pic_size': '1360,900',
 'plot_title': '',
 'cbar_label': '',
 'suffix': 'mp4',
 'fps': 5}

The same goes for the status of the tool

[8]:
animator_history["status_dict"][animator_history["status"]]
[8]:
'finished'