Plugin Developer Guide

This documentation helps to get started with creating user defined Freva plugins. This section provides a minimal example to turn your existing data analysis code into a Freva plugin. Detailed usage information can be found in the API Reference.

A Minimal Example

Definition of the base class that is used to implement user plugins.

The plugin API (Application Program Interface) is the central connection of a user plugin code and the Freva system infrastructure. The API enables the users to conveniently set up, run, and keep track of applied plugins. The reference below gives an overview of how to set up user defined plugins. For this purpose we assume that a plugin core (without Freva) already exists - for example as a command line interface tool (cli). Once such a cli has been set up a interface to Freva must be defined. This reference will introduce the possible definition options below.

Here we assume that the above mentioned cli code is stored in a directory name /mnt/freva/plugins/new_plugin furthermore the actual cli code can be executed via:

cli/calculate -c 5 -n 6.4 --overwrite --name=Test

With help of this API a Freva plugin can be created in /mnt/freva/plugin/new_plugin/plugin.py

class evaluation_system.api.plugin.PluginAbstract(*args, user: User | None = None, **kwargs)

Base class that is used as a template for all Freva plugins. Any api wrapper class defining Freva plugins must inherit from this class.

Parameters:

user (evaluation_system.model.user.User, default None) – Pre defined evaluation system user object, if None given (default) a new instance of a user object for the current user will be created.

The following attributes are mandatory and have to be set:

• __short_description__

• __version__

• __parameters__

• run_tool

Whereas the following properties can be optionally set:

• __long_description__

• __tags__

• __category__

Example

In order to configure and call this cli, a Freva wrapper api class will have the be created in /mnt/freva/plugins/new_plugin/plugin.py. A minimal configuration example would look as follows:

from evaluation_system.api import plugin, parameters
class MyPlugin(plugin.PluginAbstract):
    __short_description__ = "Short plugin description"
    __long_description__ = "Optional longer description"
    __version__ = (2022, 1, 1)
    __parameters__ =  parameters.ParameterDictionary(
                        parameters.Integer(
                            name="count",
                            default=5,
                            help=("This is an optional configurable "
                                  "int variable named number without "
                                  "default value and this description")
                        ),
                        parameters.Float(
                            name="number",
                            default=6.4,
                            mandatory=True,
                            help="Required float value without default"
                        ),
                        parameters.Bool(
                            name="overwrite",
                            default=True,
                            help=("a boolean parameter "
                                  "with default value of false")
                        ),
                        parameters.String(
                            name='name',
                            default='Test',)
                      )
    def run_tool(
        self, config_dict: dict[str, str|int|bool]
    ) -> None:
        '''Definition of the tool that runs the cli.

        Parameters:
        -----------
        config_dict: dict
            Plugin configuration stored in a dictionary
        '''

        self.call(
            (
              f"cli/calculate -c {config_dict['count']} "
              f"-n {config_dict['number']} --name={config_dict['name']}"
            )
        )
        print("MyPlugin was run with", config_dict)

Note

The actual configuration is defined by the __parameters__ property, which is of type evaluation_system.api.parameters.ParameterDictionary.

If you need to test it use the EVALUATION_SYSTEM_PLUGINS environment variable to point to the source directory and package. For example assuming you have the source code in /mnt/freva/plugins and the package holding the class implementing evaluation_system.api.plugin is my_plugin.plugin (i.e. its absolute file path is /mnt/freva/plugins/my_plugin/plugin_module.py), you would tell the system how to find the plugin by issuing the following command (bash & co):

export EVALUATION_SYSTEM_PLUGINS=/mnt/freva/plugins/my_plugin,plugin_module

Use a colon to separate multiple items:

export EVALUATION_SYSTEM_PLUGINS=/path1,plugin1:/path2,plugin2:/path3,plugin3

By telling the system where to find the packages it can find the evaluation_system.api.plugin implementations. The system just loads the packages and get to the classes using the class.__subclasses__() method. The reference speaks about weak references so it’s not clear if (and when) they get removed. We might have to change this in the future if it’s not enough. Another approach would be forcing self-registration of a class in the __metaclass__ attribute when the class is implemented.

Setting up your new plugin

This section illustrates the steps that are necessary to turn existing data analysis code into a Freva plugin. Like above, we assume that the code is stored in a specific location for example ~/workspace/tracking_tool. Also, let’s assume that the analysis tool is written in the R script language.

Creating a new repository from a template

We have created a template repository that helps you getting started with the Freva plugin development. Therefore we recommend you to use this repository. Use the following commands to turn this template repository into your new Freva plugin repository:

wget https://gitlab.dkrz.de/freva/plugins4freva/plugintemplate/-/archive/main/plugintemplate-main.zip
unzip plugintemplate-main.zip
mv plugintemplate-main freva_tracking
cd freva_tracking
git init --shared .
cp -r ~/workspace/tracking_tool src
git add .

You have now created a new Freva plugin repository. It is a good idea to use some kind of repository server, like gitlab, where you make your code accessible. Talk to your Freva admins to work out a good location for your code. Once you have agreed upon a location you should create a new repository on the server side using the web interface of your repository host system. Once created set the remote host address on the locally created repository (the one where you did a git init):

git remote set-url origin https://gitlab.com/path/to/the/remote/repo.git

Hint

If you are unfamiliar with git you can find plenty of online resources on the web. A good resource might be the official git tutorial page.

Installing dependencies

Once the git repository has been set up and configured, all dependencies the tool needs should be installed. Here we assume the analysis tool is based on a gnu-R stack. Therefore gnu-R and certain libraries have to be part of the plugin environment. This environment will be created using anaconda. To find out what dependencies you should be installing query the anaconda search page. Once you have found all packages that can be installed via anaconda you can add them to the deployment/plugin-env.yaml file. Simply add the entries that are needed to the existing file.

channels:
     - conda-forge
 dependencies:
     - conda
     - r-base
     - r-essentials
     - r-ncdf4
     - pip
     - black

Probably there are package dependencies that cannot be installed via anaconda and need to bee installed otherwise. To do so you can use a simple command line interface in deployment/install_resources.py and add the following command into the build section of the Makefile in the repository:

build:
    python deployment/install_resources.py gnu-r ncdf4.helpers

To get an overview over the full functionality of the installation cli you can query the help.

python deployment/install_resources.py --help

After everything is setup you can build use the make command to deploy the plugin environment.

make all

Note

The Makefile will use the conda command. If anaconda is not available by default on your system you can load the Freva environment, which ships anaconda.

Afterwards you can refer to the Plugin API Reference and Parameter API Reference docs to create the wrapper file and finalize the creation of the plugin.