Plugin API Reference

Definition of the base class that is used to implement user plugins.

The plugin API (Application Program Interface) is the central connection of a user plugin code and the Freva system infrastructure. The API enables the users to conveniently set up, run, and keep track of applied plugins. The reference below gives an overview of how to set up user defined plugins. For this purpose we assume that a plugin core (without Freva) already exists - for example as a command line interface tool (cli). Once such a cli has been set up a interface to Freva must be defined. This reference will introduce the possible definition options below.

Here we assume that the above mentioned cli code is stored in a directory name /mnt/freva/plugins/new_plugin furthermore the actual cli code can be executed via:

cli/calculate -c 5 -n 6.4 --overwrite --name=Test

With help of this API a Freva plugin can be created in /mnt/freva/plugin/new_plugin/plugin.py

class evaluation_system.api.plugin.PluginAbstract(*args, user: User | None = None, **kwargs)

Bases: ABC

Base class that is used as a template for all Freva plugins. Any api wrapper class defining Freva plugins must inherit from this class.

Parameters:

user (evaluation_system.model.user.User, default None) – Pre defined evaluation system user object, if None given (default) a new instance of a user object for the current user will be created.

The following attributes are mandatory and have to be set:

• __short_description__

• __version__

• __parameters__

• run_tool

Whereas the following properties can be optionally set:

• __long_description__

• __tags__

• __category__

Example

In order to configure and call this cli, a Freva wrapper api class will have the be created in /mnt/freva/plugins/new_plugin/plugin.py. A minimal configuration example would look as follows:

from evaluation_system.api import plugin, parameters
class MyPlugin(plugin.PluginAbstract):
    __short_description__ = "Short plugin description"
    __long_description__ = "Optional longer description"
    __version__ = (2022, 1, 1)
    __parameters__ =  parameters.ParameterDictionary(
                        parameters.Integer(
                            name="count",
                            default=5,
                            help=("This is an optional configurable "
                                  "int variable named number without "
                                  "default value and this description")
                        ),
                        parameters.Float(
                            name="number",
                            default=6.4,
                            mandatory=True,
                            help="Required float value without default"
                        ),
                        parameters.Bool(
                            name="overwrite",
                            default=True,
                            help=("a boolean parameter "
                                  "with default value of false")
                        ),
                        parameters.String(
                            name='name',
                            default='Test',)
                      )
    def run_tool(
        self, config_dict: dict[str, str|int|bool]
    ) -> None:
        '''Definition of the tool that runs the cli.

        Parameters:
        -----------
        config_dict: dict
            Plugin configuration stored in a dictionary
        '''

        self.call(
            (
              f"cli/calculate -c {config_dict['count']} "
              f"-n {config_dict['number']} --name={config_dict['name']}"
            )
        )
        print("MyPlugin was run with", config_dict)

Note

The actual configuration is defined by the __parameters__ property, which is of type evaluation_system.api.parameters.ParameterDictionary.

If you need to test it use the EVALUATION_SYSTEM_PLUGINS environment variable to point to the source directory and package. For example assuming you have the source code in /mnt/freva/plugins and the package holding the class implementing evaluation_system.api.plugin is my_plugin.plugin (i.e. its absolute file path is /mnt/freva/plugins/my_plugin/plugin_module.py), you would tell the system how to find the plugin by issuing the following command (bash & co):

export EVALUATION_SYSTEM_PLUGINS=/mnt/freva/plugins/my_plugin,plugin_module

Use a colon to separate multiple items:

export EVALUATION_SYSTEM_PLUGINS=/path1,plugin1:/path2,plugin2:/path3,plugin3

By telling the system where to find the packages it can find the evaluation_system.api.plugin implementations. The system just loads the packages and get to the classes using the class.__subclasses__() method. The reference speaks about weak references so it’s not clear if (and when) they get removed. We might have to change this in the future if it’s not enough. Another approach would be forcing self-registration of a class in the __metaclass__ attribute when the class is implemented.

property __category__: str

Optional category this plugin belongs to.

property __long_description__: str

Optional long description of this plugin.

abstract property __parameters__

Mandatory definitions of all known configurable parameters.

abstract property __short_description__: str

Mandatory short description of this plugin.

It will be displayed to the user in the help and when listing all plugins.

property __tags__: list[str]

Optional tags, that are the plugin can be described with.

abstract property __version__: tuple[int, int, int]

3-value tuple representing the plugin version.

Example

>>>    version = (1, 0, 3)

add_output_to_databrowser(plugin_output: PathLike, project: str, product: str, *, model: str = 'freva', institute: str | None = None, ensemble: str = 'r1i1p1', time_frequency: str | None = None, variable: str | None = None, experiment: str | None = None, index_data: bool = True) → Path

Add Plugin output data to the solr database.

This methods crawls the plugin output data directory and adds any files that were found to the apache solr database.

..note::

Use the ensemble and model arguments to specify the search facets that are going to be added to the solr server for this plugin run. This will help users better to better distinguish their plugin result search.

The following facets are fixed:

• project: user-<user_name>

• product: <project>.<product>

• realm: <plugin_name>

• dataset version: <history_id>

Parameters:

• plugin_output (os.PathLike) – Plugin output directory or file created by the files. If a directory is given all data files within the sub directories will be collected for ingestion.

• project (str) – Project facet of the input data. The project argument will distinguish plugin results for different setups.

• product (str) – Product facet of the input data. The product argument will distinguish plugin results for different setups.

• model (str, default: None) – Default model facet. If None is given (default) the model name will be set to freva. The model argument can be used to distinguish plugin results for different setups.

• institute (str, default: None) – Default institute facet. Use the argument to make plugin results from various setups better distinguishable. If None given (default) the institute will be set the the freva project name.

• ensemble (str, default: r0i0p0) – Default ensemble facet. Like for model the ensemble argument should be used to distinguish plugins results with different setups.

• time_frequency (str, default: None) – Default time frequency facet. If None is given (default) the time frequency will be retrieved from the output files.

• experiment (str, default: freva-plugin) – Default time experiment facet.

• variable (str, default: None) – Default variable facet. If None is given (default) the variable will be retrieved from the output files.

• index_data (bool, default: True) – Update the freva data browser (default: True). Setting this flag to false can be useful if the data structure should only be created but the databrowser should not be updated yet.

Returns:

Path

Return type:

Path to the new directory that contains the data.

call(cmd_string: str | list[str], check: bool = True, **kwargs) → Popen[Any]

Run command with arguments and return a CompletedProcess instance.

The returned instance will have attributes args, returncode, stdout and stderr. By default, stdout and stderr are not captured, and those attributes will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them.

Please refer to the subprocess.Popen python build in module for more details.

Parameters:

• cmd_string – the command to be submitted.

• check – If check is True and the exit code was non-zero, it raises a CalledProcessError. The CalledProcessError object will have the return code in the returncode attribute, and output & stderr attributes if those streams were captured.

• kwargs – Additional arguments passed to subprocess.Popen

Returns:

sub-process return value

Return type:

subprocess.Popen

property class_basedir: str

Get absolute path to the module defining the plugin class.

getClassBaseDir(**kwargs: Any) → Any

getHelp(**kwargs: Any) → Any

linkmydata(**kwargs: Any) → Any

property plugin_output_file: Path

Filename where stdout is written to.

prepareOutput(**kwargs: Any) → Any

prepare_output(output_files: str | list[str] | dict[str, dict[str, str]]) → dict[str, dict[str, str]]

Prepare output for files supposedly created.

This method checks if the files exist and returns a dictionary with information about them:

{ <absolute_path_to_file>: {
    'timestamp': os.path.getctime(<absolute_path_to_file>),
    'size': os.path.getsize(<absolute_path_to_file>)
    }
}

Use it for the return call of run_tool.

Parameters:

output_files – iterable of strings or single string with paths to all files that where created by the tool.

Returns:

dictionary with the paths to the files that were created as key and a dictionary as value.

Return type:

dict

readConfiguration(**kwargs: Any) → Any

run_tool(config_dict: Dict[str, str | float | int | bool | None] | None = None) → Any | None

Method executing the tool.

The method should be overridden by the custom plugin tool method.

Parameters:

config_dict – A dict with the current configuration (param name, value) which the tool will be run with

Returns:

Return values of prepare_output([<list_of_created_files>]) method

Return type:

list[str]

setupConfiguration(**kwargs: Any) → Any

special_variables: dict[str, str] | None = None

This dictionary resolves the special variables that are available

to the plugins in a standardized manner. These variables are initialized per user and plugin. They go as follow:

Variables	Description
USER_BASE_DIR	Absolute path to the central directory for this user.
USER_OUTPUT_DIR	Absolute path to where the plugin outputs for this user are stored.
USER_PLOTS_DIR	Absolute path to where the plugin plots for this user are stored.
USER_CACHE_DIR	Absolute path to the cached data (temp data) for this user.
USER_UID	The users’ User Identifier
SYSTEM_DATE	Current date in the form YYYYMMDD (e.g. 20120130).
SYSTEM_DATETIME	Current date in the form YYYYMMDD_HHmmSS (e.g. 20120130_101123).
SYSTEM_TIMESTAMP	Milliseconds since epoch (i.e. e.g. 1358929581838).
SYSTEM_RANDOM_UUID	A random Universal Unique Identifier (e.g. 912cca21-6364-4f46-9b03-4263410c9899).

A plugin/user might then use them to define a value in the following way:

output_file='$USER_OUTPUT_DIR/myfile_${SYSTEM_DATETIME}blah.nc'

tool_developer: str | None = None

Name of the developer who is responsible for the tool.

property user: User

Return the user class for which this instance was generated.

property wrapper_file: str

Get the location of the wrapper file.