Running a pipeline¶

Warning

Currently unsupported plugin groups for python pipeline’s API and the command-line gnomon-utils run :

formAlgorithm (nodes from PythonAlgorithm workspace) and imageRegistration

Pipeline containing those nodes can still be runned through gnomon’s GUI.

A pipeline is a graph of tasks made by different plugins. Input and output is done through source and sinks nodes, respectively readers and constructors for sources and writers for sinks.

Pipelines can be saved in pipeline files which are json files. A pipeline can be created in gnomon.

We will use the following pipeline as an example.

digraph pipeline { imageReader1 -> imageFilter1 imageFilter1 -> imageFilter2 imageFilter2 -> binarization binarization -> binaryImageWriter1 }

In this pipeline there is one source node and one sink, respectively imageReader1 and binaryImageWriter1. A pipeline requires at least one of each to serve as input output. Both take a path as argument which is configurable.

Another class of source available is the Constructor class which generates artificial forms.

The corresponding pipeline file is available at the bottom of this page.

Run pipeline with command line¶

The command to run a pipeline file is:

gnomon-utils run my_pipeline.json

To run a pipeline with new input and output path use:

gnomon-utils run my_pipeline.json --ios imageReader1 ./new/image1.inr binaryImageWriter1 ./new/out.inr

If the pipeline contains relative path you can change the path to the directory containing the data.

gnomon-utils run my_pipeline.json --data-dir ../gnomon-data

Python API¶

A pipeline can also be run through the python API in the subpackage gnomon.utils.pipelines.

Examples¶

Loading and executing a pipeline.

from gnomon.utils.pipelines import load_pipeline

pr = load_pipeline("./my_pipeline.json", data_dir="./data")
pr.run()

Loading changing input path and then executing a pipeline.

from gnomon.utils.pipelines import load_pipeline

pr = load_pipeline("./my_pipeline.json", data_dir="./data")
pr.path_dict["imageReader1"] = "./new/image1.inr"
pr.run()

API documentation¶

class gnomon.utils.pipelines.PNodeRunner(node: gnomon.pipeline.gnomonPipelineNode, data_dir: str = '')¶

Node wrapper which instantiate the corresponding plugin, parametrize them and provides a unified interface.

Parameters

node (gnomon.pipeline.gnomonPipelineNode) –
data_dir (pathlib.Path) –

name¶

Name of the node.

Type: str

algo¶

Algorithm plugin.

Type: gnomonAbstractAlgorithm

inputs_connections¶

Dictionary of input-port -> (source-node, source-port) which maps the input connections of the node.

Type: Dict[str, Tuple[str, str]]

inputs¶

Dictionary mapping input ports with their respective setter method.

Type: Dict[str, Callable[[gnomonAbstractDynamicForm], None]]

outputs¶

Dictionary mapping output ports with their respective getter method.

Type: Dict[str, Callable[[], gnomonAbstractDynamicForm]]

__init__(node: gnomon.pipeline.gnomonPipelineNode, data_dir: str = '')¶

Node wrapper which instantiate the corresponding plugin, parametrize them and provides a unified interface.

Parameters

node (gnomonPipelineNode) –
data_dir (str) –

run()¶: Calls the run() method of this node plugin.

has_path() → bool¶

Returns True if this node has a path parameter.

Return type: bool

setPath(path: str)¶

Sets the path of this node if it has a path parameter (else do nothing).

Parameters: path (str) – New path for the node.

class gnomon.utils.pipelines.PipelineRunner(pipeline: gnomon.pipeline.gnomonPipeline, data_dir: str = '')¶

Instantiate the plugins and runs the pipeline.

Takes a gnomonPipeline object and instantiate the plugins of each node through the use of PNodeRunner. Use run() to execute the pipeline.

Parameters

pipeline (gnomon.pipeline.gnomonPipeline) –
data_dir (pathlib.Path) –

pipeline¶

Source gnomonPipeline object used to build the object.

Type: gnomonPipeline

nodes¶

Dictionary mapping node_name –> PNodeRunner object

Type: Dict[str, PNodeRunner]

path_dict¶

Dictionary mapping node_name –> path for each node having a path parameter

Type: Dict[str, str]

data_dir¶

Path to the directory containing the data. Used to resolve relative paths.

Type: pathlib.Path, default=””

__init__(pipeline: gnomon.pipeline.gnomonPipeline, data_dir: str = '')¶

Parameters

pipeline (gnomon.pipeline.gnomonPipeline) –
data_dir (str) –

update_node_inputs(node_name: str)¶

Updates the inputs of the node from connected nodes.

Parameters: node_name (str) – Name of the node updated.

run()¶

Runs the pipeline.

Works as follows :

Gets the equirequirement groups from the gnomonPipeline object.
Sets the path from path_dict for nodes having a path parameter.
Run each source node in threads then await those threads.
For each equirequirement node group:
1. Update input of each node of the group
2. Run each node of the group inside their respective threads.
3. Await those threads

update_paths(path_dict: Dict[str, str])¶

Updates the attribute path_dict.

Parameters: path_dict (Dict[str, str]) – Dictionary mapping a node_name to a path it should use. Useful to configure reader or writer nodes.

gnomon.utils.pipelines.load_pipeline(path: str, data_dir: str = '')¶

Load a pipeline from path and returns a PipelineRunner object

Parameters

path (str) – Path to pipeline file
data_dir (str) – path to the data directory
ios (Dict[str, str]) – dictionary of [input_output name : path] to apply the pipeline to new datas

Return type

PipelineRunner object

gnomon.utils.pipelines.run_pipeline(path: str, data_dir: str = '', ios: Optional[Dict[str, str]] = None)¶

run a pipeline from a path :param path: Path to pipeline file :type path: str :param data_dir: path to the data directory :type data_dir: str :param ios: dictionary of [input_output name : path] to apply the pipeline to new datas :type ios: Dict[str, str]

Parameters

path (str) –
data_dir (str) –
ios (Optional[Dict[str, str]]) –

Pipeline file example¶

{
    "binarization": {
        "description": "",
        "inputs": {
            "initialization": null,
            "input": "imageFilter2 -> output"
        },
        "name": "binarization",
        "outputs": [
            "output"
        ],
        "parameters": {
            "greater_or_lower": {
                "doc": "Channel on which to apply the algorithm",
                "index": 1,
                "label": "> or <",
                "type": "dtkCoreParameterInList<QString>",
                "values": [
                    "greater >",
                    "lower <"
                ]
            },
            "threshold": {
                "decimals": 0,
                "doc": "Image is true where intensity > threshold",
                "label": "Threshold",
                "max": 255,
                "min": 0,
                "type": "dtkCoreParameterNumeric<qlonglong,void>",
                "value": 11
            }
        },
        "plugin_group": "binaryImageFromImage",
        "plugin_name": "binarization",
        "plugin_version": "0.1.0"
    },
    "binaryImageWriter1": {
        "description": "",
        "inputs": {
            "binaryImage": "binarization -> output"
        },
        "name": "binaryImageWriter1",
        "path": "/Users/username/data/out.inr",
        "plugin_group": "binaryImageWriter",
        "plugin_name": "binaryImageWriter",
        "plugin_version": "0.1.0"
    },
    "description": "",
    "fileFormatVersion": "0.0.1",
    "gnomonVersion": "0.20.0",
    "imageFilter1": {
        "description": "",
        "inputs": {
            "input": "imageReader1 -> image",
            "mask": null
        },
        "name": "imageFilter1",
        "outputs": [
            "output"
        ],
        "parameters": {
            "K": {
                "decimals": 1,
                "doc": "Proportional to the cell walls width",
                "label": "Const K",
                "max": 0.3,
                "min": 0.1,
                "type": "dtkCoreParameterNumeric<double,void>",
                "value": 0.2
            },
            "gamma": {
                "decimals": 1,
                "doc": "Amount of gamma correction on the image signal (add contrast in low intensties)",
                "label": "Gamma",
                "max": 15,
                "min": 0,
                "type": "dtkCoreParameterNumeric<double,void>",
                "value": 1
            },
            "nbIter": {
                "decimals": 0,
                "doc": "Number of iterations",
                "label": "Iteration",
                "max": 255,
                "min": 0,
                "type": "dtkCoreParameterNumeric<qlonglong,void>",
                "value": 15
            },
            "sigma": {
                "decimals": 2,
                "doc": "Standard deviation (in µm) of the Gaussian kernel used to smooth signal",
                "label": "Gaussian Sigma",
                "max": 10,
                "min": 0,
                "type": "dtkCoreParameterNumeric<double,void>",
                "value": 0.5
            },
            "upDir": {
                "decimals": 0,
                "doc": "Diffusion tensor is updated every upDiter iteration",
                "label": "Tensor",
                "max": 255,
                "min": 0,
                "type": "dtkCoreParameterNumeric<qlonglong,void>",
                "value": 10
            }
        },
        "plugin_group": "imageFilter",
        "plugin_name": "anisotropic3dImageEnhancement",
        "plugin_version": "0.1.0"
    },
    "imageFilter2": {
        "description": "",
        "inputs": {
            "input": "imageFilter1 -> output",
            "mask": null
        },
        "name": "imageFilter2",
        "outputs": [
            "output"
        ],
        "parameters": {
            "pc_max": {
                "decimals": 2,
                "doc": "Upper percentile use to define the lower range of the input image for contrast stretching",
                "label": "Max percentile",
                "max": 255,
                "min": 0,
                "type": "dtkCoreParameterNumeric<double,void>",
                "value": 99
            },
            "pc_min": {
                "decimals": 2,
                "doc": "Lower percentile use to define the lower range of the input image for contrast stretching",
                "label": "Min percentile",
                "max": 255,
                "min": 0,
                "type": "dtkCoreParameterNumeric<double,void>",
                "value": 2
            }
        },
        "plugin_group": "imageFilter",
        "plugin_name": "globalContrastStretch",
        "plugin_version": "0.1.0"
    },
    "imageReader1": {
        "description": "",
        "inputs": {
        },
        "name": "imageReader1",
        "outputs": [
            "image"
        ],
        "path": "/Users/username/data/0hrs_plant1_trim-acylYFP_small.inr",
        "plugin_group": "imageReader",
        "plugin_name": "imageReaderTimagetk",
        "plugin_version": "0.1.0"
    },
    "input": [
        {
            "imageReader1_path": "imageReader1 -> path"
        }
    ],
    "name": "test_pipeline1",
    "output": [
        {
            "binaryImageWriter1_path": "binaryImageWriter1 -> path"
        }
    ],
    "type": "pipeline"
}

Table of Contents

Running a pipeline¶

Run pipeline with command line¶

Python API¶

Examples¶

API documentation¶

Pipeline file example¶