oc-doc/docs/WP/workflow_design.md

Workflow design rules

1. A data resource may be directly linked to a processing
2. A processing resource is linked always linked to the next processing resource.
3. If two processing resources need to exchange file data, they need to be connected to the same file storage(s)
4. A processing shall be linked to a computing resource

Data - Resource link

A data resource may be linked to a processing resource. The processing resource shall be compatible with the data resource format and API.

Example 1 : Trivial download

For a simple example :

• the data resource will provide an http url to download a file.
• the processing resource is the simple curl command that will download the file in the current computing resource

Example 2 : Advanced download processing resource

For a more specific example :

• the data resource is a complex data archive
• the processing resource is a complex download component that can be configured to retrieve specific data or datasets from the archive

Processing - Processing link

Dependant processings must be graphically linked, those links allow build the workflow's acyclic diagram.

Processing - Storage link

A processing may be linked to one or several storage resources.

Basic storage resource types

Storage resource types generally require a list source - destination information tha describe reaw/write operations. This information is associated to the link between the processing and the storage resources.

In the case of a write to storage operation :

• the source information specifies the local path/filename in the container where the file is created by the processing.
• the destination information contains the url/path/filename where the file shall be stored

In the case of a read from storage operation :

• the source information specifies the url/path/filename where the file is stored
• the destination information contains local path/filename in the container where the file shall be created for the processing to use it.

Local cluster storage

The generated Argo workflow defines a local storage available to all containers in the current cluster This storage is available from every container under the path defined in the $LOCALSTORAGE environment varlable. On this special storage, as it is mounted in all container, the source - destination information is implicit. Any data can be read or written directly.

S3 type storages

Several S3 compatible storage may be used in a workflow

• OCS3 : The global MinIO deployed in the local OpenCloud instance
• Generic S3 : Any external accessible S3 compatible service
• WFS3 : An internal MinIO instance deployed for the workflow duration, that instance might be exposed outside the current cluster

Custom storage types

Processing - Computing link

A processing shall be connected to a computing link.