71 lines
2.8 KiB
Markdown
71 lines
2.8 KiB
Markdown
|
## Workflow design rules
|
||
|
|
|
||
|
|
1. A data resource may be directly linked to a processing
|
||
|
|
1. A processing resource is linked always linked to the next processing resource.
|
||
|
|
1. If two processing resources need to exchange file data, they need to be connected to the same file storage(s)
|
||
|
|
1. A processing shall be linked to a computing resource
|
||
|
|
|
||
|
|
|
||
|
|
### Data - Resource link
|
||
|
|
|
||
|
|
A data resource may be linked to a processing resource.
|
||
|
|
The processing resource shall be compatible with the data resource format and API.
|
||
|
|
|
||
|
|
#### Example 1 : Trivial download
|
||
|
|
For a simple example :
|
||
|
|
* the data resource will provide an http url to download a file.
|
||
|
|
* the processing resource is the simple curl command that will download the file in the current computing resource
|
||
|
|
|
||
|
|
|
||
|
|
#### Example 2 : Advanced download processing resource
|
||
|
|
|
||
|
|
For a more specific example :
|
||
|
|
* the data resource is a complex data archive
|
||
|
|
* the processing resource is a complex download component that can be configured to retrieve specific data or datasets from the archive
|
||
|
|
|
||
|
|
|
||
|
|
### Processing - Processing link
|
||
|
|
|
||
|
|
Dependant processings must be graphically linked, those links allow build the workflow's acyclic diagram.
|
||
|
|
|
||
|
|
|
||
|
|
### Processing - Storage link
|
||
|
|
|
||
|
|
A processing may be linked to one or several storage resources.
|
||
|
|
|
||
|
|
#### Basic storage resource types
|
||
|
|
|
||
|
|
Storage resource types generally require a list source - destination information tha describe reaw/write operations.
|
||
|
|
This information is associated to the link between the processing and the storage resources.
|
||
|
|
|
||
|
|
*In the case of a write to storage operation :*
|
||
|
|
* the source information specifies the local path/filename in the container where the file is created by the processing.
|
||
|
|
* the destination information contains the url/path/filename where the file shall be stored
|
||
|
|
|
||
|
|
*In the case of a read from storage operation :*
|
||
|
|
* the source information specifies the url/path/filename where the file is stored
|
||
|
|
* the destination information contains local path/filename in the container where the file shall be created for the processing to use it.
|
||
|
|
|
||
|
|
##### Local cluster storage
|
||
|
|
|
||
|
|
The generated Argo workflow defines a local storage available to all containers in the current cluster
|
||
|
|
This storage is available from every container under the path defined in the $LOCALSTORAGE environment varlable.
|
||
|
|
On this special storage, as it is mounted in all container, the source - destination information is implicit.
|
||
|
|
Any data can be read or written directly.
|
||
|
|
|
||
|
|
##### S3 type storages
|
||
|
|
|
||
|
|
Several S3 compatible storage may be used in a workflow
|
||
|
|
* OCS3 : The global MinIO deployed in the local OpenCloud instance
|
||
|
|
* Generic S3 : Any external accessible S3 compatible service
|
||
|
|
* WFS3 : An internal MinIO instance deployed for the workflow duration, that instance might be exposed outside the current cluster
|
||
|
|
|
||
|
|
##### Custom storage types
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
### Processing - Computing link
|
||
|
|
|
||
|
|
A processing shall be connected to a computing link.
|
||
|
|
|