From 33bfe79f66b2c6422b6f089479f4723d4667f77d Mon Sep 17 00:00:00 2001 From: ycc Date: Mon, 25 Nov 2024 15:45:05 +0100 Subject: [PATCH] doc upt --- docs/access_control.md | 16 ++++++++++ docs/development.md | 65 ++++++++++++++++++++++++++++++++++++++++ req/#1/oc-own_usage.md | 11 +++++++ req/#1/oc-peers_usage.md | 11 +++++++ req/#1/oc-rates.md | 13 ++++++++ req/#1/oc-sync.md | 0 req/#2/oc-accounting.md | 8 +++++ req/#2/oc-currencies.md | 4 +++ wbs/wbs.puml | 8 ++--- 9 files changed, 132 insertions(+), 4 deletions(-) create mode 100644 docs/access_control.md create mode 100644 docs/development.md create mode 100644 req/#1/oc-own_usage.md create mode 100644 req/#1/oc-peers_usage.md create mode 100644 req/#1/oc-rates.md create mode 100644 req/#1/oc-sync.md create mode 100644 req/#2/oc-accounting.md create mode 100644 req/#2/oc-currencies.md diff --git a/docs/access_control.md b/docs/access_control.md new file mode 100644 index 0000000..82ed887 --- /dev/null +++ b/docs/access_control.md @@ -0,0 +1,16 @@ +# General architecture + +Each OpenCloud instance will provide an OpenId interface. This interface may be connected to an existing LDAP Server or a dedicated one. + +# User rights definition + +Each OpenCloud instance will manage it's users and their permissions : +* a user has permition to start a distributed workflow in using remote peers +* a user has administrative rights and may change the service exchenge rates +* a user is limited to view financial information on the instance +* a user belongs to a group (that may represent a project, a department,...) + +# Authentication process + +Each OpenCloud peer will accept a company as a whole. +Upon user connection, it will receive user rights form the origninating OpenId connect server and apply them. ex: specific pricing for a group (company agreement, project agreement, ...) \ No newline at end of file diff --git a/docs/development.md b/docs/development.md new file mode 100644 index 0000000..29c0251 --- /dev/null +++ b/docs/development.md @@ -0,0 +1,65 @@ +# OpenCloud base stack + +OpenCloud relies on a micro services architecture. +Each component could be developed using specific technologies. +However, in order to preserve product consistency and ease maintenance activities, we strongly encourage using the following technological stacks. + +## Web services + +Web services are developped in Go using Beego stack + +### Environment setup + +When using pricate repositories like the OpenCloud git forge, you should define it as a private repository + + export GOPRIVATE=cloud.o-forge.io + +The Beego stack provides the bee cli tool to ease building process : + + go get github.com/beego/bee/v2@latest + +### Project initialization + +New component creation + + go mod init oc-mycomponent + +Refer to other services component main.go file to write a consitent initialisation process + +### Project build + +In order to build the software : + + bee run -downdoc=true -gendoc=true + +The -downdoc=true -gendoc=true will automatically generate swagger documentation in teh /swagger path + +If default Swagger page is displayed instead of tyour api, change url in swagger/index.html file to : + + url: "swagger.json" + +If annotations are modified without any code changed, a rebuild might not reflect the changes. +To force routing information update : + + bee generate routers + +## GUI compoenents + +The GUI are developped using Flutter framework + +### Environment setup + +* Install Flutter framework +* Install Android Studio +* In "Tools"->"SDK Manager"->"Apparenace & Behaviour/System Settings/Android SDK", go to "SDK tools" and tick the "Android SDK command line tools" +* Run flutter doctor commmand and follow instructions to accept SDK licenses +* Add Vscode flutter plugin and use Vscode Command palette to create a Flutter project +* Also set the target Device uring command Palette + +### Project build + +Depending on your target platform : + + flutter build web + flutter build linux + flutter build windows \ No newline at end of file diff --git a/req/#1/oc-own_usage.md b/req/#1/oc-own_usage.md new file mode 100644 index 0000000..a7ff7c5 --- /dev/null +++ b/req/#1/oc-own_usage.md @@ -0,0 +1,11 @@ +# Description + +The oc-own_usage service will monitor and store the consumption data for all the workflows initiated from our own OpenCloud instance. +The collected data will be accessible both in real time and for past workflows for the user that sent them and the allowed profiles in the current OpenCloud instance +Collected data will also be used to prevent abusive peers billing after a workflow execution. + +# Requirements + +* A user sending a workflow in a distributed environment shall be able to monitor it's resource consumption +* The resource consumption shall be available in both techical data (Storage/time,RAM/time,CPU/time) and monetary (coins / currency) +* The consumption information may filtered by peer, getting the full consumption data for each peer involved in the current workflow. This information may be use by the user to analyze/optimize its future workflows. it will aslo be used by the accounting system to check consistency between peers billing and monitored consumption. diff --git a/req/#1/oc-peers_usage.md b/req/#1/oc-peers_usage.md new file mode 100644 index 0000000..3782dc5 --- /dev/null +++ b/req/#1/oc-peers_usage.md @@ -0,0 +1,11 @@ +# Description + +The oc-peers_usage service will monitor and store the consumption data of all the peers workflows involving our own OpenCloud instance. +The collected data will be accessible both in real time and for monitoring the current OpenCloud instance workflows in order to perform peers billing. + +# Requirements + +* The resource consumption shall be available in both techical data (Storage/time,RAM/time,CPU/time) and monetary (coins / currency) +* The resource consumption shall be available to the user that started a workflow/donwloaded data from our instance for the related items (related workflow(s) and data) +* The complete resource consumtion for a peer/group(project) shall be available to users granted with a specific permission +* \ No newline at end of file diff --git a/req/#1/oc-rates.md b/req/#1/oc-rates.md new file mode 100644 index 0000000..b6a1fbf --- /dev/null +++ b/req/#1/oc-rates.md @@ -0,0 +1,13 @@ +# Description + +The oc-rates service define the applicable rates for services in our own OpenCloud instance +(data storage, RAM usage, CPU time, GPU time, HPC cluster execution, ...) +A default rate shall be defined for all public peers. +Peers/groups (project) having a specific agreement may benefit of custom rates + +# Requirements + +* An authorized user (specific permission) will be able to define default rates and specific peers rates. +* The default rates shall be accessible to everlonging to they user internal and external. +* The custom rates shall be only accessible to users belonging to the relevant peer +* \ No newline at end of file diff --git a/req/#1/oc-sync.md b/req/#1/oc-sync.md new file mode 100644 index 0000000..e69de29 diff --git a/req/#2/oc-accounting.md b/req/#2/oc-accounting.md new file mode 100644 index 0000000..50dc3f4 --- /dev/null +++ b/req/#2/oc-accounting.md @@ -0,0 +1,8 @@ +# Description + +The oc-acounting service will aggregate billing information for each peer in a daily(TBC) basis. +Payment will b + +# Requirements + +* \ No newline at end of file diff --git a/req/#2/oc-currencies.md b/req/#2/oc-currencies.md new file mode 100644 index 0000000..dafecc0 --- /dev/null +++ b/req/#2/oc-currencies.md @@ -0,0 +1,4 @@ +# Description + +The oc-currencies service is able to convert oc-coins current value to or from main currencies (€/$) +It allow to display real currency total cost in all user interfaces, and to update product with a real currency fixes price to the fluctuating oc-coin value \ No newline at end of file diff --git a/wbs/wbs.puml b/wbs/wbs.puml index c85f91c..cc9e836 100644 --- a/wbs/wbs.puml +++ b/wbs/wbs.puml @@ -8,8 +8,8 @@ ---[#orange] iteration 2 ---[#lightgreen] Thales proposed scopes -- OC-Catalog ----[#orange] authentication => RBAC ----[#orange] algo metadata ingress, res min max) +---[#orange] (60%) authentication => RBAC +---[#orange] (50%) algo metadata ingress, res min max) --- (OK) new resource type : workflow ---[#lightyellow] (OK) split catalog - workspace - workflow ---[#lightblue] algo metadata input output description @@ -19,7 +19,7 @@ -- OC-Scheduler / OC-Monitor ? ---[#lightyellow] (OK) automatically starting workflows --- (OK) monitoring workflows ----[#orange] workflow to service generation (deployment yaml) +---[#orange] (60%) workflow to service generation (deployment yaml) --- workflow to other targets (slurm) ++ OC-Search => Front +++[#lightblue] algo input/output description @@ -36,7 +36,7 @@ ++[#lightgreen] OC-Deploy +++[#lightyellow] (OK) repo +++[#yellow] deploy OC services -+++[#orange] deploy demo instance ++++[#orange] (docker 80% / native 40%) deploy demo instance +++ manage local cluster +++ partner sandboxing +++[#lightblue] network sandboxing