All the Ansible playbooks used to deploy k3s, argo server, admiralty and minio

ansible/Minio/README.md

@@ -0,0 +1,111 @@
+# MinIO
+
+## Deploy Minio 
+
+This playbook installs MinIO on a Kubernetes cluster using Helm and retrieves necessary credentials and access information.
+
+### Variables
+| Variable | Description |
+|----------|-------------|
+| `user_prompt` | SSH user to execute commands |
+| `host_name_prompt` | Hostname of the target machine |
+| `memory_req` | Memory allocation for MinIO (`2Gi` by default) |
+| `storage_req` | Storage allocation for MinIO (`20Gi` by default) |
+
+### Steps Executed
+1. Install necessary Python libraries.
+2. Check if Helm is installed and install it if not present.
+3. Add and update the MinIO Helm repository.
+4. Deploy MinIO using Helm if it is not already running.
+5. Retrieve the MinIO credentials (root user and password).
+6. Retrieve the MinIO UI console external IP and API internal IP.
+7. Display login credentials and connection details.
+
+### Running the Playbook
+```sh
+ansible-playbook -i inventory deploy_minio.yml --extra-vars "user_prompt=your-user host_name_prompt=your-host"
+```
+
+## Setting up MinIO access
+
+/!\ This part can be automated with this **[ansible playbook](https://github.com/pi-B/ansible-oc/blob/main/setup_minio_admiralty.yml)** which is designed to create ressources in a Argo-Workflows/Admiralty combo.
+
+/!\ If you still want to setup the host manually **and** aim to use admiralty, give the ressources an **unique name** and be sure to make this uniqueness accessible (in an environment variable, in a conf file...)
+
+- With the output of the last tasks, create a secret in argo namespace to give access to the minio API. We need to use the `create` verb because apply creates a non-functionning secret 
+
+```bash
+kubectl create secret -n <name of your argo namespace> generic argo-artifact-secret \
+    --from-literal=access-key=<your access key> \
+    --from-literal=secret-key=<your secret key>
+```
+
+- Create a ConfigMap, which will be used by argo to create the S3 artifact, the content must match the one from the previously created secret
+
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  # If you want to use this config map by default, name it "artifact-repositories".
+  name: artifact-repositories
+  # annotations:
+  #   # v3.0 and after - if you want to use a specific key, put that key into this annotation.
+  #   workflows.argoproj.io/default-artifact-repository: oc-s3-artifact-repository
+data:
+  oc-s3-artifact-repository: |
+    s3:
+      bucket: oc-bucket
+      endpoint: [ retrieve cluster with kubectl get service argo-artifacts -o jsonpath="{.spec.clusterIP}" ]:9000
+      insecure: true
+      accessKeySecret: 
+        name: argo-artifact-secret
+        key: access-key
+      secretKeySecret:
+        name: argo-artifact-secret
+        key: secret-key
+
+```
+
+## Ansible Playbook setup MinIO
+
+### Purpose
+This playbook sets up MinIO to work with Argo Workflows, including creating the required buckets and secrets.
+
+### Variables
+| Variable | Description |
+|----------|-------------|
+| `user_prompt` | SSH user to execute commands |
+| `uuid_prompt` | Unique identifier for the Argo secret |
+| `argo_namespace` | Kubernetes namespace for Argo (`argo` by default) |
+
+### Steps Executed
+1. Install necessary dependencies.
+2. Download and configure MinIO Client (`mc`).
+3. Retrieve MinIO credentials (root user and password).
+4. Configure `mc` to connect to MinIO.
+5. Create a new S3 bucket (`oc-bucket`).
+6. Generate a new access key and secret key for MinIO.
+7. Retrieve the MinIO API cluster IP.
+8. Create a Kubernetes Secret to store MinIO credentials.
+9. Create a Kubernetes ConfigMap for MinIO artifact repository configuration.
+
+### Running the Playbook
+```sh
+ansible-playbook -i inventory setup_minio_resources.yml --extra-vars "user_prompt=your-user uuid_prompt=unique-id"
+```
+
+---
+
+## Expected Output
+Upon successful execution, you should see:
+- MinIO deployed and accessible.
+- MinIO UI console credentials displayed.
+- MinIO bucket (`oc-bucket`) created.
+- Secrets and ConfigMaps properly configured in Kubernetes.
+
+For any issues, check Ansible logs and validate configurations manually using:
+```sh
+kubectl get pods -n default
+kubectl get secrets -n argo
+kubectl get configmaps -n argo
+```

ansible/Minio/deploy_minio.yml

@@ -0,0 +1,134 @@
+- name: Deploy MinIO
+  hosts: all:!localhost
+  user: "{{ user_prompt }}"
+  vars:
+    host_name: "{{ host_name_prompt }}"
+    memory_req: "2Gi"
+    storage_req: "20Gi"
+  environment:
+    KUBECONFIG: /home/{{ user_prompt }}/.kube/config
+
+  tasks:
+    - name: Install yaml library for python
+      become: true
+      ansible.builtin.package:
+        name: ansible
+        state: present
+
+    - name: Check if Helm does exist
+      ansible.builtin.command: 
+        cmd: which helm
+      register: result_which
+      failed_when: result_which.rc not in [ 0, 1 ]
+
+    - name: Install helm
+      when: result_which.rc == 1
+      block:
+        - name: Download helm from source
+          ansible.builtin.get_url:
+            url: https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
+            dest: ./get_helm.sh
+            mode: 0700
+
+        - name: Launch helm install script
+          become: true
+          ansible.builtin.shell:
+            cmd: |
+              ./get_helm.sh
+
+    - name: Test if MinIO is already installed
+      ansible.builtin.shell:
+        cmd : helm repo list | grep 'https://charts.min.io/'
+      register: minio_charts
+      failed_when: minio_charts.rc not in [0,1]
+
+    - name: Add helm repo MinIO
+      kubernetes.core.helm_repository:
+        repo_url: https://charts.min.io/
+        repo_state: present
+        repo_name: minio
+      when: minio_charts.rc == 1
+
+    - name: Update helm repo
+      ansible.builtin.command:
+        cmd : |
+          helm repo update
+      when: minio_charts.rc == 1
+
+    - name: Test is argo-artifact is already running
+      ansible.builtin.shell:
+        helm list | grep -w "argo-artifacts" | wc -l
+      register: argo_artifact_deployed
+      failed_when: argo_artifact_deployed.rc not in [ 0, 1 ]
+
+    - name: Initialize MinIO
+      when: argo_artifact_deployed.stdout == "0"
+      kubernetes.core.helm:
+        name: argo-artifacts
+        chart_ref: minio/minio
+        release_namespace: default
+        values:
+          service:
+            type: LoadBalancer
+          fullnameOverride: argo-artifacts
+          resources:
+            requests:
+              memory: "{{ memory_req }}"
+          replicas: 2
+          volumeClaimTemplates:
+            spec:
+              resources:
+                requests: "{{ storage_req }}" 
+          consoleService:
+            type: LoadBalancer 
+            # port: 9001
+        state: present
+
+    - name: Retrieve root user
+      ansible.builtin.shell:
+        cmd: |
+          kubectl get secret argo-artifacts --namespace default -o jsonpath="{.data.rootUser}"
+      register : user_encoded
+
+    - name: Decode root user 
+      ansible.builtin.shell:
+        cmd: |
+          echo {{ user_encoded.stdout }} | base64 -d
+      register: user
+
+    - name: Retrieve root password
+      ansible.builtin.shell:
+        cmd: |
+          kubectl get secret argo-artifacts --namespace default -o jsonpath="{.data.rootPassword}"
+      register : password_encoded
+
+
+    - name: Decode root password 
+      ansible.builtin.shell:
+        cmd: |
+          echo {{ password_encoded.stdout }} | base64 -d
+      register: password
+
+    - name: Retrieve console ip
+      ansible.builtin.shell:
+        cmd: |
+          kubectl get service argo-artifacts-console -o jsonpath="{.status.loadBalancer.ingress[0].ip}"
+      register : ip_console
+
+    - name: Retrieve API internal ip
+      ansible.builtin.shell:
+        cmd: |
+          kubectl get service argo-artifacts -o jsonpath="{.spec.clusterIP}"
+      register : ip_api
+
+    - name: Display info
+      debug: 
+        msg :
+          "
+          MinIO UI console info 
+            external IP GUI : {{ ip_console.stdout }}
+            user : {{ user.stdout }}
+            password : {{ password.stdout }}
+
+          IP API : {{ ip_api.stdout }}
+          "

ansible/Minio/secret-cnes.yaml

@@ -0,0 +1,8 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: cnes-secrets
+type: Opaque
+stringData:
+  weather-api: 1d2b4ad68a4375388e64f5353d33186c
+  era-5: 3e8457b6-f5eb-4405-a09c-78403a14c4d1

ansible/Minio/setup_minio_oc_bucket.yml

@@ -0,0 +1,142 @@
+- name: Installation k3s
+  hosts: all:!localhost
+  user: "{{ user_prompt }}"
+  gather_facts: true
+  become_method: sudo
+  vars:
+    - argo_namespace: argo
+    - MC_PATH: $HOME/minio-binaries
+    - MINIO_NAME: my-minio
+    - UUID: "{{ uuid_prompt }}"
+  environment:
+    - KUBECONFIG: /home/{{ user_prompt }}/.kube/config
+  tasks:
+
+  - name: Install necessary packages
+    become: true 
+    package:
+      name: 
+        - python3-kubernetes
+        - python3-jmespath
+      state: present
+
+  - name: Create destination directory
+    file:
+      path: $HOME/minio-binaries
+      state: directory
+      mode: '0755'
+
+  - name: Install mc
+    ansible.builtin.get_url:
+      url: "https://dl.min.io/client/mc/release/linux-amd64/mc"
+      dest: $HOME/minio-binaries/mc
+      mode: +x
+      headers:
+        Content-Type: "application/json"
+
+  - name: Add mc to path
+    ansible.builtin.lineinfile:
+      path: $HOME/.bashrc
+      line: export PATH=$PATH:$HOME/minio-binaries
+
+  - name: Is mc already set up for the local minio
+    ansible.builtin.shell:
+      cmd: |
+        "{{ MC_PATH }}"/mc admin info {{ MINIO_NAME }}
+    register: minio_info
+    failed_when: minio_info.rc not in [0,1]
+
+  - name: Retrieve root user
+    ansible.builtin.shell:
+      cmd: |
+        kubectl get secrets argo-artifacts -o jsonpath="{.data.rootUser}" | base64 -d -
+    register: user
+    when: minio_info.rc == 1
+
+  - name: Retrieve root password
+    ansible.builtin.shell:
+      cmd: |
+        kubectl get secret argo-artifacts --namespace default -o jsonpath="{.data.rootPassword}" | base64 -d -
+    register : password
+    when: minio_info.rc == 1
+
+  - name: Set up MinIO host in mc
+    ansible.builtin.shell:
+      cmd: |
+        "{{ MC_PATH }}"/mc alias set {{ MINIO_NAME }} http://127.0.0.1:9000 '{{ user.stdout }}' '{{ password.stdout }}'
+    failed_when: user.stdout == "" or password.stdout == ""
+    when: minio_info.rc == 1
+
+  - name: Does oc-bucket already exist
+    ansible.builtin.shell:
+      cmd: |
+        "{{ MC_PATH }}"/mc ls my-minio | grep -q oc-bucket
+    register: bucket_exists
+    failed_when: bucket_exists.rc not in [0,1]
+
+  - name: Create oc-bucket
+    ansible.builtin.shell:
+      cmd: |
+        "{{ MC_PATH }}"/mc mb {{ MINIO_NAME }}/oc-bucket
+    when: bucket_exists.rc == 1
+
+  - name: Run mc admin accesskey create command
+    ansible.builtin.shell:
+      cmd: |
+        {{ MC_PATH }}/mc admin accesskey create --json {{ MINIO_NAME }}
+    register: minio_output
+    changed_when: false  # Avoid marking the task as changed every time
+
+
+  - name: Parse JSON output
+    set_fact:
+      access_key: "{{ minio_output.stdout | from_json | json_query('accessKey') }}"
+      secret_key: "{{ minio_output.stdout | from_json | json_query('secretKey') }}"
+
+  - name: Retrieve cluster IP for minio API
+    ansible.builtin.shell:
+      cmd: |
+        kubectl get service argo-artifacts -o jsonpath="{.spec.clusterIP}"
+    register: minio_cluster_ip
+
+  - name: Create the minio secret in argo namespace
+    kubernetes.core.k8s:
+      state: present
+      namespace: '{{ argo_namespace }}'
+      name: "{{ UUID }}-argo-artifact-secret"
+      definition:
+        apiVersion: v1
+        kind: Secret
+        type: Opaque
+        stringData:
+          access-key: '{{ access_key }}'
+          secret-key: '{{ secret_key }}'
+
+
+  - name: Create the minio secret in argo namespace
+    kubernetes.core.k8s:
+      state: present
+      namespace: '{{ argo_namespace }}'
+      definition:
+        apiVersion: v1
+        kind: ConfigMap
+        metadata:
+          name: artifact-repositories
+        data:
+          oc-s3-artifact-repository: |
+            s3:
+              bucket: oc-bucket
+              endpoint: {{ minio_cluster_ip.stdout }}:9000
+              insecure: true
+              accessKeySecret:
+                name: "{{ UUID }}-argo-artifact-secret"
+                key: access-key
+              secretKeySecret:
+                name: "{{ UUID }}-argo-artifact-secret"
+                key: secret-key
+
+    # ansible.builtin.shell:
+    #   cmd: |
+    #     kubectl create secret -n '{{ argo_namespace }}' generic argo-artifact-secret \
+    #     --from-literal=access-key='{{ access_key }}' \
+    #     --from-literal=secret-key='{{ secret_key }}'