From 5f24777fe8c1f0145e999d4bba9876c25a8f843d Mon Sep 17 00:00:00 2001 From: pmembari Date: Thu, 20 Feb 2025 11:08:59 +0100 Subject: [PATCH 01/61] =?UTF-8?q?=F0=9F=93=9A=20docs:=20edit=20typo?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index fcb0ba9..01d7125 100644 --- a/README.md +++ b/README.md @@ -71,7 +71,7 @@ cd mastering-app-package skaffold dev ``` -Wait for the deployment to stablize (1-2 minutes) and the open your browser on the link printed, usually http://127.0.0.1:8000. +Wait for the deployment to stablize (1-2 minutes) and then open your browser on the link printed, usually http://127.0.0.1:8000. The typical output is: From b4683c8f549b53250e5be18db55278b96e1bb90d Mon Sep 17 00:00:00 2001 From: pmembari Date: Thu, 20 Feb 2025 11:10:16 +0100 Subject: [PATCH 02/61] =?UTF-8?q?=F0=9F=93=9A=20docs:=20adding=20docs=20fo?= =?UTF-8?q?r=20event=20driven=20installation?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- event-driven-with-argo/README.md | 118 ++++++++++++++++++++----------- 1 file changed, 75 insertions(+), 43 deletions(-) diff --git a/event-driven-with-argo/README.md b/event-driven-with-argo/README.md index d2cef86..8c314e8 100644 --- a/event-driven-with-argo/README.md +++ b/event-driven-with-argo/README.md @@ -1,57 +1,89 @@ - -``` -helm repo add argo https://argoproj.github.io/argo-helm -helm repo add localstack https://helm.localstack.cloud -``` +# Event driven with argo: +## Installation +- The user must start a Minikube cluster (you can use the `--driver` option to specify the container runtime) using the command below: -```python + ``` + minikube start --driver=docker + ``` -from os import environ -from redis import Redis -from time import sleep + Using `--driver=docker` runs Minikube inside a Docker container instead of a virtual machine, which provides several advantages. It ensures a faster and more lightweight Kubernetes cluster initialization. This approach is particularly useful for development and testing environments where quick deployment and easy cleanup. -stream_key = environ.get("STREAM", "STREAM") -producer = environ.get("PRODUCER", "user-1") +- Please add the following repositories to your Helm chart to ensure access to the required packages: + ```sh + helm repo add argo https://argoproj.github.io/argo-helm + helm repo add localstack https://helm.localstack.cloud + ``` -def connect_to_redis(): - hostname = environ.get("REDIS_HOSTNAME", "redis-service") - port = environ.get("REDIS_PORT", 6379) + Adding these repositories allows you to seamlessly install and manage **Argo** (a workflow orchestration tool) and **LocalStack** (a local AWS cloud emulator) using Helm. This ensures that your Kubernetes environment is properly configured for deploying workflows and simulating AWS services efficiently. +- Run the command below: + ``` + skaffold dev + ``` - return Redis(hostname, port, retry_on_timeout=True) +- Wait for the deployment to stablize (1-3 minutes) and then open your browser on the link printed, usually http://127.0.0.1:8000. +- You will see the code server is running. Follow the documentations under `doc` folder. -def send_event(redis_connection, aoi, reference): - count = 0 - try: - # TODO cloud events - # un-map the "data" wrt app package parameters - data = { - "producer": producer, - "href": reference, - } - resp = redis_connection.xadd(stream_key, data) - print(resp) - count += 1 +> Note: Here is an example of Redis' publisher which is discussed in detail in the repository you have on code server +> +>```python +> +>from os import environ +>from redis import Redis +>from time import sleep +> +>stream_key = environ.get("STREAM", "STREAM") +>producer = environ.get("PRODUCER", "user-1") +> +> +>def connect_to_redis(): +> hostname = environ.get("REDIS_HOSTNAME", "redis-service") +> port = environ.get("REDIS_PORT", 6379) +> +> return Redis(hostname, port, retry_on_timeout=True) +> +> +>def send_event(redis_connection, aoi, reference): +> count = 0 +> +> try: +> # TODO cloud events +> # un-map the "data" wrt app package parameters +> data = { +> "producer": producer, +> "href": reference, +> } +> resp = redis_connection.xadd(stream_key, data) +> print(resp) +> count += 1 +> +> except ConnectionError as e: +> print(f"ERROR REDIS CONNECTION: {e}") +> +> +>connection = connect_to_redis() +> +>references = [ +> "https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2A_10TFK_20210708_0_L2A", +> "https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_10TFK_20210713_0_L2A", +> "https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2A_10TFK_20210718_0_L2A", +> "https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2A_10TFK_20220524_0_L2A", +> "https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2A_10TFK_20220514_0_L2A", +> "https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2A_10TFK_20220504_0_L2A" +>] +> +>for reference in references: +> send_event(connection, aoi, reference) +> sleep(1) +>``` - except ConnectionError as e: - print(f"ERROR REDIS CONNECTION: {e}") +### **Troubleshooting** +If the **Code Server** is unreachable at `http://127.0.0.1:8000`, there may be an existing process occupying the port. You can terminate the process using the command below, allowing Kubernetes to automatically re-establish port forwarding for traffic on port `8000`: -connection = connect_to_redis() - -references = [ - "https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2A_10TFK_20210708_0_L2A", - "https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_10TFK_20210713_0_L2A", - "https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2A_10TFK_20210718_0_L2A", - "https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2A_10TFK_20220524_0_L2A", - "https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2A_10TFK_20220514_0_L2A", - "https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2A_10TFK_20220504_0_L2A" -] - -for reference in references: - send_event(connection, aoi, reference) - sleep(1) +```sh +sudo fuser -k 8000/tcp ``` \ No newline at end of file From 1142bcc8325d8a0da06c30fa97b11127ba624231 Mon Sep 17 00:00:00 2001 From: pmembari Date: Tue, 25 Feb 2025 11:02:30 +0100 Subject: [PATCH 03/61] =?UTF-8?q?=F0=9F=8E=89=20feat:=20init=20machine-lea?= =?UTF-8?q?rning-process?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- machine-learning-process/README.md | 0 .../charts/argo-cwl-runner/.helmignore | 23 ++ .../charts/argo-cwl-runner/Chart.yaml | 24 ++ .../argo-cwl-runner/files/user-settings.json | 15 + .../templates/argo-calrissian.yaml | 324 ++++++++++++++++++ .../argo-cwl-runner/templates/semaphore.yaml | 7 + .../templates/user-settings.yaml | 9 + .../charts/argo-cwl-runner/values.yaml | 2 + .../charts/event-driven/Chart.yaml | 6 + .../event-driven/templates/event-bus.yaml | 18 + .../templates/event-sensor-log.yaml | 16 + .../event-driven/templates/event-sensor.yaml | 46 +++ .../event-driven/templates/event-source.yaml | 13 + .../templates/local-stack-secret.yaml | 10 + .../event-driven/templates/role-binding.yaml | 11 + .../charts/event-driven/templates/roles.yaml | 52 +++ .../charts/event-driven/values.yaml | 4 + .../containers/stage-out/Dockerfile | 15 + .../containers/stage-out/README.md | 49 +++ .../containers/stage-out/app/__init__.py | 0 .../containers/stage-out/app/main.py | 94 +++++ .../containers/stage-out/app/stac.py | 77 +++++ .../containers/stage-out/app/usersettings.py | 63 ++++ .../containers/stage-out/requirements.txt | 5 + .../containers/stage-out/setup.cfg | 3 + .../containers/stage-out/setup.py | 12 + .../containers/validate-schema/Dockerfile | 5 + machine-learning-process/files/bash-login | 1 + machine-learning-process/files/bash-rc | 5 + machine-learning-process/files/init.sh | 25 ++ machine-learning-process/skaffold.yaml | 157 +++++++++ 31 files changed, 1091 insertions(+) create mode 100644 machine-learning-process/README.md create mode 100644 machine-learning-process/charts/argo-cwl-runner/.helmignore create mode 100644 machine-learning-process/charts/argo-cwl-runner/Chart.yaml create mode 100644 machine-learning-process/charts/argo-cwl-runner/files/user-settings.json create mode 100644 machine-learning-process/charts/argo-cwl-runner/templates/argo-calrissian.yaml create mode 100644 machine-learning-process/charts/argo-cwl-runner/templates/semaphore.yaml create mode 100644 machine-learning-process/charts/argo-cwl-runner/templates/user-settings.yaml create mode 100644 machine-learning-process/charts/argo-cwl-runner/values.yaml create mode 100644 machine-learning-process/charts/event-driven/Chart.yaml create mode 100644 machine-learning-process/charts/event-driven/templates/event-bus.yaml create mode 100644 machine-learning-process/charts/event-driven/templates/event-sensor-log.yaml create mode 100644 machine-learning-process/charts/event-driven/templates/event-sensor.yaml create mode 100644 machine-learning-process/charts/event-driven/templates/event-source.yaml create mode 100644 machine-learning-process/charts/event-driven/templates/local-stack-secret.yaml create mode 100644 machine-learning-process/charts/event-driven/templates/role-binding.yaml create mode 100644 machine-learning-process/charts/event-driven/templates/roles.yaml create mode 100644 machine-learning-process/charts/event-driven/values.yaml create mode 100644 machine-learning-process/containers/stage-out/Dockerfile create mode 100644 machine-learning-process/containers/stage-out/README.md create mode 100644 machine-learning-process/containers/stage-out/app/__init__.py create mode 100644 machine-learning-process/containers/stage-out/app/main.py create mode 100644 machine-learning-process/containers/stage-out/app/stac.py create mode 100644 machine-learning-process/containers/stage-out/app/usersettings.py create mode 100644 machine-learning-process/containers/stage-out/requirements.txt create mode 100644 machine-learning-process/containers/stage-out/setup.cfg create mode 100644 machine-learning-process/containers/stage-out/setup.py create mode 100644 machine-learning-process/containers/validate-schema/Dockerfile create mode 100644 machine-learning-process/files/bash-login create mode 100644 machine-learning-process/files/bash-rc create mode 100644 machine-learning-process/files/init.sh create mode 100644 machine-learning-process/skaffold.yaml diff --git a/machine-learning-process/README.md b/machine-learning-process/README.md new file mode 100644 index 0000000..e69de29 diff --git a/machine-learning-process/charts/argo-cwl-runner/.helmignore b/machine-learning-process/charts/argo-cwl-runner/.helmignore new file mode 100644 index 0000000..0e8a0eb --- /dev/null +++ b/machine-learning-process/charts/argo-cwl-runner/.helmignore @@ -0,0 +1,23 @@ +# Patterns to ignore when building packages. +# This supports shell glob matching, relative path matching, and +# negation (prefixed with !). Only one pattern per line. +.DS_Store +# Common VCS dirs +.git/ +.gitignore +.bzr/ +.bzrignore +.hg/ +.hgignore +.svn/ +# Common backup files +*.swp +*.bak +*.tmp +*.orig +*~ +# Various IDEs +.project +.idea/ +*.tmproj +.vscode/ diff --git a/machine-learning-process/charts/argo-cwl-runner/Chart.yaml b/machine-learning-process/charts/argo-cwl-runner/Chart.yaml new file mode 100644 index 0000000..ac56d50 --- /dev/null +++ b/machine-learning-process/charts/argo-cwl-runner/Chart.yaml @@ -0,0 +1,24 @@ +apiVersion: v2 +name: argo-cwl-runner +description: A Helm chart for Kubernetes + +# A chart can be either an 'application' or a 'library' chart. +# +# Application charts are a collection of templates that can be packaged into versioned archives +# to be deployed. +# +# Library charts provide useful utilities or functions for the chart developer. They're included as +# a dependency of application charts to inject those utilities and functions into the rendering +# pipeline. Library charts do not define any templates and therefore cannot be deployed. +type: application + +# This is the chart version. This version number should be incremented each time you make changes +# to the chart and its templates, including the app version. +# Versions are expected to follow Semantic Versioning (https://semver.org/) +version: 0.1.0 + +# This is the version number of the application being deployed. This version number should be +# incremented each time you make changes to the application. Versions are not expected to +# follow Semantic Versioning. They should reflect the version the application is using. +# It is recommended to use it with quotes. +appVersion: "1.16.0" diff --git a/machine-learning-process/charts/argo-cwl-runner/files/user-settings.json b/machine-learning-process/charts/argo-cwl-runner/files/user-settings.json new file mode 100644 index 0000000..cf9eede --- /dev/null +++ b/machine-learning-process/charts/argo-cwl-runner/files/user-settings.json @@ -0,0 +1,15 @@ +{ + "S3": { + "Services": { + "results": { + "UrlPattern": "s3:\/\/results\/.*", + "Region": "us-east-1", + "AuthenticationRegion": "us-east-1", + "AccessKey": "test", + "SecretKey": "test", + "ServiceURL": "http://eoap-event-driven-localstack.eoap-event-driven.svc.cluster.local:4566", + "ForcePathStyle": "true" + } + } + } +} \ No newline at end of file diff --git a/machine-learning-process/charts/argo-cwl-runner/templates/argo-calrissian.yaml b/machine-learning-process/charts/argo-cwl-runner/templates/argo-calrissian.yaml new file mode 100644 index 0000000..1aa265a --- /dev/null +++ b/machine-learning-process/charts/argo-cwl-runner/templates/argo-calrissian.yaml @@ -0,0 +1,324 @@ +apiVersion: argoproj.io/v1alpha1 +kind: WorkflowTemplate +metadata: + name: argo-cwl-runner + annotations: + workflows.argoproj.io/version: ">= v3.3.0" + workflows.argoproj.io/description: | + This workflow template is a CWL runner for Argo Workflows. +spec: + entrypoint: calrissian-runner + parameters: + - name: parameters + description: "Parameters in JSON format" + value: "" + - name: cwl + description: "CWL document in JSON format" + value: "" + - name: max_ram + description: "Max RAM" + value: "8G" + - name: max_cores + description: "Max cores" + value: "8" + # this is the Workflow throttling to avoid overloading the cluster + # the configMap contains the number of concurrent Workflows allowed + synchronization: + semaphore: + configMapKeyRef: + name: semaphore-argo-cwl-runner + key: workflow + + # volumes + volumes: + - name: usersettings-vol + secret: + secretName: user-settings + - name: calrissian-wdir + persistentVolumeClaim: + claimName: calrissian-wdir + + # Workflow templates are used to define the Workflow steps + templates: + - name: calrissian-runner + # this is the entry point of the Workflow + inputs: + parameters: + - name: parameters + - name: cwl + - name: max_ram + - name: max_cores + outputs: + parameters: + - name: results + valueFrom: + parameter: {{ `"{{steps.get-results.outputs.parameters.calrissian-output}}"` }} + - name: log + valueFrom: + parameter: {{ `"{{steps.get-results.outputs.parameters.calrissian-stderr}}"` }} + - name: usage-report + valueFrom: + parameter: {{ `"{{steps.get-results.outputs.parameters.calrissian-report}}"` }} + - name: stac-catalog + valueFrom: + parameter: {{ `"{{steps.stage-out.outputs.parameters.stac-catalog}}"` }} + artifacts: + - name: tool-logs + from: {{ `"{{steps.get-results.outputs.artifacts.tool-logs}}"` }} + - name: calrissian-output + from: {{ `"{{steps.get-results.outputs.artifacts.calrissian-output}}"` }} + - name: calrissian-stderr + from: {{ `"{{steps.get-results.outputs.artifacts.calrissian-stderr}}"` }} + - name: calrissian-report + from: {{ `"{{steps.get-results.outputs.artifacts.calrissian-report}}"` }} + + steps: + # Workflow steps are defined here + - - name: cwl-prepare + template: cwl-prepare + arguments: + parameters: + - name: cwl + value: {{ `"{{inputs.parameters.cwl}}"` }} + - name: parameters + value: {{ `"{{inputs.parameters.parameters}}"` }} + + - - name: cwl-runner + template: calrissian-tmpl + arguments: + parameters: + - name: max_ram + value: {{ `"{{inputs.parameters.max_ram}}"` }} + - name: max_cores + value: {{ `"{{inputs.parameters.max_cores}}"` }} + + - - name: get-results + # these can run in parallel + # the same template is used for all the steps + template: get-results + arguments: + parameters: + - name: calrissian-output + value: "/calrissian/output.json" + - name: calrissian-stderr + value: "/calrissian/stderr.log" + - name: calrissian-report + value: "/calrissian/report.json" + + - name: stage-out + template: stage-out + arguments: + parameters: + - name: file_path + value: "/calrissian/output.json" + - name: bucket + value: "results" + - name: folder + value: {{ `"{{workflow.name}}-{{workflow.uid}}"` }} + + - name: cwl-prepare + # this steps prepares the CWL inputs + # needed by Calrissian + inputs: + parameters: + - name: cwl + - name: parameters + + script: + image: busybox:1.35.0 + resources: + requests: + memory: 1Gi + cpu: 1 + volumeMounts: + - name: calrissian-wdir + mountPath: /calrissian + env: [] + command: [ash] + source: | + #!/bin/ash + echo {{ `'{{inputs.parameters.cwl}}'` }} >> /calrissian/cwl.json + echo {{ `'{{inputs.parameters.parameters}}'` }} >> /calrissian/input.json + sleep 1 + + - name: calrissian-tmpl + # this step creates the Calrissian Job, Argo creates it as a Kubernetes Job + resource: + action: create + setOwnerReference: true + successCondition: status.succeeded > 0 + failureCondition: status.failed > 3 + manifest: | + apiVersion: batch/v1 + kind: Job + metadata: + generateName: calrissian-water-bodies-detection- + spec: + backoffLimit: 1 + activeDeadlineSeconds: 86400 + ttlSecondsAfterFinished: 120 + template: + metadata: + name: calrissian_pod + spec: + serviceAccountName: argo + containers: + - name: calrissian + image: ghcr.io/duke-gcb/calrissian/calrissian:0.16.0 + imagePullPolicy: IfNotPresent + command: + - calrissian + args: + - --debug + - --pod-serviceaccount + - argo + - --stdout + - /calrissian/output.json + - --stderr + - /calrissian/stderr.log + - --usage-report + - /calrissian/report.json + - --max-ram + - {{ `'{{inputs.parameters.max_ram}}'` }} + - --max-cores + - {{ `'{{inputs.parameters.max_cores}}'` }} + - --tmp-outdir-prefix + - /calrissian/tmp/ + - --outdir + - /calrissian/results/ + - --tool-logs-basepath + - /calrissian/logs + - "/calrissian/cwl.json" + - "/calrissian/input.json" + env: + - name: CALRISSIAN_POD_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name + - name: CALRISSIAN_DELETE_PODS + value: "true" + resources: + limits: + cpu: 2000m + memory: 2G + requests: + cpu: 1000m + memory: 1G + volumeMounts: + - mountPath: /calrissian + name: calrissian-wdir + readOnly: false + restartPolicy: Never + securityContext: + fsGroup: 0 + runAsGroup: 0 + runAsUser: 0 + terminationGracePeriodSeconds: 120 + volumes: + - name: calrissian-wdir + persistentVolumeClaim: + claimName: {{"{{workflow.name}}"}}-calrissian-wdir + readOnly: false + + inputs: + parameters: + - name: max_ram + - name: max_cores + outputs: + parameters: [] + artifacts: + - name: logs + path: /calrissian/logs + s3: + key: {{ `"{{workflow.name}}-{{workflow.uid}}-artifacts/calrissian-logs.tgz"` }} + + + + - name: get-results + # reads the files generated by Calrissian + inputs: + parameters: + - name: calrissian-output + - name: calrissian-stderr + - name: calrissian-report + outputs: + parameters: + - name: calrissian-output + valueFrom: + path: /tmp/calrissian-output.json + - name: calrissian-stderr + valueFrom: + path: /tmp/calrissian-stderr.txt + - name: calrissian-report + valueFrom: + path: /tmp/calrissian-report.json + artifacts: + - name: tool-logs + path: /calrissian/logs + s3: + key: {{ `"{{workflow.name}}-{{workflow.uid}}-artifacts/tool-logs.tgz"` }} + - name: calrissian-output + path: /tmp/calrissian-output.json + s3: + key: {{ `"{{workflow.name}}-{{workflow.uid}}-artifacts/calrissian-output.tgz"` }} + - name: calrissian-stderr + path: /tmp/calrissian-stderr.txt + s3: + key: {{ `"{{workflow.name}}-{{workflow.uid}}-artifacts/calrissian-stderr.tgz"` }} + - name: calrissian-report + path: /tmp/calrissian-report.json + s3: + key: {{ `"{{workflow.name}}-{{workflow.uid}}-artifacts/calrissian-report.tgz"` }} + script: + image: busybox:1.35.0 + resources: + requests: + memory: 1Gi + cpu: 1 + volumeMounts: + - name: calrissian-wdir + mountPath: /calrissian + command: [ash] + source: | + #!/bin/ash + cat "{{"{{inputs.parameters.calrissian-output}}"}}" > /tmp/calrissian-output.json + cat "{{"{{inputs.parameters.calrissian-stderr}}"}}" > /tmp/calrissian-stderr.txt + cat "{{"{{inputs.parameters.calrissian-report}}"}}" > /tmp/calrissian-report.json + + - name: stage-out + inputs: + parameters: + - name: file_path + - name: bucket + - name: folder + outputs: + parameters: + - name: stac-catalog + valueFrom: + path: /tmp/output + script: + image: {{ .Values.stageOutImage }} + resources: + requests: + memory: 1Gi + cpu: 1 + volumeMounts: + - name: calrissian-wdir + mountPath: /calrissian + - name: usersettings-vol + readOnly: true + mountPath: "/etc/secret" + command: [bash] + source: + #!/bin/bash + set -x + + cat /etc/secret/usersettings.json + + stage-out --stac-catalog $( cat {{"{{inputs.parameters.file_path}}"}} | jq -r .stac_catalog.path - ) --user-settings /etc/secret/usersettings.json --bucket "{{"{{inputs.parameters.bucket}}"}}" --subfolder "{{"{{inputs.parameters.folder}}"}}" + + res=$? + + echo "s3://{{"{{inputs.parameters.bucket}}"}}/{{"{{inputs.parameters.folder}}"}}/catalog.json" > /tmp/output + + exit $res \ No newline at end of file diff --git a/machine-learning-process/charts/argo-cwl-runner/templates/semaphore.yaml b/machine-learning-process/charts/argo-cwl-runner/templates/semaphore.yaml new file mode 100644 index 0000000..7751209 --- /dev/null +++ b/machine-learning-process/charts/argo-cwl-runner/templates/semaphore.yaml @@ -0,0 +1,7 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: semaphore-argo-cwl-runner +data: + workflow: "{{ .Values.semaphore }}" + \ No newline at end of file diff --git a/machine-learning-process/charts/argo-cwl-runner/templates/user-settings.yaml b/machine-learning-process/charts/argo-cwl-runner/templates/user-settings.yaml new file mode 100644 index 0000000..6bdf160 --- /dev/null +++ b/machine-learning-process/charts/argo-cwl-runner/templates/user-settings.yaml @@ -0,0 +1,9 @@ +# secret with the user-settings.json, it transforms the value yaml to json +apiVersion: v1 +kind: Secret +metadata: + name: user-settings + labels: + app: argo-water-bodies +data: + usersettings.json: {{ required "A valid .Values.userSettings entry required!" ( tpl ( .Values.userSettings | default ( .Files.Get "files/user-settings.json")) . | b64enc) }} \ No newline at end of file diff --git a/machine-learning-process/charts/argo-cwl-runner/values.yaml b/machine-learning-process/charts/argo-cwl-runner/values.yaml new file mode 100644 index 0000000..23d307c --- /dev/null +++ b/machine-learning-process/charts/argo-cwl-runner/values.yaml @@ -0,0 +1,2 @@ +stageOutImage: stageout +semaphore: 3 \ No newline at end of file diff --git a/machine-learning-process/charts/event-driven/Chart.yaml b/machine-learning-process/charts/event-driven/Chart.yaml new file mode 100644 index 0000000..b23913a --- /dev/null +++ b/machine-learning-process/charts/event-driven/Chart.yaml @@ -0,0 +1,6 @@ +apiVersion: v2 +name: event-driven +description: Event driven manifests including event bus and source/sensor +type: application +version: 0.1.0 +appVersion: "1.0.0" diff --git a/machine-learning-process/charts/event-driven/templates/event-bus.yaml b/machine-learning-process/charts/event-driven/templates/event-bus.yaml new file mode 100644 index 0000000..ca28d85 --- /dev/null +++ b/machine-learning-process/charts/event-driven/templates/event-bus.yaml @@ -0,0 +1,18 @@ +apiVersion: argoproj.io/v1alpha1 +kind: EventBus +metadata: + name: jetstream-event-bus +spec: + jetstream: + version: latest + replicas: {{ .Values.jetstreamReplicas }} + persistence: + storageClassName: standard + accessMode: ReadWriteOnce + volumeSize: {{ .Values.jetstreamVolumeSize }} + streamConfig: | + maxAge: 24h + settings: | + max_file_store: 1GB + startArgs: + - "-D" \ No newline at end of file diff --git a/machine-learning-process/charts/event-driven/templates/event-sensor-log.yaml b/machine-learning-process/charts/event-driven/templates/event-sensor-log.yaml new file mode 100644 index 0000000..dd9e8d5 --- /dev/null +++ b/machine-learning-process/charts/event-driven/templates/event-sensor-log.yaml @@ -0,0 +1,16 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Sensor +metadata: + name: acme-sentinel2-stream-source-logger +spec: + eventBusName: jetstream-event-bus + + dependencies: + - name: acme-sentinel2-stream-source-logger + eventSourceName: acme-sentinel2-stream-source + eventName: acme-sentinel2-stream-source-collected + triggers: + - template: + name: log-event + log: + intervalSeconds: 5 # Frequency of log updates \ No newline at end of file diff --git a/machine-learning-process/charts/event-driven/templates/event-sensor.yaml b/machine-learning-process/charts/event-driven/templates/event-sensor.yaml new file mode 100644 index 0000000..f22053c --- /dev/null +++ b/machine-learning-process/charts/event-driven/templates/event-sensor.yaml @@ -0,0 +1,46 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Sensor +metadata: + name: acme-water-bodies-detection-sensor +spec: + eventBusName: jetstream-event-bus + template: + serviceAccountName: argo + dependencies: + - name: acme-sentinel2-stream + eventSourceName: acme-sentinel2-stream-source + eventName: acme-sentinel2-stream-source-collected + transform: + jq: ".values.href = [ .values.href ]" + triggers: + - template: + name: water-bodies-detection-trigger + k8s: + group: argoproj.io + version: v1alpha1 + resource: workflows + operation: create + source: + resource: + apiVersion: argoproj.io/v1alpha1 + kind: Workflow + metadata: + generateName: water-bodies-detection- + namespace: eoap-event-driven + spec: + serviceAccountName: argo + entrypoint: main + arguments: + parameters: + - name: items + - name: aoi + value: "-121.399,39.834,-120.74,40.472" + - name: epsg + value: "EPSG:4326" + workflowTemplateRef: + name: water-bodies-detection + parameters: + - src: + dependencyName: acme-sentinel2-stream + dataKey: values.href + dest: spec.arguments.parameters.0.value diff --git a/machine-learning-process/charts/event-driven/templates/event-source.yaml b/machine-learning-process/charts/event-driven/templates/event-source.yaml new file mode 100644 index 0000000..4811ea3 --- /dev/null +++ b/machine-learning-process/charts/event-driven/templates/event-source.yaml @@ -0,0 +1,13 @@ +apiVersion: argoproj.io/v1alpha1 +kind: EventSource +metadata: + name: acme-sentinel2-stream-source +spec: + eventBusName: jetstream-event-bus + redisStream: + acme-sentinel2-stream-source-collected: + hostAddress: redis-service:6379 + db: 0 + streams: + - {{ .Values.streamName }} + consumerGroup: acme-sentinel2-stream-source \ No newline at end of file diff --git a/machine-learning-process/charts/event-driven/templates/local-stack-secret.yaml b/machine-learning-process/charts/event-driven/templates/local-stack-secret.yaml new file mode 100644 index 0000000..b9441fa --- /dev/null +++ b/machine-learning-process/charts/event-driven/templates/local-stack-secret.yaml @@ -0,0 +1,10 @@ +apiVersion: v1 +stringData: + accesskey: test + secretkey: test +kind: Secret +metadata: + name: localstack-cred + labels: + app: localstack +type: Opaque \ No newline at end of file diff --git a/machine-learning-process/charts/event-driven/templates/role-binding.yaml b/machine-learning-process/charts/event-driven/templates/role-binding.yaml new file mode 100644 index 0000000..84d742c --- /dev/null +++ b/machine-learning-process/charts/event-driven/templates/role-binding.yaml @@ -0,0 +1,11 @@ +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: argo-extended-role-binding +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: argo-extended-role +subjects: +- kind: ServiceAccount + name: argo \ No newline at end of file diff --git a/machine-learning-process/charts/event-driven/templates/roles.yaml b/machine-learning-process/charts/event-driven/templates/roles.yaml new file mode 100644 index 0000000..e55cefe --- /dev/null +++ b/machine-learning-process/charts/event-driven/templates/roles.yaml @@ -0,0 +1,52 @@ +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: argo-extended-role +rules: + - apiGroups: + - "" + resources: + - pods + verbs: + - create + - patch + - delete + - list + - watch + - get + - patch + - apiGroups: + - "" + resources: + - pods/log + verbs: + - get + - list + - apiGroups: + - argoproj.io + resources: + - workflows + verbs: + - create + - get + - list + - watch + - update + - patch + - apiGroups: + - argoproj.io + resources: + - workflowtaskresults + verbs: + - create + - patch + - apiGroups: + - batch + resources: + - jobs + verbs: + - create + - get + - list + - watch \ No newline at end of file diff --git a/machine-learning-process/charts/event-driven/values.yaml b/machine-learning-process/charts/event-driven/values.yaml new file mode 100644 index 0000000..f5789b1 --- /dev/null +++ b/machine-learning-process/charts/event-driven/values.yaml @@ -0,0 +1,4 @@ +jetstreamReplicas: 2 +jetstreamVolumeSize: 1Gi + +streamName: STREAM \ No newline at end of file diff --git a/machine-learning-process/containers/stage-out/Dockerfile b/machine-learning-process/containers/stage-out/Dockerfile new file mode 100644 index 0000000..14d3e6e --- /dev/null +++ b/machine-learning-process/containers/stage-out/Dockerfile @@ -0,0 +1,15 @@ +FROM python:3.11 + +RUN apt update && apt install -y yq + +WORKDIR /code + +COPY ./requirements.txt /code/requirements.txt + +RUN pip install --no-cache-dir -r /code/requirements.txt + +COPY . /code + +RUN cd /code && python setup.py install + +RUN stage-out --help diff --git a/machine-learning-process/containers/stage-out/README.md b/machine-learning-process/containers/stage-out/README.md new file mode 100644 index 0000000..a7a4564 --- /dev/null +++ b/machine-learning-process/containers/stage-out/README.md @@ -0,0 +1,49 @@ +# stage-out + + +## For developers + +Create a Python environment with, e.g. `mamba`: + +``` +mamba create -n env_stage_in python +mamba activate env_stage_in +``` + +``` +pip install -r requirements.txt +``` + +## Container + +Build the container with: + +``` +docker build -t stage-out . +``` + +Test the container with: + +``` +docker run --rm docker.io/library/stage-out stage-out --help +``` + +## CWL + +You can use a cwlrunner like `cwltool` to do a stage-in operation. + +Requirement: + +* a built container tagged `docker.io/library/stage-out:latest` + +``` +cwltool stage-out.cwl --bucket iride-sentinel-2 --stac_catalog /data/work/iride-marketplace/stage-in/_ka1p9cp --subfolder pippo --usersettings usersettings.json +``` + +## Run tests + +The unit tests can be run with: + +`nose2` + +TODO: add capture stdout in CWL \ No newline at end of file diff --git a/machine-learning-process/containers/stage-out/app/__init__.py b/machine-learning-process/containers/stage-out/app/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/machine-learning-process/containers/stage-out/app/main.py b/machine-learning-process/containers/stage-out/app/main.py new file mode 100644 index 0000000..40470e7 --- /dev/null +++ b/machine-learning-process/containers/stage-out/app/main.py @@ -0,0 +1,94 @@ +""" Main module for the application. """ +import os +import shutil + +import boto3 +import botocore +import click +import pystac +from botocore.client import Config +from loguru import logger +from pystac.stac_io import StacIO +import sys +from app.stac import CustomStacIO, upload_file_with_chunk_size +from app.usersettings import UserSettings + + +@click.command() +@click.option( + "--stac-catalog", help="Local path to a folder containing catalog.json STAC Catalog", required=True +) +@click.option("--user-settings", help="S3 user settings", required=True) +@click.option("--bucket", "bucket", help="S3 bucket", required=True) +@click.option("--subfolder", "subfolder", help="S3 subfolder", required=True) +def main(stac_catalog, user_settings, bucket, subfolder): + user_settings_config = UserSettings.from_file(user_settings) + + s3_settings = user_settings_config.get_s3_settings(f"s3://{bucket}/{subfolder}") + + if not s3_settings: + raise ValueError("No S3 settings found for this bucket") + + # set the environment variables for S3 from the user settings + os.environ["aws_access_key_id"] = s3_settings["aws_access_key_id"] + os.environ["aws_secret_access_key"] = s3_settings["aws_secret_access_key"] + os.environ["aws_region_name"] = s3_settings["region_name"] + os.environ["aws_endpoint_url"] = s3_settings["endpoint_url"] + + client = boto3.client( + "s3", + **s3_settings, + config=Config(s3={"addressing_style": "path", "signature_version": "s3v4"}), + ) + + shutil.copytree(stac_catalog, "/tmp/catalog") + cat = pystac.read_file(os.path.join("/tmp/catalog", "catalog.json")) + + StacIO.set_default(CustomStacIO) + + for item in cat.get_items(): + for key, asset in item.get_assets().items(): + s3_path = os.path.normpath( + os.path.join(os.path.join(subfolder, item.id, asset.href)) + ) + logger.info(f"upload {asset.href} to s3://{bucket}/{s3_path}") + + upload_file_with_chunk_size( + client, + asset.get_absolute_href(), + bucket, + s3_path + ) + + asset.href = f"s3://{bucket}/{s3_path}" + item.add_asset(key, asset) + + cat.normalize_hrefs(f"s3://{bucket}/{subfolder}") + + for item in cat.get_items(): + # upload item to S3 + logger.info(f"upload {item.id} to s3://{bucket}/{subfolder}") + for index, link in enumerate(item.links): + if link.rel in ["collection"]: + logger.info("saving collection.json") + collection = link.target + collection.links = [] + pystac.write_file(link.target, dest_href="./collection.json") + item.links.pop(index) + item.set_collection(None) + if link.rel in ["root"]: + item.links.pop(index) + pystac.write_file(item, item.get_self_href()) + + # upload catalog to S3 + logger.info(f"upload catalog.json to s3://{bucket}/{subfolder}") + for index, link in enumerate(cat.links): + if link.rel in ["root", "collection"]: + cat.links.pop(index) + pystac.write_file(cat, cat.get_self_href()) + + logger.info("Done!") + + print(f"s3://{bucket}/{subfolder}/catalog.json", file=sys.stdout) +if __name__ == "__main__": + main() diff --git a/machine-learning-process/containers/stage-out/app/stac.py b/machine-learning-process/containers/stage-out/app/stac.py new file mode 100644 index 0000000..66709fc --- /dev/null +++ b/machine-learning-process/containers/stage-out/app/stac.py @@ -0,0 +1,77 @@ +import os +from urllib.parse import urlparse + +import boto3 +import botocore +from botocore.client import Config +from pystac.stac_io import DefaultStacIO + + +class CustomStacIO(DefaultStacIO): + """Custom STAC IO class that uses boto3 to read from S3.""" + + def __init__(self): + self.session = botocore.session.Session() + + def write_text(self, dest, txt, *args, **kwargs): + self.s3_client = self.session.create_client( + service_name="s3", + use_ssl=True, + aws_access_key_id=os.environ["aws_access_key_id"], + aws_secret_access_key=os.environ["aws_secret_access_key"], + region_name=os.environ["aws_region_name"], + endpoint_url=os.environ["aws_endpoint_url"], + config=Config(s3={"addressing_style": "path", "signature_version": "s3v4"}), + ) + + parsed = urlparse(dest) + if parsed.scheme == "s3": + self.s3_client.put_object( + Body=txt.encode("UTF-8"), + Bucket=parsed.netloc, + Key=parsed.path[1:], + ContentType="application/geo+json", + ) + else: + super().write_text(dest, txt, *args, **kwargs) + +import boto3 +from boto3.s3.transfer import TransferConfig +import os + +import boto3 +import os +import shutil +import logging +from boto3.s3.transfer import TransferConfig +from botocore.client import Config +from pystac import StacIO +import pystac + +# Assuming CustomStacIO is already defined somewhere +# from your_custom_module import CustomStacIO + +logger = logging.getLogger() +logger.setLevel(logging.INFO) + +def upload_file_with_chunk_size(client, file_path, bucket_name, s3_key, max_chunks=1000, default_chunk_size_mb=25): + # Get the size of the file + file_size = os.path.getsize(file_path) + + # Convert default chunk size from MB to bytes + default_chunk_size = default_chunk_size_mb * 1024 * 1024 + + # Calculate the optimal chunk size to ensure the number of chunks does not exceed max_chunks + optimal_chunk_size = min(default_chunk_size, file_size // max_chunks + 1) + + # Configure the transfer settings + config = TransferConfig(multipart_chunksize=optimal_chunk_size) + + # Upload the file + client.upload_file( + Filename=file_path, + Bucket=bucket_name, + Key=s3_key, + Config=config + ) + diff --git a/machine-learning-process/containers/stage-out/app/usersettings.py b/machine-learning-process/containers/stage-out/app/usersettings.py new file mode 100644 index 0000000..9f4e8ae --- /dev/null +++ b/machine-learning-process/containers/stage-out/app/usersettings.py @@ -0,0 +1,63 @@ +""" +This code defines a UserSettings class that has methods for reading JSON files, +matching regular expressions, and setting environment variables for an S3 service. +The set_s3_environment method sets environment variables for an S3 service based +on a given URL. +""" + +import json +import os +import re +from typing import Dict + + +class UserSettings: + """class for reading JSON files, matching regular expressions, + and setting environment variables for an S3 service""" + + def __init__(self, settings: Dict): + self.settings = settings + + @classmethod + def from_file(cls, file_path): + """create a UserSettings object from a JSON file""" + return cls(cls.read_json_file(file_path)) + + @staticmethod + def read_json_file(file_path): + """read a JSON file and return the contents as a dictionary""" + with open(file_path, "r", encoding="utf-8") as stream: + return json.load(stream) + + @staticmethod + def match_regex(regex, string): + """match a regular expression to a string and return the match object""" + return re.search(regex, string) + + @staticmethod + def set_s3_vars(s3_service): + """set environment variables for an S3 service""" + os.environ["AWS_ACCESS_KEY_ID"] = s3_service["AccessKey"] + os.environ["AWS_SECRET_ACCESS_KEY"] = s3_service["SecretKey"] + os.environ["AWS_DEFAULT_REGION"] = s3_service["Region"] + os.environ["AWS_REGION"] = s3_service["Region"] + os.environ["AWS_S3_ENDPOINT"] = s3_service["ServiceURL"] + + def set_s3_environment(self, url): + """set environment variables for an S3 service based on a given URL""" + for _, s3_service in self.settings["S3"]["Services"].items(): + if self.match_regex(s3_service["UrlPattern"], url): + self.set_s3_vars(s3_service) + break + + def get_s3_settings(self, url): + """return S3 settings based on a given URL""" + for _, s3_service in self.settings["S3"]["Services"].items(): + if self.match_regex(s3_service["UrlPattern"], url): + return { + "region_name": s3_service["Region"], + "endpoint_url": s3_service["ServiceURL"], + "aws_access_key_id": s3_service["AccessKey"], + "aws_secret_access_key": s3_service["SecretKey"], + } + return None diff --git a/machine-learning-process/containers/stage-out/requirements.txt b/machine-learning-process/containers/stage-out/requirements.txt new file mode 100644 index 0000000..281aba6 --- /dev/null +++ b/machine-learning-process/containers/stage-out/requirements.txt @@ -0,0 +1,5 @@ +boto3 +pystac +loguru +click +pyyaml \ No newline at end of file diff --git a/machine-learning-process/containers/stage-out/setup.cfg b/machine-learning-process/containers/stage-out/setup.cfg new file mode 100644 index 0000000..03162a4 --- /dev/null +++ b/machine-learning-process/containers/stage-out/setup.cfg @@ -0,0 +1,3 @@ +[metadata] +name=stage-out +version=0.3.4 diff --git a/machine-learning-process/containers/stage-out/setup.py b/machine-learning-process/containers/stage-out/setup.py new file mode 100644 index 0000000..85fdb58 --- /dev/null +++ b/machine-learning-process/containers/stage-out/setup.py @@ -0,0 +1,12 @@ +from setuptools import setup + +console_scripts = ["stage-out=app.main:main"] + +setup( + name="stage-out", + version="0.1", + description="The funniest joke in the world", + packages=["app"], + entry_points={"console_scripts": console_scripts}, + zip_safe=False, +) diff --git a/machine-learning-process/containers/validate-schema/Dockerfile b/machine-learning-process/containers/validate-schema/Dockerfile new file mode 100644 index 0000000..5c9c7a4 --- /dev/null +++ b/machine-learning-process/containers/validate-schema/Dockerfile @@ -0,0 +1,5 @@ +FROM python:3.11-slim + +WORKDIR /app + +RUN pip install jsonschema \ No newline at end of file diff --git a/machine-learning-process/files/bash-login b/machine-learning-process/files/bash-login new file mode 100644 index 0000000..5becb33 --- /dev/null +++ b/machine-learning-process/files/bash-login @@ -0,0 +1 @@ +source /workspace/.bashrc \ No newline at end of file diff --git a/machine-learning-process/files/bash-rc b/machine-learning-process/files/bash-rc new file mode 100644 index 0000000..ec5a11e --- /dev/null +++ b/machine-learning-process/files/bash-rc @@ -0,0 +1,5 @@ +alias ll="ls -l" + +alias aws="aws --endpoint-url=http://eoap-event-driven-localstack:4566" + +source /workspace/.venv/bin/activate \ No newline at end of file diff --git a/machine-learning-process/files/init.sh b/machine-learning-process/files/init.sh new file mode 100644 index 0000000..e371ec1 --- /dev/null +++ b/machine-learning-process/files/init.sh @@ -0,0 +1,25 @@ +cd /workspace + +#temp +mkdir event-driven-with-argo +git clone https://github.com/eoap/event-driven-with-argo.git +cd event-driven-with-argo +git checkout develop + +code-server --install-extension ms-python.python +code-server --install-extension ms-toolsai.jupyter +ln -s /workspace/.local/share/code-server/extensions /workspace/extensions + +mkdir -p /workspace/User/ +echo '{"workbench.colorTheme": "Visual Studio Dark"}' > /workspace/User/settings.json + +python -m venv /workspace/.venv +source /workspace/.venv/bin/activate +/workspace/.venv/bin/python -m pip install --no-cache-dir ipykernel requests pyyaml boto3==1.35.23 loguru redis +/workspace/.venv/bin/python -m ipykernel install --user --name zoo_env --display-name "Python (Event Driven with Argo)" + +export AWS_DEFAULT_REGION="us-east-1" +export AWS_ACCESS_KEY_ID="test" +export AWS_SECRET_ACCESS_KEY="test" +aws s3 mb s3://results --endpoint-url=http://eoap-event-driven-localstack:4566 +aws s3 mb s3://workflows --endpoint-url=http://eoap-event-driven-localstack:4566 \ No newline at end of file diff --git a/machine-learning-process/skaffold.yaml b/machine-learning-process/skaffold.yaml new file mode 100644 index 0000000..aaf7ca6 --- /dev/null +++ b/machine-learning-process/skaffold.yaml @@ -0,0 +1,157 @@ +apiVersion: skaffold/v4beta9 +kind: Config +metadata: + name: eoap-event-driven +build: + artifacts: + - image: stageout + context: containers/stage-out + docker: + dockerfile: Dockerfile + - image: validateschema + context: containers/validate-schema + docker: + dockerfile: Dockerfile + +deploy: + helm: + releases: + # argo workflows + - name: eoap-argo-workflows + remoteChart: argo/argo-workflows + namespace: eoap-event-driven + createNamespace: true + wait: true + setValues: + crds.install: true + server.authModes: [server] + singleNamespace: true + workflow: + serviceAccount: + create: true + name: "argo" + rbac: + create: true + controller: + workflowNamespaces: + - eoap-event-driven + workflowDefaults: + spec: + serviceAccountName: argo + podGC: + strategy: OnWorkflowCompletion + deleteDelayDuration: 120s + rbac.writeConfigMaps: true + securityContext.runAsNonRoot: false + artifactRepository.s3: + bucket: workflows + endpoint: eoap-event-driven-localstack:4566 + insecure: true + accessKeySecret: + name: localstack-cred + key: accesskey + secretKeySecret: + name: localstack-cred + key: secretkey + # argo events + - name: eoap-argo-events + remoteChart: argo/argo-events + namespace: eoap-event-driven + wait: true + createNamespace: true + setValues: + crds.install: true + controller: + rbac: + enabled: true + namespaced: true + managedNamespace: eoap-event-driven + + + - name: eoap-redis + chartPath: ./charts/redis + namespace: eoap-event-driven + createNamespace: true + setValues: + maxMemory: 2mb + maxMemoryPolicy: allkeys-lru + port: 6379 + + # local chart for event bus/source/sensor + - name: eoap-event-driven + chartPath: ./charts/event-driven + namespace: eoap-event-driven + createNamespace: true + wait: true + setValues: + jetstreamReplicas: 3 + jetstreamVolumeSize: 1Gi + streamName: STREAM + + # argo cwl runner + - name: argo-cwl-runner + chartPath: ./charts/argo-cwl-runner + namespace: eoap-event-driven + valuesFiles: + - ./charts/argo-cwl-runner/values.yaml + setValueTemplates: + stageOutImage: "{{.IMAGE_NAME_stageout}}:{{.IMAGE_TAG_stageout}}" + setFiles: { + userSettings: ./charts/argo-cwl-runner/files/user-settings.json + } + # argo water bodies + - name: argo-water-bodies + chartPath: ./charts/argo-water-bodies + namespace: eoap-event-driven + valuesFiles: + - ./charts/argo-water-bodies/values.yaml + setFiles: { + appPackage: ./charts/argo-water-bodies/files/app-package.json, + inputSchema: ./charts/argo-water-bodies/files/input-parameters-schema.json + } + setValues: + storageClassRWX: "standard" + + - name: eoap-event-driven-localstack + remoteChart: localstack/localstack + namespace: eoap-event-driven + createNamespace: true + wait: true + setValues: + service.type: ClusterIP + + - name: eoap-event-driven-coder + chartPath: ../charts/coder + namespace: eoap-event-driven + createNamespace: true + wait: true + setValues: + coder.coderImage: eoepca/pde-code-server:1.0.0 + coder.workspace: event-driven-with-argo + coderstorageClassName: standard + coder.workspaceStorage: 10Gi + coderResources.limits.cpu: '2' + coderResources.limits.memory: '6442450944' + coderResources.requests.cpu: '1' + coderResources.requests.memory: '4294967296' + calrissian.enabled: true + setFiles: { + initScript: ./files/init.sh, + bashrcScript: ./files/bash-rc, + bashloginScript: ./files/bash-login + } + + +portForward: + - resourceType: service + resourceName: eoap-argo-workflows-server + namespace: eoap-event-driven + address: localhost + port: 2746 + localPort: 2746 + - resourceType: service + resourceName: code-server-service + namespace: eoap-event-driven + address: localhost + port: 8080 + localPort: 8000 \ No newline at end of file From 17f803abce40891d321034ebf8eee2f5cecddfc7 Mon Sep 17 00:00:00 2001 From: pmembari Date: Tue, 25 Feb 2025 11:06:04 +0100 Subject: [PATCH 04/61] =?UTF-8?q?=E2=9C=A8=20feat:=20adding=20pre-commit?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .pre-commit-config.yaml | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100644 .pre-commit-config.yaml diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml new file mode 100644 index 0000000..39749b2 --- /dev/null +++ b/.pre-commit-config.yaml @@ -0,0 +1,36 @@ +repos: + - repo: https://github.com/pre-commit/pre-commit-hooks + rev: v4.2.0 + hooks: + - id: check-added-large-files + - id: check-merge-conflict + - id: debug-statements + - id: detect-private-key + - id: end-of-file-fixer + - id: no-commit-to-branch + args: + - --pattern=master|main + - id: requirements-txt-fixer + - id: trailing-whitespace + - repo: https://github.com/myint/autoflake + rev: v1.4 + hooks: + - id: autoflake + args: + - --in-place + - --remove-unused-variables + - --remove-all-unused-imports + - repo: https://github.com/PyCQA/flake8 + rev: 4.0.1 + hooks: + - id: flake8 + args: + - --max-line-length=88 + - --max-doc-length=90 + + - repo: https://github.com/psf/black + rev: 22.3.0 + hooks: + - id: black + args: + - --line-length=140 From b96a8af377c3549b8d2de87ecb832a7210cac060 Mon Sep 17 00:00:00 2001 From: pmembari Date: Tue, 25 Feb 2025 12:22:17 +0100 Subject: [PATCH 05/61] feat: add pystac_client to venv --- event-driven-with-argo/files/init.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/event-driven-with-argo/files/init.sh b/event-driven-with-argo/files/init.sh index e371ec1..fe17126 100644 --- a/event-driven-with-argo/files/init.sh +++ b/event-driven-with-argo/files/init.sh @@ -15,7 +15,7 @@ echo '{"workbench.colorTheme": "Visual Studio Dark"}' > /workspace/User/settings python -m venv /workspace/.venv source /workspace/.venv/bin/activate -/workspace/.venv/bin/python -m pip install --no-cache-dir ipykernel requests pyyaml boto3==1.35.23 loguru redis +/workspace/.venv/bin/python -m pip install --no-cache-dir ipykernel requests pyyaml boto3==1.35.23 loguru redis pystac_client /workspace/.venv/bin/python -m ipykernel install --user --name zoo_env --display-name "Python (Event Driven with Argo)" export AWS_DEFAULT_REGION="us-east-1" From f638e4b897538c21fcc1b5059652ae9c54e382bb Mon Sep 17 00:00:00 2001 From: pmembari Date: Tue, 25 Feb 2025 12:25:37 +0100 Subject: [PATCH 06/61] docs: edit docs --- event-driven-with-argo/README.md | 61 +++----------------------------- 1 file changed, 4 insertions(+), 57 deletions(-) diff --git a/event-driven-with-argo/README.md b/event-driven-with-argo/README.md index 8c314e8..05c4ef4 100644 --- a/event-driven-with-argo/README.md +++ b/event-driven-with-argo/README.md @@ -1,14 +1,12 @@ # Event driven with argo: ## Installation -- The user must start a Minikube cluster (you can use the `--driver` option to specify the container runtime) using the command below: +- The user must start a Minikube cluster using the command below: ``` - minikube start --driver=docker + minikube start ``` - Using `--driver=docker` runs Minikube inside a Docker container instead of a virtual machine, which provides several advantages. It ensures a faster and more lightweight Kubernetes cluster initialization. This approach is particularly useful for development and testing environments where quick deployment and easy cleanup. - - Please add the following repositories to your Helm chart to ensure access to the required packages: ```sh @@ -22,63 +20,12 @@ skaffold dev ``` -- Wait for the deployment to stablize (1-3 minutes) and then open your browser on the link printed, usually http://127.0.0.1:8000. +- Wait for the deployment to stabilize (1-3 minutes) and then open your browser on the link printed, usually http://127.0.0.1:8000. - You will see the code server is running. Follow the documentations under `doc` folder. -> Note: Here is an example of Redis' publisher which is discussed in detail in the repository you have on code server -> ->```python -> ->from os import environ ->from redis import Redis ->from time import sleep -> ->stream_key = environ.get("STREAM", "STREAM") ->producer = environ.get("PRODUCER", "user-1") -> -> ->def connect_to_redis(): -> hostname = environ.get("REDIS_HOSTNAME", "redis-service") -> port = environ.get("REDIS_PORT", 6379) -> -> return Redis(hostname, port, retry_on_timeout=True) -> -> ->def send_event(redis_connection, aoi, reference): -> count = 0 -> -> try: -> # TODO cloud events -> # un-map the "data" wrt app package parameters -> data = { -> "producer": producer, -> "href": reference, -> } -> resp = redis_connection.xadd(stream_key, data) -> print(resp) -> count += 1 -> -> except ConnectionError as e: -> print(f"ERROR REDIS CONNECTION: {e}") -> -> ->connection = connect_to_redis() -> ->references = [ -> "https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2A_10TFK_20210708_0_L2A", -> "https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_10TFK_20210713_0_L2A", -> "https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2A_10TFK_20210718_0_L2A", -> "https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2A_10TFK_20220524_0_L2A", -> "https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2A_10TFK_20220514_0_L2A", -> "https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2A_10TFK_20220504_0_L2A" ->] -> ->for reference in references: -> send_event(connection, aoi, reference) -> sleep(1) ->``` +> Note: It is highly recommended to follow [this tutorial](https://eoap.github.io/event-driven-with-argo/) to know the rationale behind this activity. ### **Troubleshooting** From 4b8b9bca8cf202633b925bfa05f5691e87efb3ac Mon Sep 17 00:00:00 2001 From: pmembari Date: Tue, 25 Feb 2025 12:25:41 +0100 Subject: [PATCH 07/61] docs: edit docs --- event-driven-with-argo/README.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/event-driven-with-argo/README.md b/event-driven-with-argo/README.md index 05c4ef4..4df9b69 100644 --- a/event-driven-with-argo/README.md +++ b/event-driven-with-argo/README.md @@ -1,4 +1,7 @@ # Event driven with argo: + +> Note: It is highly recommended to follow [this tutorial](https://eoap.github.io/event-driven-with-argo/) to know the rationale behind this activity. + ## Installation - The user must start a Minikube cluster using the command below: @@ -25,7 +28,7 @@ - You will see the code server is running. Follow the documentations under `doc` folder. -> Note: It is highly recommended to follow [this tutorial](https://eoap.github.io/event-driven-with-argo/) to know the rationale behind this activity. + ### **Troubleshooting** From 5eb3ccfc5f23891f85754f089ef2308a14eccdfe Mon Sep 17 00:00:00 2001 From: pmembari Date: Tue, 25 Feb 2025 12:27:58 +0100 Subject: [PATCH 08/61] docs: edit docs --- event-driven-with-argo/README.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/event-driven-with-argo/README.md b/event-driven-with-argo/README.md index 4df9b69..c51493d 100644 --- a/event-driven-with-argo/README.md +++ b/event-driven-with-argo/README.md @@ -1,7 +1,5 @@ # Event driven with argo: -> Note: It is highly recommended to follow [this tutorial](https://eoap.github.io/event-driven-with-argo/) to know the rationale behind this activity. - ## Installation - The user must start a Minikube cluster using the command below: @@ -26,6 +24,7 @@ - Wait for the deployment to stabilize (1-3 minutes) and then open your browser on the link printed, usually http://127.0.0.1:8000. - You will see the code server is running. Follow the documentations under `doc` folder. + > Note: It is highly recommended to follow [this tutorial](https://eoap.github.io/event-driven-with-argo/) to know the rationale behind this activity. From 837c44e7335c41400d2eb2bafabb5abbdd6540c8 Mon Sep 17 00:00:00 2001 From: pmembari Date: Wed, 26 Feb 2025 16:39:36 +0100 Subject: [PATCH 09/61] feat: init machine learning process --- machine-learning-process/README.md | 0 .../charts/argo-cwl-runner/.helmignore | 23 ++ .../charts/argo-cwl-runner/Chart.yaml | 24 ++ .../argo-cwl-runner/files/user-settings.json | 15 + .../templates/argo-calrissian.yaml | 324 +++++++++++++++ .../argo-cwl-runner/templates/semaphore.yaml | 7 + .../templates/user-settings.yaml | 9 + .../charts/argo-cwl-runner/values.yaml | 2 + .../charts/argo-water-bodies/.helmignore | 23 ++ .../charts/argo-water-bodies/Chart.yaml | 24 ++ .../argo-water-bodies/files/app-package.json | 384 ++++++++++++++++++ .../files/input-parameters-schema.json | 34 ++ .../charts/argo-water-bodies/files/prepare.sh | 15 + .../argo-water-bodies/files/validation.py | 20 + .../templates/app-package.yaml | 6 + .../templates/argo-water-bodies.yaml | 216 ++++++++++ .../templates/input-parameters-schema.yaml | 6 + .../templates/semaphore.yaml | 7 + .../charts/argo-water-bodies/values.yaml | 8 + .../charts/event-driven/Chart.yaml | 6 + .../event-driven/templates/event-bus.yaml | 18 + .../templates/event-sensor-log.yaml | 16 + .../event-driven/templates/event-sensor.yaml | 46 +++ .../event-driven/templates/event-source.yaml | 13 + .../templates/local-stack-secret.yaml | 10 + .../event-driven/templates/role-binding.yaml | 11 + .../charts/event-driven/templates/roles.yaml | 52 +++ .../charts/event-driven/values.yaml | 4 + .../charts/redis/Chart.yaml | 6 + .../charts/redis/templates/config.yaml | 8 + .../redis/templates/redis-deployment.yaml | 42 ++ .../charts/redis/templates/redis-service.yaml | 12 + .../charts/redis/values.yaml | 3 + .../containers/stage-out/Dockerfile | 15 + .../containers/stage-out/README.md | 49 +++ .../containers/stage-out/app/__init__.py | 0 .../containers/stage-out/app/main.py | 94 +++++ .../containers/stage-out/app/stac.py | 77 ++++ .../containers/stage-out/app/usersettings.py | 63 +++ .../containers/stage-out/requirements.txt | 5 + .../containers/stage-out/setup.cfg | 3 + .../containers/stage-out/setup.py | 12 + .../containers/validate-schema/Dockerfile | 5 + machine-learning-process/files/bash-login | 1 + machine-learning-process/files/bash-rc | 5 + machine-learning-process/files/init.sh | 25 ++ machine-learning-process/skaffold.yaml | 157 +++++++ 47 files changed, 1905 insertions(+) create mode 100644 machine-learning-process/README.md create mode 100644 machine-learning-process/charts/argo-cwl-runner/.helmignore create mode 100644 machine-learning-process/charts/argo-cwl-runner/Chart.yaml create mode 100644 machine-learning-process/charts/argo-cwl-runner/files/user-settings.json create mode 100644 machine-learning-process/charts/argo-cwl-runner/templates/argo-calrissian.yaml create mode 100644 machine-learning-process/charts/argo-cwl-runner/templates/semaphore.yaml create mode 100644 machine-learning-process/charts/argo-cwl-runner/templates/user-settings.yaml create mode 100644 machine-learning-process/charts/argo-cwl-runner/values.yaml create mode 100644 machine-learning-process/charts/argo-water-bodies/.helmignore create mode 100644 machine-learning-process/charts/argo-water-bodies/Chart.yaml create mode 100644 machine-learning-process/charts/argo-water-bodies/files/app-package.json create mode 100644 machine-learning-process/charts/argo-water-bodies/files/input-parameters-schema.json create mode 100644 machine-learning-process/charts/argo-water-bodies/files/prepare.sh create mode 100644 machine-learning-process/charts/argo-water-bodies/files/validation.py create mode 100644 machine-learning-process/charts/argo-water-bodies/templates/app-package.yaml create mode 100644 machine-learning-process/charts/argo-water-bodies/templates/argo-water-bodies.yaml create mode 100644 machine-learning-process/charts/argo-water-bodies/templates/input-parameters-schema.yaml create mode 100644 machine-learning-process/charts/argo-water-bodies/templates/semaphore.yaml create mode 100644 machine-learning-process/charts/argo-water-bodies/values.yaml create mode 100644 machine-learning-process/charts/event-driven/Chart.yaml create mode 100644 machine-learning-process/charts/event-driven/templates/event-bus.yaml create mode 100644 machine-learning-process/charts/event-driven/templates/event-sensor-log.yaml create mode 100644 machine-learning-process/charts/event-driven/templates/event-sensor.yaml create mode 100644 machine-learning-process/charts/event-driven/templates/event-source.yaml create mode 100644 machine-learning-process/charts/event-driven/templates/local-stack-secret.yaml create mode 100644 machine-learning-process/charts/event-driven/templates/role-binding.yaml create mode 100644 machine-learning-process/charts/event-driven/templates/roles.yaml create mode 100644 machine-learning-process/charts/event-driven/values.yaml create mode 100644 machine-learning-process/charts/redis/Chart.yaml create mode 100644 machine-learning-process/charts/redis/templates/config.yaml create mode 100644 machine-learning-process/charts/redis/templates/redis-deployment.yaml create mode 100644 machine-learning-process/charts/redis/templates/redis-service.yaml create mode 100644 machine-learning-process/charts/redis/values.yaml create mode 100644 machine-learning-process/containers/stage-out/Dockerfile create mode 100644 machine-learning-process/containers/stage-out/README.md create mode 100644 machine-learning-process/containers/stage-out/app/__init__.py create mode 100644 machine-learning-process/containers/stage-out/app/main.py create mode 100644 machine-learning-process/containers/stage-out/app/stac.py create mode 100644 machine-learning-process/containers/stage-out/app/usersettings.py create mode 100644 machine-learning-process/containers/stage-out/requirements.txt create mode 100644 machine-learning-process/containers/stage-out/setup.cfg create mode 100644 machine-learning-process/containers/stage-out/setup.py create mode 100644 machine-learning-process/containers/validate-schema/Dockerfile create mode 100644 machine-learning-process/files/bash-login create mode 100644 machine-learning-process/files/bash-rc create mode 100644 machine-learning-process/files/init.sh create mode 100644 machine-learning-process/skaffold.yaml diff --git a/machine-learning-process/README.md b/machine-learning-process/README.md new file mode 100644 index 0000000..e69de29 diff --git a/machine-learning-process/charts/argo-cwl-runner/.helmignore b/machine-learning-process/charts/argo-cwl-runner/.helmignore new file mode 100644 index 0000000..0e8a0eb --- /dev/null +++ b/machine-learning-process/charts/argo-cwl-runner/.helmignore @@ -0,0 +1,23 @@ +# Patterns to ignore when building packages. +# This supports shell glob matching, relative path matching, and +# negation (prefixed with !). Only one pattern per line. +.DS_Store +# Common VCS dirs +.git/ +.gitignore +.bzr/ +.bzrignore +.hg/ +.hgignore +.svn/ +# Common backup files +*.swp +*.bak +*.tmp +*.orig +*~ +# Various IDEs +.project +.idea/ +*.tmproj +.vscode/ diff --git a/machine-learning-process/charts/argo-cwl-runner/Chart.yaml b/machine-learning-process/charts/argo-cwl-runner/Chart.yaml new file mode 100644 index 0000000..ac56d50 --- /dev/null +++ b/machine-learning-process/charts/argo-cwl-runner/Chart.yaml @@ -0,0 +1,24 @@ +apiVersion: v2 +name: argo-cwl-runner +description: A Helm chart for Kubernetes + +# A chart can be either an 'application' or a 'library' chart. +# +# Application charts are a collection of templates that can be packaged into versioned archives +# to be deployed. +# +# Library charts provide useful utilities or functions for the chart developer. They're included as +# a dependency of application charts to inject those utilities and functions into the rendering +# pipeline. Library charts do not define any templates and therefore cannot be deployed. +type: application + +# This is the chart version. This version number should be incremented each time you make changes +# to the chart and its templates, including the app version. +# Versions are expected to follow Semantic Versioning (https://semver.org/) +version: 0.1.0 + +# This is the version number of the application being deployed. This version number should be +# incremented each time you make changes to the application. Versions are not expected to +# follow Semantic Versioning. They should reflect the version the application is using. +# It is recommended to use it with quotes. +appVersion: "1.16.0" diff --git a/machine-learning-process/charts/argo-cwl-runner/files/user-settings.json b/machine-learning-process/charts/argo-cwl-runner/files/user-settings.json new file mode 100644 index 0000000..cf9eede --- /dev/null +++ b/machine-learning-process/charts/argo-cwl-runner/files/user-settings.json @@ -0,0 +1,15 @@ +{ + "S3": { + "Services": { + "results": { + "UrlPattern": "s3:\/\/results\/.*", + "Region": "us-east-1", + "AuthenticationRegion": "us-east-1", + "AccessKey": "test", + "SecretKey": "test", + "ServiceURL": "http://eoap-event-driven-localstack.eoap-event-driven.svc.cluster.local:4566", + "ForcePathStyle": "true" + } + } + } +} \ No newline at end of file diff --git a/machine-learning-process/charts/argo-cwl-runner/templates/argo-calrissian.yaml b/machine-learning-process/charts/argo-cwl-runner/templates/argo-calrissian.yaml new file mode 100644 index 0000000..1aa265a --- /dev/null +++ b/machine-learning-process/charts/argo-cwl-runner/templates/argo-calrissian.yaml @@ -0,0 +1,324 @@ +apiVersion: argoproj.io/v1alpha1 +kind: WorkflowTemplate +metadata: + name: argo-cwl-runner + annotations: + workflows.argoproj.io/version: ">= v3.3.0" + workflows.argoproj.io/description: | + This workflow template is a CWL runner for Argo Workflows. +spec: + entrypoint: calrissian-runner + parameters: + - name: parameters + description: "Parameters in JSON format" + value: "" + - name: cwl + description: "CWL document in JSON format" + value: "" + - name: max_ram + description: "Max RAM" + value: "8G" + - name: max_cores + description: "Max cores" + value: "8" + # this is the Workflow throttling to avoid overloading the cluster + # the configMap contains the number of concurrent Workflows allowed + synchronization: + semaphore: + configMapKeyRef: + name: semaphore-argo-cwl-runner + key: workflow + + # volumes + volumes: + - name: usersettings-vol + secret: + secretName: user-settings + - name: calrissian-wdir + persistentVolumeClaim: + claimName: calrissian-wdir + + # Workflow templates are used to define the Workflow steps + templates: + - name: calrissian-runner + # this is the entry point of the Workflow + inputs: + parameters: + - name: parameters + - name: cwl + - name: max_ram + - name: max_cores + outputs: + parameters: + - name: results + valueFrom: + parameter: {{ `"{{steps.get-results.outputs.parameters.calrissian-output}}"` }} + - name: log + valueFrom: + parameter: {{ `"{{steps.get-results.outputs.parameters.calrissian-stderr}}"` }} + - name: usage-report + valueFrom: + parameter: {{ `"{{steps.get-results.outputs.parameters.calrissian-report}}"` }} + - name: stac-catalog + valueFrom: + parameter: {{ `"{{steps.stage-out.outputs.parameters.stac-catalog}}"` }} + artifacts: + - name: tool-logs + from: {{ `"{{steps.get-results.outputs.artifacts.tool-logs}}"` }} + - name: calrissian-output + from: {{ `"{{steps.get-results.outputs.artifacts.calrissian-output}}"` }} + - name: calrissian-stderr + from: {{ `"{{steps.get-results.outputs.artifacts.calrissian-stderr}}"` }} + - name: calrissian-report + from: {{ `"{{steps.get-results.outputs.artifacts.calrissian-report}}"` }} + + steps: + # Workflow steps are defined here + - - name: cwl-prepare + template: cwl-prepare + arguments: + parameters: + - name: cwl + value: {{ `"{{inputs.parameters.cwl}}"` }} + - name: parameters + value: {{ `"{{inputs.parameters.parameters}}"` }} + + - - name: cwl-runner + template: calrissian-tmpl + arguments: + parameters: + - name: max_ram + value: {{ `"{{inputs.parameters.max_ram}}"` }} + - name: max_cores + value: {{ `"{{inputs.parameters.max_cores}}"` }} + + - - name: get-results + # these can run in parallel + # the same template is used for all the steps + template: get-results + arguments: + parameters: + - name: calrissian-output + value: "/calrissian/output.json" + - name: calrissian-stderr + value: "/calrissian/stderr.log" + - name: calrissian-report + value: "/calrissian/report.json" + + - name: stage-out + template: stage-out + arguments: + parameters: + - name: file_path + value: "/calrissian/output.json" + - name: bucket + value: "results" + - name: folder + value: {{ `"{{workflow.name}}-{{workflow.uid}}"` }} + + - name: cwl-prepare + # this steps prepares the CWL inputs + # needed by Calrissian + inputs: + parameters: + - name: cwl + - name: parameters + + script: + image: busybox:1.35.0 + resources: + requests: + memory: 1Gi + cpu: 1 + volumeMounts: + - name: calrissian-wdir + mountPath: /calrissian + env: [] + command: [ash] + source: | + #!/bin/ash + echo {{ `'{{inputs.parameters.cwl}}'` }} >> /calrissian/cwl.json + echo {{ `'{{inputs.parameters.parameters}}'` }} >> /calrissian/input.json + sleep 1 + + - name: calrissian-tmpl + # this step creates the Calrissian Job, Argo creates it as a Kubernetes Job + resource: + action: create + setOwnerReference: true + successCondition: status.succeeded > 0 + failureCondition: status.failed > 3 + manifest: | + apiVersion: batch/v1 + kind: Job + metadata: + generateName: calrissian-water-bodies-detection- + spec: + backoffLimit: 1 + activeDeadlineSeconds: 86400 + ttlSecondsAfterFinished: 120 + template: + metadata: + name: calrissian_pod + spec: + serviceAccountName: argo + containers: + - name: calrissian + image: ghcr.io/duke-gcb/calrissian/calrissian:0.16.0 + imagePullPolicy: IfNotPresent + command: + - calrissian + args: + - --debug + - --pod-serviceaccount + - argo + - --stdout + - /calrissian/output.json + - --stderr + - /calrissian/stderr.log + - --usage-report + - /calrissian/report.json + - --max-ram + - {{ `'{{inputs.parameters.max_ram}}'` }} + - --max-cores + - {{ `'{{inputs.parameters.max_cores}}'` }} + - --tmp-outdir-prefix + - /calrissian/tmp/ + - --outdir + - /calrissian/results/ + - --tool-logs-basepath + - /calrissian/logs + - "/calrissian/cwl.json" + - "/calrissian/input.json" + env: + - name: CALRISSIAN_POD_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name + - name: CALRISSIAN_DELETE_PODS + value: "true" + resources: + limits: + cpu: 2000m + memory: 2G + requests: + cpu: 1000m + memory: 1G + volumeMounts: + - mountPath: /calrissian + name: calrissian-wdir + readOnly: false + restartPolicy: Never + securityContext: + fsGroup: 0 + runAsGroup: 0 + runAsUser: 0 + terminationGracePeriodSeconds: 120 + volumes: + - name: calrissian-wdir + persistentVolumeClaim: + claimName: {{"{{workflow.name}}"}}-calrissian-wdir + readOnly: false + + inputs: + parameters: + - name: max_ram + - name: max_cores + outputs: + parameters: [] + artifacts: + - name: logs + path: /calrissian/logs + s3: + key: {{ `"{{workflow.name}}-{{workflow.uid}}-artifacts/calrissian-logs.tgz"` }} + + + + - name: get-results + # reads the files generated by Calrissian + inputs: + parameters: + - name: calrissian-output + - name: calrissian-stderr + - name: calrissian-report + outputs: + parameters: + - name: calrissian-output + valueFrom: + path: /tmp/calrissian-output.json + - name: calrissian-stderr + valueFrom: + path: /tmp/calrissian-stderr.txt + - name: calrissian-report + valueFrom: + path: /tmp/calrissian-report.json + artifacts: + - name: tool-logs + path: /calrissian/logs + s3: + key: {{ `"{{workflow.name}}-{{workflow.uid}}-artifacts/tool-logs.tgz"` }} + - name: calrissian-output + path: /tmp/calrissian-output.json + s3: + key: {{ `"{{workflow.name}}-{{workflow.uid}}-artifacts/calrissian-output.tgz"` }} + - name: calrissian-stderr + path: /tmp/calrissian-stderr.txt + s3: + key: {{ `"{{workflow.name}}-{{workflow.uid}}-artifacts/calrissian-stderr.tgz"` }} + - name: calrissian-report + path: /tmp/calrissian-report.json + s3: + key: {{ `"{{workflow.name}}-{{workflow.uid}}-artifacts/calrissian-report.tgz"` }} + script: + image: busybox:1.35.0 + resources: + requests: + memory: 1Gi + cpu: 1 + volumeMounts: + - name: calrissian-wdir + mountPath: /calrissian + command: [ash] + source: | + #!/bin/ash + cat "{{"{{inputs.parameters.calrissian-output}}"}}" > /tmp/calrissian-output.json + cat "{{"{{inputs.parameters.calrissian-stderr}}"}}" > /tmp/calrissian-stderr.txt + cat "{{"{{inputs.parameters.calrissian-report}}"}}" > /tmp/calrissian-report.json + + - name: stage-out + inputs: + parameters: + - name: file_path + - name: bucket + - name: folder + outputs: + parameters: + - name: stac-catalog + valueFrom: + path: /tmp/output + script: + image: {{ .Values.stageOutImage }} + resources: + requests: + memory: 1Gi + cpu: 1 + volumeMounts: + - name: calrissian-wdir + mountPath: /calrissian + - name: usersettings-vol + readOnly: true + mountPath: "/etc/secret" + command: [bash] + source: + #!/bin/bash + set -x + + cat /etc/secret/usersettings.json + + stage-out --stac-catalog $( cat {{"{{inputs.parameters.file_path}}"}} | jq -r .stac_catalog.path - ) --user-settings /etc/secret/usersettings.json --bucket "{{"{{inputs.parameters.bucket}}"}}" --subfolder "{{"{{inputs.parameters.folder}}"}}" + + res=$? + + echo "s3://{{"{{inputs.parameters.bucket}}"}}/{{"{{inputs.parameters.folder}}"}}/catalog.json" > /tmp/output + + exit $res \ No newline at end of file diff --git a/machine-learning-process/charts/argo-cwl-runner/templates/semaphore.yaml b/machine-learning-process/charts/argo-cwl-runner/templates/semaphore.yaml new file mode 100644 index 0000000..7751209 --- /dev/null +++ b/machine-learning-process/charts/argo-cwl-runner/templates/semaphore.yaml @@ -0,0 +1,7 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: semaphore-argo-cwl-runner +data: + workflow: "{{ .Values.semaphore }}" + \ No newline at end of file diff --git a/machine-learning-process/charts/argo-cwl-runner/templates/user-settings.yaml b/machine-learning-process/charts/argo-cwl-runner/templates/user-settings.yaml new file mode 100644 index 0000000..6bdf160 --- /dev/null +++ b/machine-learning-process/charts/argo-cwl-runner/templates/user-settings.yaml @@ -0,0 +1,9 @@ +# secret with the user-settings.json, it transforms the value yaml to json +apiVersion: v1 +kind: Secret +metadata: + name: user-settings + labels: + app: argo-water-bodies +data: + usersettings.json: {{ required "A valid .Values.userSettings entry required!" ( tpl ( .Values.userSettings | default ( .Files.Get "files/user-settings.json")) . | b64enc) }} \ No newline at end of file diff --git a/machine-learning-process/charts/argo-cwl-runner/values.yaml b/machine-learning-process/charts/argo-cwl-runner/values.yaml new file mode 100644 index 0000000..23d307c --- /dev/null +++ b/machine-learning-process/charts/argo-cwl-runner/values.yaml @@ -0,0 +1,2 @@ +stageOutImage: stageout +semaphore: 3 \ No newline at end of file diff --git a/machine-learning-process/charts/argo-water-bodies/.helmignore b/machine-learning-process/charts/argo-water-bodies/.helmignore new file mode 100644 index 0000000..0e8a0eb --- /dev/null +++ b/machine-learning-process/charts/argo-water-bodies/.helmignore @@ -0,0 +1,23 @@ +# Patterns to ignore when building packages. +# This supports shell glob matching, relative path matching, and +# negation (prefixed with !). Only one pattern per line. +.DS_Store +# Common VCS dirs +.git/ +.gitignore +.bzr/ +.bzrignore +.hg/ +.hgignore +.svn/ +# Common backup files +*.swp +*.bak +*.tmp +*.orig +*~ +# Various IDEs +.project +.idea/ +*.tmproj +.vscode/ diff --git a/machine-learning-process/charts/argo-water-bodies/Chart.yaml b/machine-learning-process/charts/argo-water-bodies/Chart.yaml new file mode 100644 index 0000000..a83b246 --- /dev/null +++ b/machine-learning-process/charts/argo-water-bodies/Chart.yaml @@ -0,0 +1,24 @@ +apiVersion: v2 +name: argo-water-bodies +description: A Helm chart for Kubernetes + +# A chart can be either an 'application' or a 'library' chart. +# +# Application charts are a collection of templates that can be packaged into versioned archives +# to be deployed. +# +# Library charts provide useful utilities or functions for the chart developer. They're included as +# a dependency of application charts to inject those utilities and functions into the rendering +# pipeline. Library charts do not define any templates and therefore cannot be deployed. +type: application + +# This is the chart version. This version number should be incremented each time you make changes +# to the chart and its templates, including the app version. +# Versions are expected to follow Semantic Versioning (https://semver.org/) +version: 0.1.0 + +# This is the version number of the application being deployed. This version number should be +# incremented each time you make changes to the application. Versions are not expected to +# follow Semantic Versioning. They should reflect the version the application is using. +# It is recommended to use it with quotes. +appVersion: "1.16.0" diff --git a/machine-learning-process/charts/argo-water-bodies/files/app-package.json b/machine-learning-process/charts/argo-water-bodies/files/app-package.json new file mode 100644 index 0000000..bc09f22 --- /dev/null +++ b/machine-learning-process/charts/argo-water-bodies/files/app-package.json @@ -0,0 +1,384 @@ +{ + "cwlVersion": "v1.0", + "$namespaces": { + "s": "https://schema.org/" + }, + "s:softwareVersion": "1.0.0", + "schemas": [ + "http://schema.org/version/9.0/schemaorg-current-http.rdf" + ], + "$graph": [ + { + "class": "Workflow", + "id": "main", + "label": "Water bodies detection based on NDWI and otsu threshold", + "doc": "Water bodies detection based on NDWI and otsu threshold applied to Sentinel-2 COG STAC items", + "requirements": [ + { + "class": "ScatterFeatureRequirement" + }, + { + "class": "SubworkflowFeatureRequirement" + } + ], + "inputs": { + "aoi": { + "label": "area of interest", + "doc": "area of interest as a bounding box", + "type": "string" + }, + "epsg": { + "label": "EPSG code", + "doc": "EPSG code", + "type": "string", + "default": "EPSG:4326" + }, + "stac_items": { + "label": "Sentinel-2 STAC items", + "doc": "list of Sentinel-2 COG STAC items", + "type": "string[]" + }, + "bands": { + "label": "bands used for the NDWI", + "doc": "bands used for the NDWI", + "type": "string[]", + "default": [ + "green", + "nir" + ] + } + }, + "outputs": [ + { + "id": "stac_catalog", + "outputSource": [ + "node_stac/stac_catalog" + ], + "type": "Directory" + } + ], + "steps": { + "node_water_bodies": { + "run": "#detect_water_body", + "in": { + "item": "stac_items", + "aoi": "aoi", + "epsg": "epsg", + "bands": "bands" + }, + "out": [ + "detected_water_body" + ], + "scatter": "item", + "scatterMethod": "dotproduct" + }, + "node_stac": { + "run": "#stac", + "in": { + "item": "stac_items", + "rasters": { + "source": "node_water_bodies/detected_water_body" + } + }, + "out": [ + "stac_catalog" + ] + } + } + }, + { + "class": "Workflow", + "id": "detect_water_body", + "label": "Water body detection based on NDWI and otsu threshold", + "doc": "Water body detection based on NDWI and otsu threshold", + "requirements": [ + { + "class": "ScatterFeatureRequirement" + } + ], + "inputs": { + "aoi": { + "doc": "area of interest as a bounding box", + "type": "string" + }, + "epsg": { + "doc": "EPSG code", + "type": "string", + "default": "EPSG:4326" + }, + "bands": { + "doc": "bands used for the NDWI", + "type": "string[]" + }, + "item": { + "doc": "STAC item", + "type": "string" + } + }, + "outputs": [ + { + "id": "detected_water_body", + "outputSource": [ + "node_otsu/binary_mask_item" + ], + "type": "File" + } + ], + "steps": { + "node_crop": { + "run": "#crop", + "in": { + "item": "item", + "aoi": "aoi", + "epsg": "epsg", + "band": "bands" + }, + "out": [ + "cropped" + ], + "scatter": "band", + "scatterMethod": "dotproduct" + }, + "node_normalized_difference": { + "run": "#norm_diff", + "in": { + "rasters": { + "source": "node_crop/cropped" + } + }, + "out": [ + "ndwi" + ] + }, + "node_otsu": { + "run": "#otsu", + "in": { + "raster": { + "source": "node_normalized_difference/ndwi" + } + }, + "out": [ + "binary_mask_item" + ] + } + } + }, + { + "class": "CommandLineTool", + "id": "crop", + "requirements": { + "InlineJavascriptRequirement": {}, + "EnvVarRequirement": { + "envDef": { + "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", + "PYTHONPATH": "/app" + } + }, + "ResourceRequirement": { + "coresMax": 1, + "ramMax": 512 + } + }, + "hints": { + "DockerRequirement": { + "dockerPull": "ghcr.io/eoap/mastering-app-package/crop@sha256:61c2f37b3a1bd56a3eddd0f4224c72de665d42957d3325dd0397845cff198dab" + } + }, + "baseCommand": [ + "python", + "-m", + "app" + ], + "arguments": [], + "inputs": { + "item": { + "type": "string", + "inputBinding": { + "prefix": "--input-item" + } + }, + "aoi": { + "type": "string", + "inputBinding": { + "prefix": "--aoi" + } + }, + "epsg": { + "type": "string", + "inputBinding": { + "prefix": "--epsg" + } + }, + "band": { + "type": "string", + "inputBinding": { + "prefix": "--band" + } + } + }, + "outputs": { + "cropped": { + "outputBinding": { + "glob": "*.tif" + }, + "type": "File" + } + } + }, + { + "class": "CommandLineTool", + "id": "norm_diff", + "requirements": { + "InlineJavascriptRequirement": {}, + "EnvVarRequirement": { + "envDef": { + "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", + "PYTHONPATH": "/app" + } + }, + "ResourceRequirement": { + "coresMax": 1, + "ramMax": 512 + } + }, + "hints": { + "DockerRequirement": { + "dockerPull": "ghcr.io/eoap/mastering-app-package/norm_diff@sha256:641ab55968d7f500b641969d0a7422c4c9a80dd5cc1459ee07f3e0c5ee4fa230" + } + }, + "baseCommand": [ + "python", + "-m", + "app" + ], + "arguments": [], + "inputs": { + "rasters": { + "type": "File[]", + "inputBinding": { + "position": 1 + } + } + }, + "outputs": { + "ndwi": { + "outputBinding": { + "glob": "*.tif" + }, + "type": "File" + } + } + }, + { + "class": "CommandLineTool", + "id": "otsu", + "requirements": { + "InlineJavascriptRequirement": {}, + "EnvVarRequirement": { + "envDef": { + "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", + "PYTHONPATH": "/app" + } + }, + "ResourceRequirement": { + "coresMax": 1, + "ramMax": 512 + } + }, + "hints": { + "DockerRequirement": { + "dockerPull": "ghcr.io/eoap/mastering-app-package/otsu@sha256:ec02baf6a007ebb5a2dc037af2da8ef94db67bb8a6139682ad61041f9d27c779" + } + }, + "baseCommand": [ + "python", + "-m", + "app" + ], + "arguments": [], + "inputs": { + "raster": { + "type": "File", + "inputBinding": { + "position": 1 + } + } + }, + "outputs": { + "binary_mask_item": { + "outputBinding": { + "glob": "*.tif" + }, + "type": "File" + } + } + }, + { + "class": "CommandLineTool", + "id": "stac", + "requirements": { + "InlineJavascriptRequirement": {}, + "EnvVarRequirement": { + "envDef": { + "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", + "PYTHONPATH": "/app" + } + }, + "ResourceRequirement": { + "coresMax": 1, + "ramMax": 512 + } + }, + "hints": { + "DockerRequirement": { + "dockerPull": "ghcr.io/eoap/mastering-app-package/stac@sha256:8ad7e7d6999c0c4c91c2b19ce231f24734d2af8f41b855ef9a06276d23345601" + } + }, + "baseCommand": [ + "python", + "-m", + "app" + ], + "arguments": [], + "inputs": { + "item": { + "type": { + "type": "array", + "items": "string", + "inputBinding": { + "prefix": "--input-item" + } + } + }, + "rasters": { + "type": { + "type": "array", + "items": "File", + "inputBinding": { + "prefix": "--water-body" + } + } + } + }, + "outputs": { + "stac_catalog": { + "outputBinding": { + "glob": "." + }, + "type": "Directory" + } + } + } + ], + "s:codeRepository": { + "URL": "https://github.com/eoap/mastering-app-package.git" + }, + "s:author": [ + { + "class": "s:Person", + "s.name": "Jane Doe", + "s.email": "jane.doe@acme.earth", + "s.affiliation": "ACME" + } + ] +} diff --git a/machine-learning-process/charts/argo-water-bodies/files/input-parameters-schema.json b/machine-learning-process/charts/argo-water-bodies/files/input-parameters-schema.json new file mode 100644 index 0000000..e5154dd --- /dev/null +++ b/machine-learning-process/charts/argo-water-bodies/files/input-parameters-schema.json @@ -0,0 +1,34 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "title": "Water Bodies Detection Input", + "description": "Schema for input parameters to the water bodies detection workflow", + "type": "object", + "properties": { + "stac_items": { + "type": "array", + "description": "Array of URLs pointing to STAC items", + "items": { + "type": "string", + "format": "uri", + "description": "A valid URL pointing to a STAC item" + }, + "minItems": 1, + "uniqueItems": true + }, + "aoi": { + "type": "string", + "description": "Bounding box for the area of interest in format 'minLon,minLat,maxLon,maxLat'", + "pattern": "^-?\\d+(\\.\\d+)?,-?\\d+(\\.\\d+)?,-?\\d+(\\.\\d+)?,-?\\d+(\\.\\d+)?$", + "examples": ["-121.399,39.834,-120.74,40.472"] + }, + "epsg": { + "type": "string", + "description": "EPSG code for the coordinate reference system", + "pattern": "^EPSG:\\d+$", + "examples": ["EPSG:4326"] + } + }, + "required": ["stac_items", "aoi", "epsg"], + "additionalProperties": false + } + \ No newline at end of file diff --git a/machine-learning-process/charts/argo-water-bodies/files/prepare.sh b/machine-learning-process/charts/argo-water-bodies/files/prepare.sh new file mode 100644 index 0000000..135be70 --- /dev/null +++ b/machine-learning-process/charts/argo-water-bodies/files/prepare.sh @@ -0,0 +1,15 @@ +#!/bin/bash +set -x + +cat >/tmp/cwl_parameters.json < /tmp/cwl_workflow.json +sleep 1 + \ No newline at end of file diff --git a/machine-learning-process/charts/argo-water-bodies/files/validation.py b/machine-learning-process/charts/argo-water-bodies/files/validation.py new file mode 100644 index 0000000..4eeb376 --- /dev/null +++ b/machine-learning-process/charts/argo-water-bodies/files/validation.py @@ -0,0 +1,20 @@ +import sys +import json +import ast +import jsonschema +from jsonschema import validate + +inputs_raw = """{{`{{inputs.parameters.inputs}}`}}""" + +inputs = ast.literal_eval(inputs_raw) + +# Load the JSON schema from a file +with open("/schema/input-schema.json", "r") as schema_file: + schema = json.load(schema_file) + +try: + validate(instance=inputs, schema=schema) + print("Input JSON is valid.") +except jsonschema.exceptions.ValidationError as err: + print(f"Input JSON is invalid: {err.message}") + sys.exit(2) \ No newline at end of file diff --git a/machine-learning-process/charts/argo-water-bodies/templates/app-package.yaml b/machine-learning-process/charts/argo-water-bodies/templates/app-package.yaml new file mode 100644 index 0000000..a73add2 --- /dev/null +++ b/machine-learning-process/charts/argo-water-bodies/templates/app-package.yaml @@ -0,0 +1,6 @@ +kind: ConfigMap +apiVersion: v1 +metadata: + name: app-package-water-bodies-1-0-0 +data: + app-package.json: {{ required "A valid .Values.appPackage entry required!" ( tpl ( .Values.appPackage | default ( .Files.Get "files/app-package.json")) . | quote ) }} \ No newline at end of file diff --git a/machine-learning-process/charts/argo-water-bodies/templates/argo-water-bodies.yaml b/machine-learning-process/charts/argo-water-bodies/templates/argo-water-bodies.yaml new file mode 100644 index 0000000..8532f90 --- /dev/null +++ b/machine-learning-process/charts/argo-water-bodies/templates/argo-water-bodies.yaml @@ -0,0 +1,216 @@ +apiVersion: argoproj.io/v1alpha1 +kind: WorkflowTemplate +metadata: + name: water-bodies-detection + annotations: + workflows.argoproj.io/version: ">= v3.3.0" + workflows.argoproj.io/description: | + This workflow template detects water bodies. + eoap.ogc.org/version: "1.0.0" + eoap.ogc.org/title: "Water bodies detection based on NDWI and otsu threshold" + eoap.ogc.org/abstract: "Water bodies detection based on NDWI and otsu threshold applied to Sentinel-2 COG STAC items" + eoap.ogc.org/schema: "app-package-input-schema-water-bodies-1-0-0/input-schema.json" + eoap.ogc.org/app-package: "app-package-water-bodies-1-0-0/app-package.json" +spec: + onExit: exit-handler + # this is the Workflow throttling to avoid overloading the cluster + # the configMap contains the number of concurrent Workflows allowed + synchronization: + semaphore: + configMapKeyRef: + name: semaphore-water-bodies + key: workflow + # Claim the RWX volume + volumeClaimTemplates: + - metadata: + name: calrissian-wdir + spec: + accessModes: [ "ReadWriteMany" ] + storageClassName: "{{ .Values.storageClassRWX }}" + resources: + requests: + storage: "{{ .Values.storageSize }}" + volumes: + - name: usersettings-vol + secret: + secretName: user-settings + - name: volume-input-schema + configMap: + name: app-package-input-schema-water-bodies-1-0-0 + items: + - key: input-schema.json + path: input-schema.json + mode: 420 + defaultMode: 420 + optional: false + - name: volume-app-package + configMap: + name: app-package-water-bodies-1-0-0 + items: + - key: app-package.json + path: app-package.json + mode: 420 + defaultMode: 420 + optional: false + + entrypoint: main + parameters: + - name: items + value: "['https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2A_10TFK_20210708_0_L2A']" + - name: "aoi" + value: "-121.399,39.834,-120.74,40.472" + - name: "epsg" + value: "EPSG:4326" + + templates: + - name: main + inputs: + parameters: + - name: items + - name: aoi + - name: epsg + outputs: + parameters: + - name: results + valueFrom: + expression: "steps['argo-cwl'].outputs.parameters['results']" + - name: log + valueFrom: + expression: "steps['argo-cwl'].outputs.parameters['log']" + - name: usage-report + valueFrom: + expression: "steps['argo-cwl'].outputs.parameters['usage-report']" + - name: stac-catalog + valueFrom: + expression: "steps['argo-cwl'].outputs.parameters['stac-catalog']" + artifacts: + - name: tool-logs + fromExpression: "steps['argo-cwl'].outputs.artifacts['tool-logs']" + - name: calrissian-output + fromExpression: "steps['argo-cwl'].outputs.artifacts['calrissian-output']" + - name: calrissian-stderr + fromExpression: "steps['argo-cwl'].outputs.artifacts['calrissian-stderr']" + - name: calrissian-report + fromExpression: "steps['argo-cwl'].outputs.artifacts['calrissian-report']" + steps: + - - name: prepare + template: prepare + arguments: + parameters: + - name: items + value: {{ `"{{inputs.parameters.items}}"` }} + - name: aoi + value: {{ `"{{inputs.parameters.aoi}}"` }} + - name: epsg + value: {{ `"{{inputs.parameters.epsg}}"` }} + + - - name: validate-inputs + template: validate-inputs + arguments: + parameters: + - name: inputs + value: {{ `"{{steps.prepare.outputs.parameters.inputs}}"` }} + + - - name: argo-cwl # this steps invokes a WorkflowTemplate + templateRef: + name: argo-cwl-runner + template: calrissian-runner + arguments: + parameters: + - name: max_ram + value: "{{ .Values.maxRam }}" + - name: max_cores + value: "{{ .Values.maxCores }}" + - name: parameters + value: {{ `"{{steps.prepare.outputs.parameters.inputs}}"` }} + - name: cwl + value: {{ `"{{steps.prepare.outputs.parameters.workflow}}"` }} + + - name: prepare + inputs: + parameters: + - name: items + - name: aoi + - name: epsg + outputs: + parameters: + - name: inputs + valueFrom: + path: /tmp/cwl_parameters.json + - name: workflow + valueFrom: + path: /tmp/cwl_workflow.json + script: + image: ubuntu:20.04 + resources: + requests: + memory: 1Gi + cpu: 1 + + volumeMounts: + - name: volume-app-package + mountPath: /config + + command: [bash] + source: | + {{ required "A valid .Values.prepareScript entry required!" ( tpl ( .Values.prepareScript | default ( .Files.Get "files/prepare.sh")) . | nindent 10 ) }} + + - name: validate-inputs + inputs: + parameters: + - name: inputs + + script: + image: {{ .Values.validateSchemaImage }} + resources: + requests: + memory: 1Gi + cpu: 1 + + volumeMounts: + - name: volume-input-schema + mountPath: /schema + + command: [python] + source: | + {{ required "A valid .Values.validationScript entry required!" ( tpl ( .Values.validationScript | default ( .Files.Get "files/validation.py")) . | nindent 10 ) }} + + + - name: exit-handler + steps: + - - name: celebrate + template: notify-redis-success + when: "{{"{{workflow.status}}"}} == Succeeded" + - name: cry + template: notify-redis-failure + when: "{{"{{workflow.status}}"}} != Succeeded" + + + - name: notify-redis-success + script: + image: redis:alpine + command: [sh, -c] + source: | + item="{{ `{{workflow.parameters.items}}` }}" + item="${item:1:-1}" + catalog="s3://results/{{ `{{workflow.name}}`}}-{{ `{{workflow.uid}}` }}/catalog.json" + echo "XADD WATERBODIES * subject \"$item\" \ + workflow_id \"{{ `{{workflow.uid}}` }}\" \ + workflow_name \"{{ `{{workflow.name}}`}}\" \ + workflow_status \"{{ `{{workflow.status}}`}}\" \ + href \"$item\" \ + stac_catalog \"$catalog\"" | redis-cli -h redis-service -p 6379 + + - name: notify-redis-failure + script: + image: redis:alpine + command: [sh, -c] + source: | + item="{{ `{{workflow.parameters.items}}` }}" + item="${item:1:-1}" + echo "XADD WATERBODIESFAILURE * subject \"$item\" \ + workflow_id \"{{ `{{workflow.uid}}` }}\" \ + workflow_name \"{{ `{{workflow.name}}`}}\" \ + workflow_status \"{{ `{{workflow.status}}`}}\" \ + href \"$item\"" | redis-cli -h redis-service -p 6379 + diff --git a/machine-learning-process/charts/argo-water-bodies/templates/input-parameters-schema.yaml b/machine-learning-process/charts/argo-water-bodies/templates/input-parameters-schema.yaml new file mode 100644 index 0000000..5e88dcd --- /dev/null +++ b/machine-learning-process/charts/argo-water-bodies/templates/input-parameters-schema.yaml @@ -0,0 +1,6 @@ +kind: ConfigMap +apiVersion: v1 +metadata: + name: app-package-input-schema-water-bodies-1-0-0 +data: + input-schema.json: {{ required "A valid .Values.inputSchema entry required!" ( tpl ( .Values.inputSchema | default ( .Files.Get "files/input-parameters-schema.json")) . | quote ) }} \ No newline at end of file diff --git a/machine-learning-process/charts/argo-water-bodies/templates/semaphore.yaml b/machine-learning-process/charts/argo-water-bodies/templates/semaphore.yaml new file mode 100644 index 0000000..1cafcd8 --- /dev/null +++ b/machine-learning-process/charts/argo-water-bodies/templates/semaphore.yaml @@ -0,0 +1,7 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: semaphore-water-bodies +data: + workflow: "{{ .Values.semaphore }}" + \ No newline at end of file diff --git a/machine-learning-process/charts/argo-water-bodies/values.yaml b/machine-learning-process/charts/argo-water-bodies/values.yaml new file mode 100644 index 0000000..5ed76d5 --- /dev/null +++ b/machine-learning-process/charts/argo-water-bodies/values.yaml @@ -0,0 +1,8 @@ +semaphore: 3 +validateSchemaImage: validateschema + +storageClassRWX: standard +storageSize: "10Gi" + +maxRam: "4G" +maxCores: "4" \ No newline at end of file diff --git a/machine-learning-process/charts/event-driven/Chart.yaml b/machine-learning-process/charts/event-driven/Chart.yaml new file mode 100644 index 0000000..b23913a --- /dev/null +++ b/machine-learning-process/charts/event-driven/Chart.yaml @@ -0,0 +1,6 @@ +apiVersion: v2 +name: event-driven +description: Event driven manifests including event bus and source/sensor +type: application +version: 0.1.0 +appVersion: "1.0.0" diff --git a/machine-learning-process/charts/event-driven/templates/event-bus.yaml b/machine-learning-process/charts/event-driven/templates/event-bus.yaml new file mode 100644 index 0000000..ca28d85 --- /dev/null +++ b/machine-learning-process/charts/event-driven/templates/event-bus.yaml @@ -0,0 +1,18 @@ +apiVersion: argoproj.io/v1alpha1 +kind: EventBus +metadata: + name: jetstream-event-bus +spec: + jetstream: + version: latest + replicas: {{ .Values.jetstreamReplicas }} + persistence: + storageClassName: standard + accessMode: ReadWriteOnce + volumeSize: {{ .Values.jetstreamVolumeSize }} + streamConfig: | + maxAge: 24h + settings: | + max_file_store: 1GB + startArgs: + - "-D" \ No newline at end of file diff --git a/machine-learning-process/charts/event-driven/templates/event-sensor-log.yaml b/machine-learning-process/charts/event-driven/templates/event-sensor-log.yaml new file mode 100644 index 0000000..dd9e8d5 --- /dev/null +++ b/machine-learning-process/charts/event-driven/templates/event-sensor-log.yaml @@ -0,0 +1,16 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Sensor +metadata: + name: acme-sentinel2-stream-source-logger +spec: + eventBusName: jetstream-event-bus + + dependencies: + - name: acme-sentinel2-stream-source-logger + eventSourceName: acme-sentinel2-stream-source + eventName: acme-sentinel2-stream-source-collected + triggers: + - template: + name: log-event + log: + intervalSeconds: 5 # Frequency of log updates \ No newline at end of file diff --git a/machine-learning-process/charts/event-driven/templates/event-sensor.yaml b/machine-learning-process/charts/event-driven/templates/event-sensor.yaml new file mode 100644 index 0000000..f22053c --- /dev/null +++ b/machine-learning-process/charts/event-driven/templates/event-sensor.yaml @@ -0,0 +1,46 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Sensor +metadata: + name: acme-water-bodies-detection-sensor +spec: + eventBusName: jetstream-event-bus + template: + serviceAccountName: argo + dependencies: + - name: acme-sentinel2-stream + eventSourceName: acme-sentinel2-stream-source + eventName: acme-sentinel2-stream-source-collected + transform: + jq: ".values.href = [ .values.href ]" + triggers: + - template: + name: water-bodies-detection-trigger + k8s: + group: argoproj.io + version: v1alpha1 + resource: workflows + operation: create + source: + resource: + apiVersion: argoproj.io/v1alpha1 + kind: Workflow + metadata: + generateName: water-bodies-detection- + namespace: eoap-event-driven + spec: + serviceAccountName: argo + entrypoint: main + arguments: + parameters: + - name: items + - name: aoi + value: "-121.399,39.834,-120.74,40.472" + - name: epsg + value: "EPSG:4326" + workflowTemplateRef: + name: water-bodies-detection + parameters: + - src: + dependencyName: acme-sentinel2-stream + dataKey: values.href + dest: spec.arguments.parameters.0.value diff --git a/machine-learning-process/charts/event-driven/templates/event-source.yaml b/machine-learning-process/charts/event-driven/templates/event-source.yaml new file mode 100644 index 0000000..4811ea3 --- /dev/null +++ b/machine-learning-process/charts/event-driven/templates/event-source.yaml @@ -0,0 +1,13 @@ +apiVersion: argoproj.io/v1alpha1 +kind: EventSource +metadata: + name: acme-sentinel2-stream-source +spec: + eventBusName: jetstream-event-bus + redisStream: + acme-sentinel2-stream-source-collected: + hostAddress: redis-service:6379 + db: 0 + streams: + - {{ .Values.streamName }} + consumerGroup: acme-sentinel2-stream-source \ No newline at end of file diff --git a/machine-learning-process/charts/event-driven/templates/local-stack-secret.yaml b/machine-learning-process/charts/event-driven/templates/local-stack-secret.yaml new file mode 100644 index 0000000..b9441fa --- /dev/null +++ b/machine-learning-process/charts/event-driven/templates/local-stack-secret.yaml @@ -0,0 +1,10 @@ +apiVersion: v1 +stringData: + accesskey: test + secretkey: test +kind: Secret +metadata: + name: localstack-cred + labels: + app: localstack +type: Opaque \ No newline at end of file diff --git a/machine-learning-process/charts/event-driven/templates/role-binding.yaml b/machine-learning-process/charts/event-driven/templates/role-binding.yaml new file mode 100644 index 0000000..84d742c --- /dev/null +++ b/machine-learning-process/charts/event-driven/templates/role-binding.yaml @@ -0,0 +1,11 @@ +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: argo-extended-role-binding +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: argo-extended-role +subjects: +- kind: ServiceAccount + name: argo \ No newline at end of file diff --git a/machine-learning-process/charts/event-driven/templates/roles.yaml b/machine-learning-process/charts/event-driven/templates/roles.yaml new file mode 100644 index 0000000..e55cefe --- /dev/null +++ b/machine-learning-process/charts/event-driven/templates/roles.yaml @@ -0,0 +1,52 @@ +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: argo-extended-role +rules: + - apiGroups: + - "" + resources: + - pods + verbs: + - create + - patch + - delete + - list + - watch + - get + - patch + - apiGroups: + - "" + resources: + - pods/log + verbs: + - get + - list + - apiGroups: + - argoproj.io + resources: + - workflows + verbs: + - create + - get + - list + - watch + - update + - patch + - apiGroups: + - argoproj.io + resources: + - workflowtaskresults + verbs: + - create + - patch + - apiGroups: + - batch + resources: + - jobs + verbs: + - create + - get + - list + - watch \ No newline at end of file diff --git a/machine-learning-process/charts/event-driven/values.yaml b/machine-learning-process/charts/event-driven/values.yaml new file mode 100644 index 0000000..f5789b1 --- /dev/null +++ b/machine-learning-process/charts/event-driven/values.yaml @@ -0,0 +1,4 @@ +jetstreamReplicas: 2 +jetstreamVolumeSize: 1Gi + +streamName: STREAM \ No newline at end of file diff --git a/machine-learning-process/charts/redis/Chart.yaml b/machine-learning-process/charts/redis/Chart.yaml new file mode 100644 index 0000000..e615c41 --- /dev/null +++ b/machine-learning-process/charts/redis/Chart.yaml @@ -0,0 +1,6 @@ +apiVersion: v2 +name: redis +description: simple redis chart +type: application +version: 0.1.0 +appVersion: "1.0.0" diff --git a/machine-learning-process/charts/redis/templates/config.yaml b/machine-learning-process/charts/redis/templates/config.yaml new file mode 100644 index 0000000..e67dbf1 --- /dev/null +++ b/machine-learning-process/charts/redis/templates/config.yaml @@ -0,0 +1,8 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: redis-config +data: + redis-config: | + maxmemory {{ .Values.maxMemory }} + maxmemory-policy {{ .Values.maxMemoryPolicy }} diff --git a/machine-learning-process/charts/redis/templates/redis-deployment.yaml b/machine-learning-process/charts/redis/templates/redis-deployment.yaml new file mode 100644 index 0000000..ba296db --- /dev/null +++ b/machine-learning-process/charts/redis/templates/redis-deployment.yaml @@ -0,0 +1,42 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: redis +spec: + replicas: 1 + selector: + matchLabels: + app: redis + template: + metadata: + labels: + app: redis + spec: + containers: + - name: redis + image: redis:5.0.4 + command: + - redis-server + - "/redis-master/redis.conf" + env: + - name: MASTER + value: "true" + ports: + - containerPort: {{ .Values.port }} + resources: + limits: + cpu: "0.5" + volumeMounts: + - mountPath: /redis-master-data + name: data + - mountPath: /redis-master + name: config + volumes: + - name: data + emptyDir: {} + - name: config + configMap: + name: redis-config + items: + - key: redis-config + path: redis.conf diff --git a/machine-learning-process/charts/redis/templates/redis-service.yaml b/machine-learning-process/charts/redis/templates/redis-service.yaml new file mode 100644 index 0000000..5d7dbf9 --- /dev/null +++ b/machine-learning-process/charts/redis/templates/redis-service.yaml @@ -0,0 +1,12 @@ +apiVersion: v1 +kind: Service +metadata: + name: redis-service +spec: + selector: + app: redis + ports: + - protocol: TCP + port: 6379 + targetPort: 6379 + type: ClusterIP \ No newline at end of file diff --git a/machine-learning-process/charts/redis/values.yaml b/machine-learning-process/charts/redis/values.yaml new file mode 100644 index 0000000..a2c43dd --- /dev/null +++ b/machine-learning-process/charts/redis/values.yaml @@ -0,0 +1,3 @@ +maxMemory: 2mb +maxMemoryPolicy: allkeys-lru +port: 6379 \ No newline at end of file diff --git a/machine-learning-process/containers/stage-out/Dockerfile b/machine-learning-process/containers/stage-out/Dockerfile new file mode 100644 index 0000000..14d3e6e --- /dev/null +++ b/machine-learning-process/containers/stage-out/Dockerfile @@ -0,0 +1,15 @@ +FROM python:3.11 + +RUN apt update && apt install -y yq + +WORKDIR /code + +COPY ./requirements.txt /code/requirements.txt + +RUN pip install --no-cache-dir -r /code/requirements.txt + +COPY . /code + +RUN cd /code && python setup.py install + +RUN stage-out --help diff --git a/machine-learning-process/containers/stage-out/README.md b/machine-learning-process/containers/stage-out/README.md new file mode 100644 index 0000000..a7a4564 --- /dev/null +++ b/machine-learning-process/containers/stage-out/README.md @@ -0,0 +1,49 @@ +# stage-out + + +## For developers + +Create a Python environment with, e.g. `mamba`: + +``` +mamba create -n env_stage_in python +mamba activate env_stage_in +``` + +``` +pip install -r requirements.txt +``` + +## Container + +Build the container with: + +``` +docker build -t stage-out . +``` + +Test the container with: + +``` +docker run --rm docker.io/library/stage-out stage-out --help +``` + +## CWL + +You can use a cwlrunner like `cwltool` to do a stage-in operation. + +Requirement: + +* a built container tagged `docker.io/library/stage-out:latest` + +``` +cwltool stage-out.cwl --bucket iride-sentinel-2 --stac_catalog /data/work/iride-marketplace/stage-in/_ka1p9cp --subfolder pippo --usersettings usersettings.json +``` + +## Run tests + +The unit tests can be run with: + +`nose2` + +TODO: add capture stdout in CWL \ No newline at end of file diff --git a/machine-learning-process/containers/stage-out/app/__init__.py b/machine-learning-process/containers/stage-out/app/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/machine-learning-process/containers/stage-out/app/main.py b/machine-learning-process/containers/stage-out/app/main.py new file mode 100644 index 0000000..40470e7 --- /dev/null +++ b/machine-learning-process/containers/stage-out/app/main.py @@ -0,0 +1,94 @@ +""" Main module for the application. """ +import os +import shutil + +import boto3 +import botocore +import click +import pystac +from botocore.client import Config +from loguru import logger +from pystac.stac_io import StacIO +import sys +from app.stac import CustomStacIO, upload_file_with_chunk_size +from app.usersettings import UserSettings + + +@click.command() +@click.option( + "--stac-catalog", help="Local path to a folder containing catalog.json STAC Catalog", required=True +) +@click.option("--user-settings", help="S3 user settings", required=True) +@click.option("--bucket", "bucket", help="S3 bucket", required=True) +@click.option("--subfolder", "subfolder", help="S3 subfolder", required=True) +def main(stac_catalog, user_settings, bucket, subfolder): + user_settings_config = UserSettings.from_file(user_settings) + + s3_settings = user_settings_config.get_s3_settings(f"s3://{bucket}/{subfolder}") + + if not s3_settings: + raise ValueError("No S3 settings found for this bucket") + + # set the environment variables for S3 from the user settings + os.environ["aws_access_key_id"] = s3_settings["aws_access_key_id"] + os.environ["aws_secret_access_key"] = s3_settings["aws_secret_access_key"] + os.environ["aws_region_name"] = s3_settings["region_name"] + os.environ["aws_endpoint_url"] = s3_settings["endpoint_url"] + + client = boto3.client( + "s3", + **s3_settings, + config=Config(s3={"addressing_style": "path", "signature_version": "s3v4"}), + ) + + shutil.copytree(stac_catalog, "/tmp/catalog") + cat = pystac.read_file(os.path.join("/tmp/catalog", "catalog.json")) + + StacIO.set_default(CustomStacIO) + + for item in cat.get_items(): + for key, asset in item.get_assets().items(): + s3_path = os.path.normpath( + os.path.join(os.path.join(subfolder, item.id, asset.href)) + ) + logger.info(f"upload {asset.href} to s3://{bucket}/{s3_path}") + + upload_file_with_chunk_size( + client, + asset.get_absolute_href(), + bucket, + s3_path + ) + + asset.href = f"s3://{bucket}/{s3_path}" + item.add_asset(key, asset) + + cat.normalize_hrefs(f"s3://{bucket}/{subfolder}") + + for item in cat.get_items(): + # upload item to S3 + logger.info(f"upload {item.id} to s3://{bucket}/{subfolder}") + for index, link in enumerate(item.links): + if link.rel in ["collection"]: + logger.info("saving collection.json") + collection = link.target + collection.links = [] + pystac.write_file(link.target, dest_href="./collection.json") + item.links.pop(index) + item.set_collection(None) + if link.rel in ["root"]: + item.links.pop(index) + pystac.write_file(item, item.get_self_href()) + + # upload catalog to S3 + logger.info(f"upload catalog.json to s3://{bucket}/{subfolder}") + for index, link in enumerate(cat.links): + if link.rel in ["root", "collection"]: + cat.links.pop(index) + pystac.write_file(cat, cat.get_self_href()) + + logger.info("Done!") + + print(f"s3://{bucket}/{subfolder}/catalog.json", file=sys.stdout) +if __name__ == "__main__": + main() diff --git a/machine-learning-process/containers/stage-out/app/stac.py b/machine-learning-process/containers/stage-out/app/stac.py new file mode 100644 index 0000000..66709fc --- /dev/null +++ b/machine-learning-process/containers/stage-out/app/stac.py @@ -0,0 +1,77 @@ +import os +from urllib.parse import urlparse + +import boto3 +import botocore +from botocore.client import Config +from pystac.stac_io import DefaultStacIO + + +class CustomStacIO(DefaultStacIO): + """Custom STAC IO class that uses boto3 to read from S3.""" + + def __init__(self): + self.session = botocore.session.Session() + + def write_text(self, dest, txt, *args, **kwargs): + self.s3_client = self.session.create_client( + service_name="s3", + use_ssl=True, + aws_access_key_id=os.environ["aws_access_key_id"], + aws_secret_access_key=os.environ["aws_secret_access_key"], + region_name=os.environ["aws_region_name"], + endpoint_url=os.environ["aws_endpoint_url"], + config=Config(s3={"addressing_style": "path", "signature_version": "s3v4"}), + ) + + parsed = urlparse(dest) + if parsed.scheme == "s3": + self.s3_client.put_object( + Body=txt.encode("UTF-8"), + Bucket=parsed.netloc, + Key=parsed.path[1:], + ContentType="application/geo+json", + ) + else: + super().write_text(dest, txt, *args, **kwargs) + +import boto3 +from boto3.s3.transfer import TransferConfig +import os + +import boto3 +import os +import shutil +import logging +from boto3.s3.transfer import TransferConfig +from botocore.client import Config +from pystac import StacIO +import pystac + +# Assuming CustomStacIO is already defined somewhere +# from your_custom_module import CustomStacIO + +logger = logging.getLogger() +logger.setLevel(logging.INFO) + +def upload_file_with_chunk_size(client, file_path, bucket_name, s3_key, max_chunks=1000, default_chunk_size_mb=25): + # Get the size of the file + file_size = os.path.getsize(file_path) + + # Convert default chunk size from MB to bytes + default_chunk_size = default_chunk_size_mb * 1024 * 1024 + + # Calculate the optimal chunk size to ensure the number of chunks does not exceed max_chunks + optimal_chunk_size = min(default_chunk_size, file_size // max_chunks + 1) + + # Configure the transfer settings + config = TransferConfig(multipart_chunksize=optimal_chunk_size) + + # Upload the file + client.upload_file( + Filename=file_path, + Bucket=bucket_name, + Key=s3_key, + Config=config + ) + diff --git a/machine-learning-process/containers/stage-out/app/usersettings.py b/machine-learning-process/containers/stage-out/app/usersettings.py new file mode 100644 index 0000000..9f4e8ae --- /dev/null +++ b/machine-learning-process/containers/stage-out/app/usersettings.py @@ -0,0 +1,63 @@ +""" +This code defines a UserSettings class that has methods for reading JSON files, +matching regular expressions, and setting environment variables for an S3 service. +The set_s3_environment method sets environment variables for an S3 service based +on a given URL. +""" + +import json +import os +import re +from typing import Dict + + +class UserSettings: + """class for reading JSON files, matching regular expressions, + and setting environment variables for an S3 service""" + + def __init__(self, settings: Dict): + self.settings = settings + + @classmethod + def from_file(cls, file_path): + """create a UserSettings object from a JSON file""" + return cls(cls.read_json_file(file_path)) + + @staticmethod + def read_json_file(file_path): + """read a JSON file and return the contents as a dictionary""" + with open(file_path, "r", encoding="utf-8") as stream: + return json.load(stream) + + @staticmethod + def match_regex(regex, string): + """match a regular expression to a string and return the match object""" + return re.search(regex, string) + + @staticmethod + def set_s3_vars(s3_service): + """set environment variables for an S3 service""" + os.environ["AWS_ACCESS_KEY_ID"] = s3_service["AccessKey"] + os.environ["AWS_SECRET_ACCESS_KEY"] = s3_service["SecretKey"] + os.environ["AWS_DEFAULT_REGION"] = s3_service["Region"] + os.environ["AWS_REGION"] = s3_service["Region"] + os.environ["AWS_S3_ENDPOINT"] = s3_service["ServiceURL"] + + def set_s3_environment(self, url): + """set environment variables for an S3 service based on a given URL""" + for _, s3_service in self.settings["S3"]["Services"].items(): + if self.match_regex(s3_service["UrlPattern"], url): + self.set_s3_vars(s3_service) + break + + def get_s3_settings(self, url): + """return S3 settings based on a given URL""" + for _, s3_service in self.settings["S3"]["Services"].items(): + if self.match_regex(s3_service["UrlPattern"], url): + return { + "region_name": s3_service["Region"], + "endpoint_url": s3_service["ServiceURL"], + "aws_access_key_id": s3_service["AccessKey"], + "aws_secret_access_key": s3_service["SecretKey"], + } + return None diff --git a/machine-learning-process/containers/stage-out/requirements.txt b/machine-learning-process/containers/stage-out/requirements.txt new file mode 100644 index 0000000..281aba6 --- /dev/null +++ b/machine-learning-process/containers/stage-out/requirements.txt @@ -0,0 +1,5 @@ +boto3 +pystac +loguru +click +pyyaml \ No newline at end of file diff --git a/machine-learning-process/containers/stage-out/setup.cfg b/machine-learning-process/containers/stage-out/setup.cfg new file mode 100644 index 0000000..03162a4 --- /dev/null +++ b/machine-learning-process/containers/stage-out/setup.cfg @@ -0,0 +1,3 @@ +[metadata] +name=stage-out +version=0.3.4 diff --git a/machine-learning-process/containers/stage-out/setup.py b/machine-learning-process/containers/stage-out/setup.py new file mode 100644 index 0000000..85fdb58 --- /dev/null +++ b/machine-learning-process/containers/stage-out/setup.py @@ -0,0 +1,12 @@ +from setuptools import setup + +console_scripts = ["stage-out=app.main:main"] + +setup( + name="stage-out", + version="0.1", + description="The funniest joke in the world", + packages=["app"], + entry_points={"console_scripts": console_scripts}, + zip_safe=False, +) diff --git a/machine-learning-process/containers/validate-schema/Dockerfile b/machine-learning-process/containers/validate-schema/Dockerfile new file mode 100644 index 0000000..5c9c7a4 --- /dev/null +++ b/machine-learning-process/containers/validate-schema/Dockerfile @@ -0,0 +1,5 @@ +FROM python:3.11-slim + +WORKDIR /app + +RUN pip install jsonschema \ No newline at end of file diff --git a/machine-learning-process/files/bash-login b/machine-learning-process/files/bash-login new file mode 100644 index 0000000..5becb33 --- /dev/null +++ b/machine-learning-process/files/bash-login @@ -0,0 +1 @@ +source /workspace/.bashrc \ No newline at end of file diff --git a/machine-learning-process/files/bash-rc b/machine-learning-process/files/bash-rc new file mode 100644 index 0000000..ec5a11e --- /dev/null +++ b/machine-learning-process/files/bash-rc @@ -0,0 +1,5 @@ +alias ll="ls -l" + +alias aws="aws --endpoint-url=http://eoap-event-driven-localstack:4566" + +source /workspace/.venv/bin/activate \ No newline at end of file diff --git a/machine-learning-process/files/init.sh b/machine-learning-process/files/init.sh new file mode 100644 index 0000000..fe17126 --- /dev/null +++ b/machine-learning-process/files/init.sh @@ -0,0 +1,25 @@ +cd /workspace + +#temp +mkdir event-driven-with-argo +git clone https://github.com/eoap/event-driven-with-argo.git +cd event-driven-with-argo +git checkout develop + +code-server --install-extension ms-python.python +code-server --install-extension ms-toolsai.jupyter +ln -s /workspace/.local/share/code-server/extensions /workspace/extensions + +mkdir -p /workspace/User/ +echo '{"workbench.colorTheme": "Visual Studio Dark"}' > /workspace/User/settings.json + +python -m venv /workspace/.venv +source /workspace/.venv/bin/activate +/workspace/.venv/bin/python -m pip install --no-cache-dir ipykernel requests pyyaml boto3==1.35.23 loguru redis pystac_client +/workspace/.venv/bin/python -m ipykernel install --user --name zoo_env --display-name "Python (Event Driven with Argo)" + +export AWS_DEFAULT_REGION="us-east-1" +export AWS_ACCESS_KEY_ID="test" +export AWS_SECRET_ACCESS_KEY="test" +aws s3 mb s3://results --endpoint-url=http://eoap-event-driven-localstack:4566 +aws s3 mb s3://workflows --endpoint-url=http://eoap-event-driven-localstack:4566 \ No newline at end of file diff --git a/machine-learning-process/skaffold.yaml b/machine-learning-process/skaffold.yaml new file mode 100644 index 0000000..aaf7ca6 --- /dev/null +++ b/machine-learning-process/skaffold.yaml @@ -0,0 +1,157 @@ +apiVersion: skaffold/v4beta9 +kind: Config +metadata: + name: eoap-event-driven +build: + artifacts: + - image: stageout + context: containers/stage-out + docker: + dockerfile: Dockerfile + - image: validateschema + context: containers/validate-schema + docker: + dockerfile: Dockerfile + +deploy: + helm: + releases: + # argo workflows + - name: eoap-argo-workflows + remoteChart: argo/argo-workflows + namespace: eoap-event-driven + createNamespace: true + wait: true + setValues: + crds.install: true + server.authModes: [server] + singleNamespace: true + workflow: + serviceAccount: + create: true + name: "argo" + rbac: + create: true + controller: + workflowNamespaces: + - eoap-event-driven + workflowDefaults: + spec: + serviceAccountName: argo + podGC: + strategy: OnWorkflowCompletion + deleteDelayDuration: 120s + rbac.writeConfigMaps: true + securityContext.runAsNonRoot: false + artifactRepository.s3: + bucket: workflows + endpoint: eoap-event-driven-localstack:4566 + insecure: true + accessKeySecret: + name: localstack-cred + key: accesskey + secretKeySecret: + name: localstack-cred + key: secretkey + # argo events + - name: eoap-argo-events + remoteChart: argo/argo-events + namespace: eoap-event-driven + wait: true + createNamespace: true + setValues: + crds.install: true + controller: + rbac: + enabled: true + namespaced: true + managedNamespace: eoap-event-driven + + + - name: eoap-redis + chartPath: ./charts/redis + namespace: eoap-event-driven + createNamespace: true + setValues: + maxMemory: 2mb + maxMemoryPolicy: allkeys-lru + port: 6379 + + # local chart for event bus/source/sensor + - name: eoap-event-driven + chartPath: ./charts/event-driven + namespace: eoap-event-driven + createNamespace: true + wait: true + setValues: + jetstreamReplicas: 3 + jetstreamVolumeSize: 1Gi + streamName: STREAM + + # argo cwl runner + - name: argo-cwl-runner + chartPath: ./charts/argo-cwl-runner + namespace: eoap-event-driven + valuesFiles: + - ./charts/argo-cwl-runner/values.yaml + setValueTemplates: + stageOutImage: "{{.IMAGE_NAME_stageout}}:{{.IMAGE_TAG_stageout}}" + setFiles: { + userSettings: ./charts/argo-cwl-runner/files/user-settings.json + } + # argo water bodies + - name: argo-water-bodies + chartPath: ./charts/argo-water-bodies + namespace: eoap-event-driven + valuesFiles: + - ./charts/argo-water-bodies/values.yaml + setFiles: { + appPackage: ./charts/argo-water-bodies/files/app-package.json, + inputSchema: ./charts/argo-water-bodies/files/input-parameters-schema.json + } + setValues: + storageClassRWX: "standard" + + - name: eoap-event-driven-localstack + remoteChart: localstack/localstack + namespace: eoap-event-driven + createNamespace: true + wait: true + setValues: + service.type: ClusterIP + + - name: eoap-event-driven-coder + chartPath: ../charts/coder + namespace: eoap-event-driven + createNamespace: true + wait: true + setValues: + coder.coderImage: eoepca/pde-code-server:1.0.0 + coder.workspace: event-driven-with-argo + coderstorageClassName: standard + coder.workspaceStorage: 10Gi + coderResources.limits.cpu: '2' + coderResources.limits.memory: '6442450944' + coderResources.requests.cpu: '1' + coderResources.requests.memory: '4294967296' + calrissian.enabled: true + setFiles: { + initScript: ./files/init.sh, + bashrcScript: ./files/bash-rc, + bashloginScript: ./files/bash-login + } + + +portForward: + - resourceType: service + resourceName: eoap-argo-workflows-server + namespace: eoap-event-driven + address: localhost + port: 2746 + localPort: 2746 + - resourceType: service + resourceName: code-server-service + namespace: eoap-event-driven + address: localhost + port: 8080 + localPort: 8000 \ No newline at end of file From f2aecb1869a5ead6266f965fa668c36e54f6e6a1 Mon Sep 17 00:00:00 2001 From: Fabrice Brito Date: Fri, 28 Feb 2025 16:01:37 +0100 Subject: [PATCH 10/61] adds config to dask gateway --- charts/coder/templates/coder.yaml | 17 ++- .../coder/templates/daskgateway-config.yaml | 13 ++ charts/coder/values.yaml | 7 +- dask-gateway/Dockerfile.client | 33 +++++ dask-gateway/Dockerfile.coder | 125 ++++++++++++++++++ dask-gateway/Dockerfile.worker | 8 ++ dask-gateway/entrypoint.sh | 37 ++++++ dask-gateway/files/bash-login | 1 + dask-gateway/files/bash-rc | 5 + dask-gateway/files/init.sh | 42 ++++++ dask-gateway/requirements.txt | 12 ++ dask-gateway/skaffold.yaml | 83 ++++++++++++ dask-gateway/values.yaml | 11 ++ dask-gateway/wait-for-it.sh | 23 ++++ 14 files changed, 415 insertions(+), 2 deletions(-) create mode 100644 charts/coder/templates/daskgateway-config.yaml create mode 100644 dask-gateway/Dockerfile.client create mode 100644 dask-gateway/Dockerfile.coder create mode 100644 dask-gateway/Dockerfile.worker create mode 100644 dask-gateway/entrypoint.sh create mode 100644 dask-gateway/files/bash-login create mode 100644 dask-gateway/files/bash-rc create mode 100644 dask-gateway/files/init.sh create mode 100644 dask-gateway/requirements.txt create mode 100644 dask-gateway/skaffold.yaml create mode 100644 dask-gateway/values.yaml create mode 100755 dask-gateway/wait-for-it.sh diff --git a/charts/coder/templates/coder.yaml b/charts/coder/templates/coder.yaml index 163ff06..fa7524f 100644 --- a/charts/coder/templates/coder.yaml +++ b/charts/coder/templates/coder.yaml @@ -66,6 +66,10 @@ spec: mountPath: /etc/calrissian/pod-node-selector.yaml subPath: pod-node-selector.yaml {{ end }} + {{- if .Values.daskGateway.enabled }} + - name: dask-gateway-config + mountPath: /etc/dask + {{- end }} env: - name: XDG_CONFIG_HOME value: /workspace/.local @@ -86,7 +90,13 @@ spec: valueFrom: fieldRef: fieldPath: metadata.name - {{ end }} + {{- end }} + {{- if .Values.daskGateway.enabled }} + - name: DASK_GATEWAY_ADDRESS + value: "{{ .Values.daskGateway.daskGatewayUrl }}" + - name: DASK_IMAGE + value: "{{ .Values.daskGateway.image }}" + {{- end }} resources: limits: cpu: {{ .Values.coderResources.limits.cpu }} @@ -119,6 +129,11 @@ spec: name: node-selectors defaultMode: 420 {{ end }} + {{- if .Values.daskGateway.enabled }} + - name: dask-gateway-config + configMap: + name: dask-gateway-config + {{- end }} --- apiVersion: v1 kind: PersistentVolumeClaim diff --git a/charts/coder/templates/daskgateway-config.yaml b/charts/coder/templates/daskgateway-config.yaml new file mode 100644 index 0000000..c56e6e9 --- /dev/null +++ b/charts/coder/templates/daskgateway-config.yaml @@ -0,0 +1,13 @@ +{{- if .Values.daskGateway.enabled }} +kind: ConfigMap +apiVersion: v1 +metadata: + name: dask-gateway-config +data: + gateway.yaml: | + gateway: + address: "{{ .Values.daskGateway.daskGatewayUrl }}" +{{- end }} + + + \ No newline at end of file diff --git a/charts/coder/values.yaml b/charts/coder/values.yaml index 820f35e..a2aae37 100644 --- a/charts/coder/values.yaml +++ b/charts/coder/values.yaml @@ -23,4 +23,9 @@ coderResources: memory: '4294967296' skaffold: - enabled: False \ No newline at end of file + enabled: False + +daskGateway: + enabled: False + daskGatewayUrl: "" + image: "" \ No newline at end of file diff --git a/dask-gateway/Dockerfile.client b/dask-gateway/Dockerfile.client new file mode 100644 index 0000000..dddaead --- /dev/null +++ b/dask-gateway/Dockerfile.client @@ -0,0 +1,33 @@ +FROM docker.io/library/python:3.10.12@sha256:bac3a0e0d16125977e351c861e2f4b12ecafeaa6f72431dc970d0b9155103232 + +# Install necessary packages +RUN apt-get update && apt-get install -y --no-install-recommends \ + git \ + curl \ + wget \ + vim \ + tree \ + sudo && \ + rm -rf /var/lib/apt/lists + +# Create a user named 'dask-user' with sudo privileges +RUN useradd -m dask-user && \ + echo "dask-user:password" | chpasswd && \ + adduser dask-user sudo && \ + echo "dask-user ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers + + +ADD requirements.txt /tmp/requirements.txt + +RUN pip install --no-cache-dir -r /tmp/requirements.txt && \ + rm -rf /tmp/requirements.txt + +# Set the user to 'dask-user' +USER dask-user + +# Add alias for ll="ls -l" to the bash profile of dask-user +RUN echo 'alias ll="ls -l"' >> /home/dask-user/.bashrc + +WORKDIR /home/dask-user + +ENTRYPOINT [] diff --git a/dask-gateway/Dockerfile.coder b/dask-gateway/Dockerfile.coder new file mode 100644 index 0000000..c19f367 --- /dev/null +++ b/dask-gateway/Dockerfile.coder @@ -0,0 +1,125 @@ +FROM docker.io/library/python:3.10.12@sha256:bac3a0e0d16125977e351c861e2f4b12ecafeaa6f72431dc970d0b9155103232 + +RUN \ + apt-get update && \ + apt-get install -y \ + build-essential \ + curl \ + gcc \ + vim \ + tree \ + file + +RUN \ + echo "**** install node repo ****" && \ + curl -s https://deb.nodesource.com/gpgkey/nodesource.gpg.key | apt-key add - && \ + echo 'deb https://deb.nodesource.com/node_14.x jammy main' \ + > /etc/apt/sources.list.d/nodesource.list && \ + echo "**** install build dependencies ****" && \ + apt-get update && \ + apt-get install -y \ + nodejs + +RUN \ + echo "**** install runtime dependencies ****" && \ + apt-get install -y \ + git \ + jq \ + libatomic1 \ + nano \ + net-tools \ + sudo \ + podman \ + wget \ + python3 \ + python3-pip + +RUN \ + echo "**** install code-server ****" && \ + if [ -z ${CODE_RELEASE+x} ]; then \ + CODE_RELEASE=$(curl -sX GET https://api.github.com/repos/coder/code-server/releases/latest \ + | awk '/tag_name/{print $4;exit}' FS='[""]' | sed 's|^v||'); \ + fi && \ + mkdir -p /app/code-server && \ + curl -o \ + /tmp/code-server.tar.gz -L \ + "https://github.com/coder/code-server/releases/download/v${CODE_RELEASE}/code-server-${CODE_RELEASE}-linux-amd64.tar.gz" && \ + tar xf /tmp/code-server.tar.gz -C \ + /app/code-server --strip-components=1 && \ + echo "**** patch 4.0.2 ****" && \ + if [ "${CODE_RELEASE}" = "4.0.2" ] && [ "$(uname -m)" != "x86_64" ]; then \ + cd /app/code-server && \ + npm i --production @node-rs/argon2; \ + fi && \ + echo "**** clean up ****" && \ + apt-get purge --auto-remove -y \ + build-essential \ + nodejs && \ + apt-get clean && \ + rm -rf \ + /config/* \ + /tmp/* \ + /var/lib/apt/lists/* \ + /var/tmp/* \ + /etc/apt/sources.list.d/nodesource.list + +ENV USER=jovyan \ + UID=1001 \ + GID=100 \ + HOME=/workspace \ + PATH=/opt/conda/bin:/app/code-server/bin/:$PATH:/app/code-server/ + + +# RUN \ +# echo "**** install conda ****" && \ +# wget https://repo.anaconda.com/miniconda/Miniconda3-py39_4.12.0-Linux-x86_64.sh -O miniconda.sh -q && \ +# sh miniconda.sh -b -p /opt/conda && \ +# conda install -n base -c conda-forge mamba && \ +# conda config --system --append channels conda-forge && \ +# conda config --system --append channels terradue && \ +# conda config --system --append channels eoepca && \ +# conda config --system --append channels r && \ +# conda config --system --set auto_update_conda false && \ +# conda config --system --set show_channel_urls true && \ +# conda config --system --set channel_priority "flexible" + + +# RUN \ +# mamba install -n base cwltool cwl-wrapper==0.12.2 nodejs && \ +# mamba clean -a + +RUN \ + echo "**** install yq, aws cli ****" && \ + VERSION="v4.12.2" && \ + BINARY="yq_linux_amd64" && \ + wget --quiet https://github.com/mikefarah/yq/releases/download/${VERSION}/${BINARY}.tar.gz -O - |\ + tar xz && mv ${BINARY} /usr/bin/yq && \ + pip install awscli && \ + pip install awscli-plugin-endpoint + +RUN \ + echo "**** install jupyter-hub native proxy ****" && \ + pip install jhsingle-native-proxy>=0.0.9 && \ + echo "**** install bash kernel ****" && \ + pip install bash_kernel && \ + python -m bash_kernel.install + +RUN \ + echo "**** adds user jovyan ****" && \ + useradd -m -s /bin/bash -N -u $UID $USER + +COPY entrypoint.sh /opt/entrypoint.sh + +RUN chmod +x /opt/entrypoint.sh + +# RUN chown -R jovyan:100 /opt/conda + +# RUN \ +# echo "**** required by cwltool docker pull even if running with --podman ****" && \ +# ln -s /usr/bin/podman /usr/bin/docker + +ENTRYPOINT ["/opt/entrypoint.sh"] + +EXPOSE 8888 + +USER jovyan \ No newline at end of file diff --git a/dask-gateway/Dockerfile.worker b/dask-gateway/Dockerfile.worker new file mode 100644 index 0000000..72b95d8 --- /dev/null +++ b/dask-gateway/Dockerfile.worker @@ -0,0 +1,8 @@ +FROM docker.io/library/python:3.10.12@sha256:bac3a0e0d16125977e351c861e2f4b12ecafeaa6f72431dc970d0b9155103232 + +ADD requirements.txt /tmp/requirements.txt + +RUN pip install --no-cache-dir -r /tmp/requirements.txt && \ + rm -rf /tmp/requirements.txt + +ENTRYPOINT [] diff --git a/dask-gateway/entrypoint.sh b/dask-gateway/entrypoint.sh new file mode 100644 index 0000000..f5b1057 --- /dev/null +++ b/dask-gateway/entrypoint.sh @@ -0,0 +1,37 @@ +#!/bin/bash + +collect_port=0 +port="8888" +delim='=' + +for var in "$@" +do + echo "$var" + + if [ "$collect_port" == "1" ]; then + echo "Collecting external port $var" + port=$var + collect_port=0 + fi + + splitarg=${var%%$delim*} + + if [ "$splitarg" == "--port" ]; then + if [ ${#splitarg} == ${#var} ]; then + collect_port=1 + else + port=${var#*$delim} + echo "Setting external port $port" + fi + fi +done + +destport=$((port + 1)) + +echo "Using internal port $destport" + +if [ -z "$CODE_SERVER_WS" ]; then + CODE_SERVER_WS="/workspace" +fi + +jhsingle-native-proxy --port $port --destport $destport code-server {--}auth none {--}bind-addr 0.0.0.0:$destport {--}user-data-dir /workspace $CODE_SERVER_WS \ No newline at end of file diff --git a/dask-gateway/files/bash-login b/dask-gateway/files/bash-login new file mode 100644 index 0000000..5becb33 --- /dev/null +++ b/dask-gateway/files/bash-login @@ -0,0 +1 @@ +source /workspace/.bashrc \ No newline at end of file diff --git a/dask-gateway/files/bash-rc b/dask-gateway/files/bash-rc new file mode 100644 index 0000000..00dd390 --- /dev/null +++ b/dask-gateway/files/bash-rc @@ -0,0 +1,5 @@ +alias ll="ls -l" + +alias aws="aws --endpoint-url=http://eoap-mastering-app-package-localstack:4566" + +source /workspace/.venv/bin/activate \ No newline at end of file diff --git a/dask-gateway/files/init.sh b/dask-gateway/files/init.sh new file mode 100644 index 0000000..1a17a5e --- /dev/null +++ b/dask-gateway/files/init.sh @@ -0,0 +1,42 @@ +#!/bin/bash + +set -x + +cd /workspace + +git clone 'https://github.com/eoap/dask-app-package.git' + +code-server --install-extension ms-python.python +code-server --install-extension redhat.vscode-yaml +code-server --install-extension sbg-rabix.benten-cwl +code-server --install-extension ms-toolsai.jupyter + +ln -s /workspace/.local/share/code-server/extensions /workspace/extensions + +mkdir -p /workspace/User/ + +echo '{"workbench.colorTheme": "Visual Studio Dark"}' > /workspace/User/settings.json + +python -m venv /workspace/.venv +source /workspace/.venv/bin/activate +/workspace/.venv/bin/python -m pip install --no-cache-dir numpy==1.26.3 dask==2024.1.0 distributed==2024.1.0 dask-gateway==2024.1.0 pystac bokeh rioxarray==0.18.1 loguru==0.7.3 odc-stac[botocore]==0.3.10 stackstac==0.5.1 pystac-client planetary-computer +res=$? +if [ $res -ne 0 ]; then + echo "Failed to install packages" + exit $res +fi + +/workspace/.venv/bin/python -m ipykernel install --user --name dask-gateway --display-name "Python (Dask Application Package)" + +echo "**** install kubectl ****" +curl -s -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" +chmod +x kubectl +mv ./kubectl /workspace/.venv/bin/kubectl + +export AWS_DEFAULT_REGION="us-east-1" + +export AWS_ACCESS_KEY_ID="test" + +export AWS_SECRET_ACCESS_KEY="test" + +aws s3 mb s3://results --endpoint-url=http://eoap-dask-gateway-localstack:4566 \ No newline at end of file diff --git a/dask-gateway/requirements.txt b/dask-gateway/requirements.txt new file mode 100644 index 0000000..0914146 --- /dev/null +++ b/dask-gateway/requirements.txt @@ -0,0 +1,12 @@ +numpy==1.26.3 +dask==2024.1.0 +distributed==2024.1.0 +dask-gateway==2024.1.0 +pystac +bokeh==3.6.2 +rioxarray==0.18.1 +loguru==0.7.3 +odc-stac[botocore]==0.3.10 +stackstac==0.5.1 +pystac-client +planetary-computer \ No newline at end of file diff --git a/dask-gateway/skaffold.yaml b/dask-gateway/skaffold.yaml new file mode 100644 index 0000000..6b3162f --- /dev/null +++ b/dask-gateway/skaffold.yaml @@ -0,0 +1,83 @@ +apiVersion: skaffold/v4beta9 +kind: Config +metadata: + name: eoap-dask-gateway + +build: + artifacts: + - image: worker + context: . + docker: + dockerfile: Dockerfile.worker + - image: daskclient + context: . + docker: + dockerfile: Dockerfile.client + - image: coder + context: . + docker: + dockerfile: Dockerfile.coder + +deploy: + helm: + releases: + - name: dask-gateway + remoteChart: https://helm.dask.org/dask-gateway-2024.1.0.tgz + namespace: eoap-dask-gateway + createNamespace: true + valuesFiles: + - values.yaml + setValueTemplates: + gateway.backend.image.name: "{{.IMAGE_NAME_worker}}" + gateway.backend.image.tag: "{{.IMAGE_TAG_worker}}" + traefik.service.type: "ClusterIP" + + - name: eoap-dask-gateway + chartPath: ../charts/coder + namespace: eoap-dask-gateway + createNamespace: true + setValueTemplates: + coder.coderImage: "{{.IMAGE_NAME_coder}}" + daskGateway.image: "{{.IMAGE_NAME_worker}}:{{.IMAGE_TAG_worker}}" + setValues: + coder.workspace: dask-app-package + coderstorageClassName: standard + coder.workspaceStorage: 10Gi + coderResources.limits.cpu: '2' + coderResources.limits.memory: '6442450944' + coderResources.requests.cpu: '1' + coderResources.requests.memory: '4294967296' + calrissian.enabled: true + daskGateway.enabled: true + daskGateway.daskGatewayUrl: "http://traefik-dask-gateway.eoap-dask-gateway.svc.cluster.local:80" + setFiles: { + initScript: ./files/init.sh, + bashrcScript: ./files/bash-rc, + bashloginScript: ./files/bash-login + } + - name: eoap-dask-gateway-localstack + remoteChart: localstack/localstack + namespace: eoap-dask-gateway + createNamespace: true + setValues: + service.type: ClusterIP + hooks: + before: [] + after: + - host: + command: ["sh", "-c", "./wait-for-it.sh"] + os: [darwin, linux] + +portForward: + - resourceType: service + resourceName: code-server-service + namespace: eoap-dask-gateway + address: localhost + port: 8080 + localPort: 8000 + - resourceType: service + resourceName: traefik-dask-gateway + namespace: eoap-dask-gateway + address: localhost + port: 80 + localPort: 8001 \ No newline at end of file diff --git a/dask-gateway/values.yaml b/dask-gateway/values.yaml new file mode 100644 index 0000000..e1d580a --- /dev/null +++ b/dask-gateway/values.yaml @@ -0,0 +1,11 @@ +gateway: + extraConfig: + dask_gateway_config.py: | + c = get_config() + from dask_gateway_server.options import Options, String, Integer, Float + c.Backend.cluster_options = Options( + Float("worker_cores_limit", default=1, label="Worker Cores Limit"), + Float("worker_cores", default=1, label="Worker Cores"), + String("worker_memory", default="1 G", label="Worker Memory"), + String("image", default="daskgateway/dask-worker:latest", label="Worker Image") + ) \ No newline at end of file diff --git a/dask-gateway/wait-for-it.sh b/dask-gateway/wait-for-it.sh new file mode 100755 index 0000000..8f5d851 --- /dev/null +++ b/dask-gateway/wait-for-it.sh @@ -0,0 +1,23 @@ +#!/bin/bash + + +# Label to identify the deployment +LABEL="app.kubernetes.io/name=dask-gateway" +NAMESPACE="eoap-dask-gateway" + +# Check if the deployment is running +while true; do + # Get the status of the deployment with the specified label + DEPLOYMENT_STATUS=$(kubectl get deployment traefik-dask-gateway -n $NAMESPACE -o "jsonpath={.status.availableReplicas}") + echo "Deployment replicas: $DEPLOYMENT_STATUS" + # Check if the status is 'True' + if [ "$DEPLOYMENT_STATUS" = "1" ]; then + echo "Deployment with label $LABEL is running" + break + else + echo "Waiting for deployment to be running..." + fi + + # Wait for a short period before checking again + sleep 5 +done \ No newline at end of file From 986e1284a60b18f02bec3b369adaf0dbb0225880 Mon Sep 17 00:00:00 2001 From: Fabrice Brito Date: Mon, 3 Mar 2025 12:15:34 +0100 Subject: [PATCH 11/61] updates "how-to" config to support taskfiles --- how-to/files/init.sh | 14 ++++++++++++-- how-to/skaffold.yaml | 1 + ogc-api-processes-with-zoo/skaffold.yaml | 4 ++++ 3 files changed, 17 insertions(+), 2 deletions(-) diff --git a/how-to/files/init.sh b/how-to/files/init.sh index a7ce028..6103190 100644 --- a/how-to/files/init.sh +++ b/how-to/files/init.sh @@ -17,6 +17,16 @@ echo '{"workbench.colorTheme": "Visual Studio Dark"}' > /workspace/User/settings python -m venv /workspace/.venv source /workspace/.venv/bin/activate -/workspace/.venv/bin/python -m pip install --no-cache-dir stactools rasterio requests stac-asset click-logging tabulate tqdm pystac-client ipykernel loguru scikit-image rio_stac +/workspace/.venv/bin/python -m pip install --no-cache-dir stactools rasterio requests stac-asset click-logging tabulate tqdm pystac-client ipykernel loguru scikit-image rio_stac calrissian tomlq -/workspace/.venv/bin/python -m ipykernel install --user --name cwl_how_to_env --display-name "Python (CWL How-To's)" \ No newline at end of file +/workspace/.venv/bin/python -m ipykernel install --user --name cwl_how_to_env --display-name "Python (CWL How-To's)" + +curl -s -L https://github.com/go-task/task/releases/download/v3.41.0/task_linux_amd64.tar.gz | tar -xzvf - -C /workspace/.venv/bin/ +chmod +x /workspace/.venv/bin/task + +curl -s -L https://storage.googleapis.com/skaffold/releases/latest/skaffold-linux-amd64 > /workspace/.venv/bin/skaffold +chmod +x /workspace/.venv/bin/skaffold + +curl -s -LO https://github.com/mikefarah/yq/releases/download/v4.45.1/yq_linux_amd64.tar.gz +tar -xvf yq_linux_amd64.tar.gz +mv yq_linux_amd64 /workspace/.venv/bin/yq \ No newline at end of file diff --git a/how-to/skaffold.yaml b/how-to/skaffold.yaml index fa81fc6..1e61065 100644 --- a/how-to/skaffold.yaml +++ b/how-to/skaffold.yaml @@ -24,6 +24,7 @@ deploy: coderResources.limits.memory: '6442450944' coderResources.requests.cpu: '1' coderResources.requests.memory: '4294967296' + skaffold.enabled: True setFiles: { initScript: ./files/init.sh, bashrcScript: ./files/bash-rc, diff --git a/ogc-api-processes-with-zoo/skaffold.yaml b/ogc-api-processes-with-zoo/skaffold.yaml index 7a80dfa..5f50664 100644 --- a/ogc-api-processes-with-zoo/skaffold.yaml +++ b/ogc-api-processes-with-zoo/skaffold.yaml @@ -26,6 +26,10 @@ deploy: aws_secret_access_key=test aws_access_key_id=test endpoint_url=http://eoap-zoo-project-localstack.eoap-zoo-project.svc.cluster.local:4566 + #customConfig.main.pod_env_vars: |- + # C="1" + #customConfig.main.pod_node_selector: |- + # "kubernetes.io/hostname"=minikube - name: eoap-zoo-project-coder chartPath: ../charts/coder namespace: eoap-zoo-project From bcaaaaaa6e1485b35394e475ab4c929e232edca6 Mon Sep 17 00:00:00 2001 From: pmembari Date: Mon, 3 Mar 2025 18:00:53 +0100 Subject: [PATCH 12/61] feat: update skaffold and init.sh --- .../charts/argo-water-bodies/.helmignore | 23 -- .../charts/argo-water-bodies/Chart.yaml | 24 -- .../argo-water-bodies/files/app-package.json | 384 ------------------ .../files/input-parameters-schema.json | 34 -- .../charts/argo-water-bodies/files/prepare.sh | 15 - .../argo-water-bodies/files/validation.py | 20 - .../templates/app-package.yaml | 6 - .../templates/argo-water-bodies.yaml | 216 ---------- .../templates/input-parameters-schema.yaml | 6 - .../templates/semaphore.yaml | 7 - .../charts/argo-water-bodies/values.yaml | 8 - .../charts/event-driven/Chart.yaml | 6 - .../event-driven/templates/event-bus.yaml | 18 - .../templates/event-sensor-log.yaml | 16 - .../event-driven/templates/event-sensor.yaml | 46 --- .../event-driven/templates/event-source.yaml | 13 - .../templates/local-stack-secret.yaml | 10 - .../event-driven/templates/role-binding.yaml | 11 - .../charts/event-driven/templates/roles.yaml | 52 --- .../charts/event-driven/values.yaml | 4 - .../charts/redis/Chart.yaml | 6 - .../charts/redis/templates/config.yaml | 8 - .../redis/templates/redis-deployment.yaml | 42 -- .../charts/redis/templates/redis-service.yaml | 12 - .../charts/redis/values.yaml | 3 - machine-learning-process/files/init.sh | 6 +- machine-learning-process/skaffold.yaml | 117 ++---- 27 files changed, 42 insertions(+), 1071 deletions(-) delete mode 100644 machine-learning-process/charts/argo-water-bodies/.helmignore delete mode 100644 machine-learning-process/charts/argo-water-bodies/Chart.yaml delete mode 100644 machine-learning-process/charts/argo-water-bodies/files/app-package.json delete mode 100644 machine-learning-process/charts/argo-water-bodies/files/input-parameters-schema.json delete mode 100644 machine-learning-process/charts/argo-water-bodies/files/prepare.sh delete mode 100644 machine-learning-process/charts/argo-water-bodies/files/validation.py delete mode 100644 machine-learning-process/charts/argo-water-bodies/templates/app-package.yaml delete mode 100644 machine-learning-process/charts/argo-water-bodies/templates/argo-water-bodies.yaml delete mode 100644 machine-learning-process/charts/argo-water-bodies/templates/input-parameters-schema.yaml delete mode 100644 machine-learning-process/charts/argo-water-bodies/templates/semaphore.yaml delete mode 100644 machine-learning-process/charts/argo-water-bodies/values.yaml delete mode 100644 machine-learning-process/charts/event-driven/Chart.yaml delete mode 100644 machine-learning-process/charts/event-driven/templates/event-bus.yaml delete mode 100644 machine-learning-process/charts/event-driven/templates/event-sensor-log.yaml delete mode 100644 machine-learning-process/charts/event-driven/templates/event-sensor.yaml delete mode 100644 machine-learning-process/charts/event-driven/templates/event-source.yaml delete mode 100644 machine-learning-process/charts/event-driven/templates/local-stack-secret.yaml delete mode 100644 machine-learning-process/charts/event-driven/templates/role-binding.yaml delete mode 100644 machine-learning-process/charts/event-driven/templates/roles.yaml delete mode 100644 machine-learning-process/charts/event-driven/values.yaml delete mode 100644 machine-learning-process/charts/redis/Chart.yaml delete mode 100644 machine-learning-process/charts/redis/templates/config.yaml delete mode 100644 machine-learning-process/charts/redis/templates/redis-deployment.yaml delete mode 100644 machine-learning-process/charts/redis/templates/redis-service.yaml delete mode 100644 machine-learning-process/charts/redis/values.yaml diff --git a/machine-learning-process/charts/argo-water-bodies/.helmignore b/machine-learning-process/charts/argo-water-bodies/.helmignore deleted file mode 100644 index 0e8a0eb..0000000 --- a/machine-learning-process/charts/argo-water-bodies/.helmignore +++ /dev/null @@ -1,23 +0,0 @@ -# Patterns to ignore when building packages. -# This supports shell glob matching, relative path matching, and -# negation (prefixed with !). Only one pattern per line. -.DS_Store -# Common VCS dirs -.git/ -.gitignore -.bzr/ -.bzrignore -.hg/ -.hgignore -.svn/ -# Common backup files -*.swp -*.bak -*.tmp -*.orig -*~ -# Various IDEs -.project -.idea/ -*.tmproj -.vscode/ diff --git a/machine-learning-process/charts/argo-water-bodies/Chart.yaml b/machine-learning-process/charts/argo-water-bodies/Chart.yaml deleted file mode 100644 index a83b246..0000000 --- a/machine-learning-process/charts/argo-water-bodies/Chart.yaml +++ /dev/null @@ -1,24 +0,0 @@ -apiVersion: v2 -name: argo-water-bodies -description: A Helm chart for Kubernetes - -# A chart can be either an 'application' or a 'library' chart. -# -# Application charts are a collection of templates that can be packaged into versioned archives -# to be deployed. -# -# Library charts provide useful utilities or functions for the chart developer. They're included as -# a dependency of application charts to inject those utilities and functions into the rendering -# pipeline. Library charts do not define any templates and therefore cannot be deployed. -type: application - -# This is the chart version. This version number should be incremented each time you make changes -# to the chart and its templates, including the app version. -# Versions are expected to follow Semantic Versioning (https://semver.org/) -version: 0.1.0 - -# This is the version number of the application being deployed. This version number should be -# incremented each time you make changes to the application. Versions are not expected to -# follow Semantic Versioning. They should reflect the version the application is using. -# It is recommended to use it with quotes. -appVersion: "1.16.0" diff --git a/machine-learning-process/charts/argo-water-bodies/files/app-package.json b/machine-learning-process/charts/argo-water-bodies/files/app-package.json deleted file mode 100644 index bc09f22..0000000 --- a/machine-learning-process/charts/argo-water-bodies/files/app-package.json +++ /dev/null @@ -1,384 +0,0 @@ -{ - "cwlVersion": "v1.0", - "$namespaces": { - "s": "https://schema.org/" - }, - "s:softwareVersion": "1.0.0", - "schemas": [ - "http://schema.org/version/9.0/schemaorg-current-http.rdf" - ], - "$graph": [ - { - "class": "Workflow", - "id": "main", - "label": "Water bodies detection based on NDWI and otsu threshold", - "doc": "Water bodies detection based on NDWI and otsu threshold applied to Sentinel-2 COG STAC items", - "requirements": [ - { - "class": "ScatterFeatureRequirement" - }, - { - "class": "SubworkflowFeatureRequirement" - } - ], - "inputs": { - "aoi": { - "label": "area of interest", - "doc": "area of interest as a bounding box", - "type": "string" - }, - "epsg": { - "label": "EPSG code", - "doc": "EPSG code", - "type": "string", - "default": "EPSG:4326" - }, - "stac_items": { - "label": "Sentinel-2 STAC items", - "doc": "list of Sentinel-2 COG STAC items", - "type": "string[]" - }, - "bands": { - "label": "bands used for the NDWI", - "doc": "bands used for the NDWI", - "type": "string[]", - "default": [ - "green", - "nir" - ] - } - }, - "outputs": [ - { - "id": "stac_catalog", - "outputSource": [ - "node_stac/stac_catalog" - ], - "type": "Directory" - } - ], - "steps": { - "node_water_bodies": { - "run": "#detect_water_body", - "in": { - "item": "stac_items", - "aoi": "aoi", - "epsg": "epsg", - "bands": "bands" - }, - "out": [ - "detected_water_body" - ], - "scatter": "item", - "scatterMethod": "dotproduct" - }, - "node_stac": { - "run": "#stac", - "in": { - "item": "stac_items", - "rasters": { - "source": "node_water_bodies/detected_water_body" - } - }, - "out": [ - "stac_catalog" - ] - } - } - }, - { - "class": "Workflow", - "id": "detect_water_body", - "label": "Water body detection based on NDWI and otsu threshold", - "doc": "Water body detection based on NDWI and otsu threshold", - "requirements": [ - { - "class": "ScatterFeatureRequirement" - } - ], - "inputs": { - "aoi": { - "doc": "area of interest as a bounding box", - "type": "string" - }, - "epsg": { - "doc": "EPSG code", - "type": "string", - "default": "EPSG:4326" - }, - "bands": { - "doc": "bands used for the NDWI", - "type": "string[]" - }, - "item": { - "doc": "STAC item", - "type": "string" - } - }, - "outputs": [ - { - "id": "detected_water_body", - "outputSource": [ - "node_otsu/binary_mask_item" - ], - "type": "File" - } - ], - "steps": { - "node_crop": { - "run": "#crop", - "in": { - "item": "item", - "aoi": "aoi", - "epsg": "epsg", - "band": "bands" - }, - "out": [ - "cropped" - ], - "scatter": "band", - "scatterMethod": "dotproduct" - }, - "node_normalized_difference": { - "run": "#norm_diff", - "in": { - "rasters": { - "source": "node_crop/cropped" - } - }, - "out": [ - "ndwi" - ] - }, - "node_otsu": { - "run": "#otsu", - "in": { - "raster": { - "source": "node_normalized_difference/ndwi" - } - }, - "out": [ - "binary_mask_item" - ] - } - } - }, - { - "class": "CommandLineTool", - "id": "crop", - "requirements": { - "InlineJavascriptRequirement": {}, - "EnvVarRequirement": { - "envDef": { - "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", - "PYTHONPATH": "/app" - } - }, - "ResourceRequirement": { - "coresMax": 1, - "ramMax": 512 - } - }, - "hints": { - "DockerRequirement": { - "dockerPull": "ghcr.io/eoap/mastering-app-package/crop@sha256:61c2f37b3a1bd56a3eddd0f4224c72de665d42957d3325dd0397845cff198dab" - } - }, - "baseCommand": [ - "python", - "-m", - "app" - ], - "arguments": [], - "inputs": { - "item": { - "type": "string", - "inputBinding": { - "prefix": "--input-item" - } - }, - "aoi": { - "type": "string", - "inputBinding": { - "prefix": "--aoi" - } - }, - "epsg": { - "type": "string", - "inputBinding": { - "prefix": "--epsg" - } - }, - "band": { - "type": "string", - "inputBinding": { - "prefix": "--band" - } - } - }, - "outputs": { - "cropped": { - "outputBinding": { - "glob": "*.tif" - }, - "type": "File" - } - } - }, - { - "class": "CommandLineTool", - "id": "norm_diff", - "requirements": { - "InlineJavascriptRequirement": {}, - "EnvVarRequirement": { - "envDef": { - "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", - "PYTHONPATH": "/app" - } - }, - "ResourceRequirement": { - "coresMax": 1, - "ramMax": 512 - } - }, - "hints": { - "DockerRequirement": { - "dockerPull": "ghcr.io/eoap/mastering-app-package/norm_diff@sha256:641ab55968d7f500b641969d0a7422c4c9a80dd5cc1459ee07f3e0c5ee4fa230" - } - }, - "baseCommand": [ - "python", - "-m", - "app" - ], - "arguments": [], - "inputs": { - "rasters": { - "type": "File[]", - "inputBinding": { - "position": 1 - } - } - }, - "outputs": { - "ndwi": { - "outputBinding": { - "glob": "*.tif" - }, - "type": "File" - } - } - }, - { - "class": "CommandLineTool", - "id": "otsu", - "requirements": { - "InlineJavascriptRequirement": {}, - "EnvVarRequirement": { - "envDef": { - "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", - "PYTHONPATH": "/app" - } - }, - "ResourceRequirement": { - "coresMax": 1, - "ramMax": 512 - } - }, - "hints": { - "DockerRequirement": { - "dockerPull": "ghcr.io/eoap/mastering-app-package/otsu@sha256:ec02baf6a007ebb5a2dc037af2da8ef94db67bb8a6139682ad61041f9d27c779" - } - }, - "baseCommand": [ - "python", - "-m", - "app" - ], - "arguments": [], - "inputs": { - "raster": { - "type": "File", - "inputBinding": { - "position": 1 - } - } - }, - "outputs": { - "binary_mask_item": { - "outputBinding": { - "glob": "*.tif" - }, - "type": "File" - } - } - }, - { - "class": "CommandLineTool", - "id": "stac", - "requirements": { - "InlineJavascriptRequirement": {}, - "EnvVarRequirement": { - "envDef": { - "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", - "PYTHONPATH": "/app" - } - }, - "ResourceRequirement": { - "coresMax": 1, - "ramMax": 512 - } - }, - "hints": { - "DockerRequirement": { - "dockerPull": "ghcr.io/eoap/mastering-app-package/stac@sha256:8ad7e7d6999c0c4c91c2b19ce231f24734d2af8f41b855ef9a06276d23345601" - } - }, - "baseCommand": [ - "python", - "-m", - "app" - ], - "arguments": [], - "inputs": { - "item": { - "type": { - "type": "array", - "items": "string", - "inputBinding": { - "prefix": "--input-item" - } - } - }, - "rasters": { - "type": { - "type": "array", - "items": "File", - "inputBinding": { - "prefix": "--water-body" - } - } - } - }, - "outputs": { - "stac_catalog": { - "outputBinding": { - "glob": "." - }, - "type": "Directory" - } - } - } - ], - "s:codeRepository": { - "URL": "https://github.com/eoap/mastering-app-package.git" - }, - "s:author": [ - { - "class": "s:Person", - "s.name": "Jane Doe", - "s.email": "jane.doe@acme.earth", - "s.affiliation": "ACME" - } - ] -} diff --git a/machine-learning-process/charts/argo-water-bodies/files/input-parameters-schema.json b/machine-learning-process/charts/argo-water-bodies/files/input-parameters-schema.json deleted file mode 100644 index e5154dd..0000000 --- a/machine-learning-process/charts/argo-water-bodies/files/input-parameters-schema.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "$schema": "http://json-schema.org/draft-07/schema#", - "title": "Water Bodies Detection Input", - "description": "Schema for input parameters to the water bodies detection workflow", - "type": "object", - "properties": { - "stac_items": { - "type": "array", - "description": "Array of URLs pointing to STAC items", - "items": { - "type": "string", - "format": "uri", - "description": "A valid URL pointing to a STAC item" - }, - "minItems": 1, - "uniqueItems": true - }, - "aoi": { - "type": "string", - "description": "Bounding box for the area of interest in format 'minLon,minLat,maxLon,maxLat'", - "pattern": "^-?\\d+(\\.\\d+)?,-?\\d+(\\.\\d+)?,-?\\d+(\\.\\d+)?,-?\\d+(\\.\\d+)?$", - "examples": ["-121.399,39.834,-120.74,40.472"] - }, - "epsg": { - "type": "string", - "description": "EPSG code for the coordinate reference system", - "pattern": "^EPSG:\\d+$", - "examples": ["EPSG:4326"] - } - }, - "required": ["stac_items", "aoi", "epsg"], - "additionalProperties": false - } - \ No newline at end of file diff --git a/machine-learning-process/charts/argo-water-bodies/files/prepare.sh b/machine-learning-process/charts/argo-water-bodies/files/prepare.sh deleted file mode 100644 index 135be70..0000000 --- a/machine-learning-process/charts/argo-water-bodies/files/prepare.sh +++ /dev/null @@ -1,15 +0,0 @@ -#!/bin/bash -set -x - -cat >/tmp/cwl_parameters.json < /tmp/cwl_workflow.json -sleep 1 - \ No newline at end of file diff --git a/machine-learning-process/charts/argo-water-bodies/files/validation.py b/machine-learning-process/charts/argo-water-bodies/files/validation.py deleted file mode 100644 index 4eeb376..0000000 --- a/machine-learning-process/charts/argo-water-bodies/files/validation.py +++ /dev/null @@ -1,20 +0,0 @@ -import sys -import json -import ast -import jsonschema -from jsonschema import validate - -inputs_raw = """{{`{{inputs.parameters.inputs}}`}}""" - -inputs = ast.literal_eval(inputs_raw) - -# Load the JSON schema from a file -with open("/schema/input-schema.json", "r") as schema_file: - schema = json.load(schema_file) - -try: - validate(instance=inputs, schema=schema) - print("Input JSON is valid.") -except jsonschema.exceptions.ValidationError as err: - print(f"Input JSON is invalid: {err.message}") - sys.exit(2) \ No newline at end of file diff --git a/machine-learning-process/charts/argo-water-bodies/templates/app-package.yaml b/machine-learning-process/charts/argo-water-bodies/templates/app-package.yaml deleted file mode 100644 index a73add2..0000000 --- a/machine-learning-process/charts/argo-water-bodies/templates/app-package.yaml +++ /dev/null @@ -1,6 +0,0 @@ -kind: ConfigMap -apiVersion: v1 -metadata: - name: app-package-water-bodies-1-0-0 -data: - app-package.json: {{ required "A valid .Values.appPackage entry required!" ( tpl ( .Values.appPackage | default ( .Files.Get "files/app-package.json")) . | quote ) }} \ No newline at end of file diff --git a/machine-learning-process/charts/argo-water-bodies/templates/argo-water-bodies.yaml b/machine-learning-process/charts/argo-water-bodies/templates/argo-water-bodies.yaml deleted file mode 100644 index 8532f90..0000000 --- a/machine-learning-process/charts/argo-water-bodies/templates/argo-water-bodies.yaml +++ /dev/null @@ -1,216 +0,0 @@ -apiVersion: argoproj.io/v1alpha1 -kind: WorkflowTemplate -metadata: - name: water-bodies-detection - annotations: - workflows.argoproj.io/version: ">= v3.3.0" - workflows.argoproj.io/description: | - This workflow template detects water bodies. - eoap.ogc.org/version: "1.0.0" - eoap.ogc.org/title: "Water bodies detection based on NDWI and otsu threshold" - eoap.ogc.org/abstract: "Water bodies detection based on NDWI and otsu threshold applied to Sentinel-2 COG STAC items" - eoap.ogc.org/schema: "app-package-input-schema-water-bodies-1-0-0/input-schema.json" - eoap.ogc.org/app-package: "app-package-water-bodies-1-0-0/app-package.json" -spec: - onExit: exit-handler - # this is the Workflow throttling to avoid overloading the cluster - # the configMap contains the number of concurrent Workflows allowed - synchronization: - semaphore: - configMapKeyRef: - name: semaphore-water-bodies - key: workflow - # Claim the RWX volume - volumeClaimTemplates: - - metadata: - name: calrissian-wdir - spec: - accessModes: [ "ReadWriteMany" ] - storageClassName: "{{ .Values.storageClassRWX }}" - resources: - requests: - storage: "{{ .Values.storageSize }}" - volumes: - - name: usersettings-vol - secret: - secretName: user-settings - - name: volume-input-schema - configMap: - name: app-package-input-schema-water-bodies-1-0-0 - items: - - key: input-schema.json - path: input-schema.json - mode: 420 - defaultMode: 420 - optional: false - - name: volume-app-package - configMap: - name: app-package-water-bodies-1-0-0 - items: - - key: app-package.json - path: app-package.json - mode: 420 - defaultMode: 420 - optional: false - - entrypoint: main - parameters: - - name: items - value: "['https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2A_10TFK_20210708_0_L2A']" - - name: "aoi" - value: "-121.399,39.834,-120.74,40.472" - - name: "epsg" - value: "EPSG:4326" - - templates: - - name: main - inputs: - parameters: - - name: items - - name: aoi - - name: epsg - outputs: - parameters: - - name: results - valueFrom: - expression: "steps['argo-cwl'].outputs.parameters['results']" - - name: log - valueFrom: - expression: "steps['argo-cwl'].outputs.parameters['log']" - - name: usage-report - valueFrom: - expression: "steps['argo-cwl'].outputs.parameters['usage-report']" - - name: stac-catalog - valueFrom: - expression: "steps['argo-cwl'].outputs.parameters['stac-catalog']" - artifacts: - - name: tool-logs - fromExpression: "steps['argo-cwl'].outputs.artifacts['tool-logs']" - - name: calrissian-output - fromExpression: "steps['argo-cwl'].outputs.artifacts['calrissian-output']" - - name: calrissian-stderr - fromExpression: "steps['argo-cwl'].outputs.artifacts['calrissian-stderr']" - - name: calrissian-report - fromExpression: "steps['argo-cwl'].outputs.artifacts['calrissian-report']" - steps: - - - name: prepare - template: prepare - arguments: - parameters: - - name: items - value: {{ `"{{inputs.parameters.items}}"` }} - - name: aoi - value: {{ `"{{inputs.parameters.aoi}}"` }} - - name: epsg - value: {{ `"{{inputs.parameters.epsg}}"` }} - - - - name: validate-inputs - template: validate-inputs - arguments: - parameters: - - name: inputs - value: {{ `"{{steps.prepare.outputs.parameters.inputs}}"` }} - - - - name: argo-cwl # this steps invokes a WorkflowTemplate - templateRef: - name: argo-cwl-runner - template: calrissian-runner - arguments: - parameters: - - name: max_ram - value: "{{ .Values.maxRam }}" - - name: max_cores - value: "{{ .Values.maxCores }}" - - name: parameters - value: {{ `"{{steps.prepare.outputs.parameters.inputs}}"` }} - - name: cwl - value: {{ `"{{steps.prepare.outputs.parameters.workflow}}"` }} - - - name: prepare - inputs: - parameters: - - name: items - - name: aoi - - name: epsg - outputs: - parameters: - - name: inputs - valueFrom: - path: /tmp/cwl_parameters.json - - name: workflow - valueFrom: - path: /tmp/cwl_workflow.json - script: - image: ubuntu:20.04 - resources: - requests: - memory: 1Gi - cpu: 1 - - volumeMounts: - - name: volume-app-package - mountPath: /config - - command: [bash] - source: | - {{ required "A valid .Values.prepareScript entry required!" ( tpl ( .Values.prepareScript | default ( .Files.Get "files/prepare.sh")) . | nindent 10 ) }} - - - name: validate-inputs - inputs: - parameters: - - name: inputs - - script: - image: {{ .Values.validateSchemaImage }} - resources: - requests: - memory: 1Gi - cpu: 1 - - volumeMounts: - - name: volume-input-schema - mountPath: /schema - - command: [python] - source: | - {{ required "A valid .Values.validationScript entry required!" ( tpl ( .Values.validationScript | default ( .Files.Get "files/validation.py")) . | nindent 10 ) }} - - - - name: exit-handler - steps: - - - name: celebrate - template: notify-redis-success - when: "{{"{{workflow.status}}"}} == Succeeded" - - name: cry - template: notify-redis-failure - when: "{{"{{workflow.status}}"}} != Succeeded" - - - - name: notify-redis-success - script: - image: redis:alpine - command: [sh, -c] - source: | - item="{{ `{{workflow.parameters.items}}` }}" - item="${item:1:-1}" - catalog="s3://results/{{ `{{workflow.name}}`}}-{{ `{{workflow.uid}}` }}/catalog.json" - echo "XADD WATERBODIES * subject \"$item\" \ - workflow_id \"{{ `{{workflow.uid}}` }}\" \ - workflow_name \"{{ `{{workflow.name}}`}}\" \ - workflow_status \"{{ `{{workflow.status}}`}}\" \ - href \"$item\" \ - stac_catalog \"$catalog\"" | redis-cli -h redis-service -p 6379 - - - name: notify-redis-failure - script: - image: redis:alpine - command: [sh, -c] - source: | - item="{{ `{{workflow.parameters.items}}` }}" - item="${item:1:-1}" - echo "XADD WATERBODIESFAILURE * subject \"$item\" \ - workflow_id \"{{ `{{workflow.uid}}` }}\" \ - workflow_name \"{{ `{{workflow.name}}`}}\" \ - workflow_status \"{{ `{{workflow.status}}`}}\" \ - href \"$item\"" | redis-cli -h redis-service -p 6379 - diff --git a/machine-learning-process/charts/argo-water-bodies/templates/input-parameters-schema.yaml b/machine-learning-process/charts/argo-water-bodies/templates/input-parameters-schema.yaml deleted file mode 100644 index 5e88dcd..0000000 --- a/machine-learning-process/charts/argo-water-bodies/templates/input-parameters-schema.yaml +++ /dev/null @@ -1,6 +0,0 @@ -kind: ConfigMap -apiVersion: v1 -metadata: - name: app-package-input-schema-water-bodies-1-0-0 -data: - input-schema.json: {{ required "A valid .Values.inputSchema entry required!" ( tpl ( .Values.inputSchema | default ( .Files.Get "files/input-parameters-schema.json")) . | quote ) }} \ No newline at end of file diff --git a/machine-learning-process/charts/argo-water-bodies/templates/semaphore.yaml b/machine-learning-process/charts/argo-water-bodies/templates/semaphore.yaml deleted file mode 100644 index 1cafcd8..0000000 --- a/machine-learning-process/charts/argo-water-bodies/templates/semaphore.yaml +++ /dev/null @@ -1,7 +0,0 @@ -apiVersion: v1 -kind: ConfigMap -metadata: - name: semaphore-water-bodies -data: - workflow: "{{ .Values.semaphore }}" - \ No newline at end of file diff --git a/machine-learning-process/charts/argo-water-bodies/values.yaml b/machine-learning-process/charts/argo-water-bodies/values.yaml deleted file mode 100644 index 5ed76d5..0000000 --- a/machine-learning-process/charts/argo-water-bodies/values.yaml +++ /dev/null @@ -1,8 +0,0 @@ -semaphore: 3 -validateSchemaImage: validateschema - -storageClassRWX: standard -storageSize: "10Gi" - -maxRam: "4G" -maxCores: "4" \ No newline at end of file diff --git a/machine-learning-process/charts/event-driven/Chart.yaml b/machine-learning-process/charts/event-driven/Chart.yaml deleted file mode 100644 index b23913a..0000000 --- a/machine-learning-process/charts/event-driven/Chart.yaml +++ /dev/null @@ -1,6 +0,0 @@ -apiVersion: v2 -name: event-driven -description: Event driven manifests including event bus and source/sensor -type: application -version: 0.1.0 -appVersion: "1.0.0" diff --git a/machine-learning-process/charts/event-driven/templates/event-bus.yaml b/machine-learning-process/charts/event-driven/templates/event-bus.yaml deleted file mode 100644 index ca28d85..0000000 --- a/machine-learning-process/charts/event-driven/templates/event-bus.yaml +++ /dev/null @@ -1,18 +0,0 @@ -apiVersion: argoproj.io/v1alpha1 -kind: EventBus -metadata: - name: jetstream-event-bus -spec: - jetstream: - version: latest - replicas: {{ .Values.jetstreamReplicas }} - persistence: - storageClassName: standard - accessMode: ReadWriteOnce - volumeSize: {{ .Values.jetstreamVolumeSize }} - streamConfig: | - maxAge: 24h - settings: | - max_file_store: 1GB - startArgs: - - "-D" \ No newline at end of file diff --git a/machine-learning-process/charts/event-driven/templates/event-sensor-log.yaml b/machine-learning-process/charts/event-driven/templates/event-sensor-log.yaml deleted file mode 100644 index dd9e8d5..0000000 --- a/machine-learning-process/charts/event-driven/templates/event-sensor-log.yaml +++ /dev/null @@ -1,16 +0,0 @@ -apiVersion: argoproj.io/v1alpha1 -kind: Sensor -metadata: - name: acme-sentinel2-stream-source-logger -spec: - eventBusName: jetstream-event-bus - - dependencies: - - name: acme-sentinel2-stream-source-logger - eventSourceName: acme-sentinel2-stream-source - eventName: acme-sentinel2-stream-source-collected - triggers: - - template: - name: log-event - log: - intervalSeconds: 5 # Frequency of log updates \ No newline at end of file diff --git a/machine-learning-process/charts/event-driven/templates/event-sensor.yaml b/machine-learning-process/charts/event-driven/templates/event-sensor.yaml deleted file mode 100644 index f22053c..0000000 --- a/machine-learning-process/charts/event-driven/templates/event-sensor.yaml +++ /dev/null @@ -1,46 +0,0 @@ -apiVersion: argoproj.io/v1alpha1 -kind: Sensor -metadata: - name: acme-water-bodies-detection-sensor -spec: - eventBusName: jetstream-event-bus - template: - serviceAccountName: argo - dependencies: - - name: acme-sentinel2-stream - eventSourceName: acme-sentinel2-stream-source - eventName: acme-sentinel2-stream-source-collected - transform: - jq: ".values.href = [ .values.href ]" - triggers: - - template: - name: water-bodies-detection-trigger - k8s: - group: argoproj.io - version: v1alpha1 - resource: workflows - operation: create - source: - resource: - apiVersion: argoproj.io/v1alpha1 - kind: Workflow - metadata: - generateName: water-bodies-detection- - namespace: eoap-event-driven - spec: - serviceAccountName: argo - entrypoint: main - arguments: - parameters: - - name: items - - name: aoi - value: "-121.399,39.834,-120.74,40.472" - - name: epsg - value: "EPSG:4326" - workflowTemplateRef: - name: water-bodies-detection - parameters: - - src: - dependencyName: acme-sentinel2-stream - dataKey: values.href - dest: spec.arguments.parameters.0.value diff --git a/machine-learning-process/charts/event-driven/templates/event-source.yaml b/machine-learning-process/charts/event-driven/templates/event-source.yaml deleted file mode 100644 index 4811ea3..0000000 --- a/machine-learning-process/charts/event-driven/templates/event-source.yaml +++ /dev/null @@ -1,13 +0,0 @@ -apiVersion: argoproj.io/v1alpha1 -kind: EventSource -metadata: - name: acme-sentinel2-stream-source -spec: - eventBusName: jetstream-event-bus - redisStream: - acme-sentinel2-stream-source-collected: - hostAddress: redis-service:6379 - db: 0 - streams: - - {{ .Values.streamName }} - consumerGroup: acme-sentinel2-stream-source \ No newline at end of file diff --git a/machine-learning-process/charts/event-driven/templates/local-stack-secret.yaml b/machine-learning-process/charts/event-driven/templates/local-stack-secret.yaml deleted file mode 100644 index b9441fa..0000000 --- a/machine-learning-process/charts/event-driven/templates/local-stack-secret.yaml +++ /dev/null @@ -1,10 +0,0 @@ -apiVersion: v1 -stringData: - accesskey: test - secretkey: test -kind: Secret -metadata: - name: localstack-cred - labels: - app: localstack -type: Opaque \ No newline at end of file diff --git a/machine-learning-process/charts/event-driven/templates/role-binding.yaml b/machine-learning-process/charts/event-driven/templates/role-binding.yaml deleted file mode 100644 index 84d742c..0000000 --- a/machine-learning-process/charts/event-driven/templates/role-binding.yaml +++ /dev/null @@ -1,11 +0,0 @@ -apiVersion: rbac.authorization.k8s.io/v1 -kind: RoleBinding -metadata: - name: argo-extended-role-binding -roleRef: - apiGroup: rbac.authorization.k8s.io - kind: Role - name: argo-extended-role -subjects: -- kind: ServiceAccount - name: argo \ No newline at end of file diff --git a/machine-learning-process/charts/event-driven/templates/roles.yaml b/machine-learning-process/charts/event-driven/templates/roles.yaml deleted file mode 100644 index e55cefe..0000000 --- a/machine-learning-process/charts/event-driven/templates/roles.yaml +++ /dev/null @@ -1,52 +0,0 @@ ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: Role -metadata: - name: argo-extended-role -rules: - - apiGroups: - - "" - resources: - - pods - verbs: - - create - - patch - - delete - - list - - watch - - get - - patch - - apiGroups: - - "" - resources: - - pods/log - verbs: - - get - - list - - apiGroups: - - argoproj.io - resources: - - workflows - verbs: - - create - - get - - list - - watch - - update - - patch - - apiGroups: - - argoproj.io - resources: - - workflowtaskresults - verbs: - - create - - patch - - apiGroups: - - batch - resources: - - jobs - verbs: - - create - - get - - list - - watch \ No newline at end of file diff --git a/machine-learning-process/charts/event-driven/values.yaml b/machine-learning-process/charts/event-driven/values.yaml deleted file mode 100644 index f5789b1..0000000 --- a/machine-learning-process/charts/event-driven/values.yaml +++ /dev/null @@ -1,4 +0,0 @@ -jetstreamReplicas: 2 -jetstreamVolumeSize: 1Gi - -streamName: STREAM \ No newline at end of file diff --git a/machine-learning-process/charts/redis/Chart.yaml b/machine-learning-process/charts/redis/Chart.yaml deleted file mode 100644 index e615c41..0000000 --- a/machine-learning-process/charts/redis/Chart.yaml +++ /dev/null @@ -1,6 +0,0 @@ -apiVersion: v2 -name: redis -description: simple redis chart -type: application -version: 0.1.0 -appVersion: "1.0.0" diff --git a/machine-learning-process/charts/redis/templates/config.yaml b/machine-learning-process/charts/redis/templates/config.yaml deleted file mode 100644 index e67dbf1..0000000 --- a/machine-learning-process/charts/redis/templates/config.yaml +++ /dev/null @@ -1,8 +0,0 @@ -apiVersion: v1 -kind: ConfigMap -metadata: - name: redis-config -data: - redis-config: | - maxmemory {{ .Values.maxMemory }} - maxmemory-policy {{ .Values.maxMemoryPolicy }} diff --git a/machine-learning-process/charts/redis/templates/redis-deployment.yaml b/machine-learning-process/charts/redis/templates/redis-deployment.yaml deleted file mode 100644 index ba296db..0000000 --- a/machine-learning-process/charts/redis/templates/redis-deployment.yaml +++ /dev/null @@ -1,42 +0,0 @@ -apiVersion: apps/v1 -kind: Deployment -metadata: - name: redis -spec: - replicas: 1 - selector: - matchLabels: - app: redis - template: - metadata: - labels: - app: redis - spec: - containers: - - name: redis - image: redis:5.0.4 - command: - - redis-server - - "/redis-master/redis.conf" - env: - - name: MASTER - value: "true" - ports: - - containerPort: {{ .Values.port }} - resources: - limits: - cpu: "0.5" - volumeMounts: - - mountPath: /redis-master-data - name: data - - mountPath: /redis-master - name: config - volumes: - - name: data - emptyDir: {} - - name: config - configMap: - name: redis-config - items: - - key: redis-config - path: redis.conf diff --git a/machine-learning-process/charts/redis/templates/redis-service.yaml b/machine-learning-process/charts/redis/templates/redis-service.yaml deleted file mode 100644 index 5d7dbf9..0000000 --- a/machine-learning-process/charts/redis/templates/redis-service.yaml +++ /dev/null @@ -1,12 +0,0 @@ -apiVersion: v1 -kind: Service -metadata: - name: redis-service -spec: - selector: - app: redis - ports: - - protocol: TCP - port: 6379 - targetPort: 6379 - type: ClusterIP \ No newline at end of file diff --git a/machine-learning-process/charts/redis/values.yaml b/machine-learning-process/charts/redis/values.yaml deleted file mode 100644 index a2c43dd..0000000 --- a/machine-learning-process/charts/redis/values.yaml +++ /dev/null @@ -1,3 +0,0 @@ -maxMemory: 2mb -maxMemoryPolicy: allkeys-lru -port: 6379 \ No newline at end of file diff --git a/machine-learning-process/files/init.sh b/machine-learning-process/files/init.sh index fe17126..088a8eb 100644 --- a/machine-learning-process/files/init.sh +++ b/machine-learning-process/files/init.sh @@ -1,9 +1,9 @@ cd /workspace #temp -mkdir event-driven-with-argo -git clone https://github.com/eoap/event-driven-with-argo.git -cd event-driven-with-argo +mkdir notebook +git clone https://github.com/ai-extensions/notebooks.git +cd notebook git checkout develop code-server --install-extension ms-python.python diff --git a/machine-learning-process/skaffold.yaml b/machine-learning-process/skaffold.yaml index aaf7ca6..229825e 100644 --- a/machine-learning-process/skaffold.yaml +++ b/machine-learning-process/skaffold.yaml @@ -1,17 +1,17 @@ apiVersion: skaffold/v4beta9 kind: Config metadata: - name: eoap-event-driven + name: tile-based-training build: artifacts: - - image: stageout - context: containers/stage-out - docker: - dockerfile: Dockerfile - - image: validateschema - context: containers/validate-schema - docker: - dockerfile: Dockerfile + - image: stageout + context: containers/stage-out + docker: + dockerfile: Dockerfile + - image: validateschema + context: containers/validate-schema + docker: + dockerfile: Dockerfile deploy: helm: @@ -19,7 +19,7 @@ deploy: # argo workflows - name: eoap-argo-workflows remoteChart: argo/argo-workflows - namespace: eoap-event-driven + namespace: tile-based-training createNamespace: true wait: true setValues: @@ -34,7 +34,7 @@ deploy: create: true controller: workflowNamespaces: - - eoap-event-driven + - tile-based-training workflowDefaults: spec: serviceAccountName: argo @@ -53,105 +53,66 @@ deploy: secretKeySecret: name: localstack-cred key: secretkey - # argo events - - name: eoap-argo-events - remoteChart: argo/argo-events - namespace: eoap-event-driven - wait: true - createNamespace: true - setValues: - crds.install: true - controller: - rbac: - enabled: true - namespaced: true - managedNamespace: eoap-event-driven - - - - name: eoap-redis - chartPath: ./charts/redis - namespace: eoap-event-driven - createNamespace: true - setValues: - maxMemory: 2mb - maxMemoryPolicy: allkeys-lru - port: 6379 - - # local chart for event bus/source/sensor - - name: eoap-event-driven - chartPath: ./charts/event-driven - namespace: eoap-event-driven - createNamespace: true - wait: true - setValues: - jetstreamReplicas: 3 - jetstreamVolumeSize: 1Gi - streamName: STREAM - - # argo cwl runner + + # Argo CWL Runner - name: argo-cwl-runner chartPath: ./charts/argo-cwl-runner - namespace: eoap-event-driven + namespace: tile-based-training valuesFiles: - ./charts/argo-cwl-runner/values.yaml setValueTemplates: stageOutImage: "{{.IMAGE_NAME_stageout}}:{{.IMAGE_TAG_stageout}}" - setFiles: { + setFiles: userSettings: ./charts/argo-cwl-runner/files/user-settings.json - } - # argo water bodies - - name: argo-water-bodies - chartPath: ./charts/argo-water-bodies - namespace: eoap-event-driven - valuesFiles: - - ./charts/argo-water-bodies/values.yaml - setFiles: { - appPackage: ./charts/argo-water-bodies/files/app-package.json, - inputSchema: ./charts/argo-water-bodies/files/input-parameters-schema.json - } - setValues: - storageClassRWX: "standard" - - name: eoap-event-driven-localstack + # LocalStack + - name: tile-based-training-localstack remoteChart: localstack/localstack - namespace: eoap-event-driven + namespace: tile-based-training createNamespace: true wait: true setValues: service.type: ClusterIP - - name: eoap-event-driven-coder + # Code-Server with Calrissian + - name: tile-based-training-coder chartPath: ../charts/coder - namespace: eoap-event-driven + namespace: tile-based-training createNamespace: true wait: true setValues: coder.coderImage: eoepca/pde-code-server:1.0.0 - coder.workspace: event-driven-with-argo + coder.workspace: machine-learning-process coderstorageClassName: standard coder.workspaceStorage: 10Gi coderResources.limits.cpu: '2' coderResources.limits.memory: '6442450944' coderResources.requests.cpu: '1' coderResources.requests.memory: '4294967296' - calrissian.enabled: true + calrissian.enabled: true setFiles: { initScript: ./files/init.sh, bashrcScript: ./files/bash-rc, bashloginScript: ./files/bash-login } - + # MLflow + - name: my-mlflow + remoteChart: community-charts/mlflow + namespace: tile-based-training + createNamespace: true + wait: true + setValues: + service.type: ClusterIP -portForward: +portForward: - resourceType: service - resourceName: eoap-argo-workflows-server - namespace: eoap-event-driven - address: localhost - port: 2746 - localPort: 2746 + resourceName: my-mlflow + namespace: tile-based-training + port: 5000 + localPort: 5000 + - resourceType: service resourceName: code-server-service - namespace: eoap-event-driven - address: localhost + namespace: tile-based-training port: 8080 - localPort: 8000 \ No newline at end of file + localPort: 8000 From 69369b399390cd9d38edd3aff9d415ff93468bdc Mon Sep 17 00:00:00 2001 From: pmembari Date: Mon, 3 Mar 2025 18:10:11 +0100 Subject: [PATCH 13/61] docs: add intructions --- machine-learning-process/README.md | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/machine-learning-process/README.md b/machine-learning-process/README.md index e69de29..9ebffd7 100644 --- a/machine-learning-process/README.md +++ b/machine-learning-process/README.md @@ -0,0 +1,24 @@ +# Installation + +1- Create namespace: +``` +kubectl create ns tile-based-training +``` +2- install mlflow using helm: +``` +helm repo add community-charts https://community-charts.github.io/helm-charts +helm repo update +helm install my-mlflow community-charts/mlflow --namespace tile-based-training + +``` + +3- edit [init.sh](./files/init.sh) with a correct repository: +ex: +``` +git clone https://{your_git_account}:{your_access_token}@github.com/ai-extensions/notebooks.git +``` + +4- Deploy everything using command below: +``` +skaffold dev +``` \ No newline at end of file From 4f4c4348d87762442305d3371640f45905d51349 Mon Sep 17 00:00:00 2001 From: pmembari Date: Mon, 10 Mar 2025 17:26:53 +0100 Subject: [PATCH 14/61] feat: edit machine learning code server --- machine-learning-process/files/init.sh | 6 +- machine-learning-process/skaffold.yaml | 136 +++++++++++-------------- 2 files changed, 63 insertions(+), 79 deletions(-) diff --git a/machine-learning-process/files/init.sh b/machine-learning-process/files/init.sh index 088a8eb..fe17126 100644 --- a/machine-learning-process/files/init.sh +++ b/machine-learning-process/files/init.sh @@ -1,9 +1,9 @@ cd /workspace #temp -mkdir notebook -git clone https://github.com/ai-extensions/notebooks.git -cd notebook +mkdir event-driven-with-argo +git clone https://github.com/eoap/event-driven-with-argo.git +cd event-driven-with-argo git checkout develop code-server --install-extension ms-python.python diff --git a/machine-learning-process/skaffold.yaml b/machine-learning-process/skaffold.yaml index 229825e..b79af69 100644 --- a/machine-learning-process/skaffold.yaml +++ b/machine-learning-process/skaffold.yaml @@ -4,75 +4,25 @@ metadata: name: tile-based-training build: artifacts: - - image: stageout - context: containers/stage-out - docker: - dockerfile: Dockerfile - - image: validateschema - context: containers/validate-schema - docker: - dockerfile: Dockerfile - + - image: stageout + context: containers/stage-out + docker: + dockerfile: Dockerfile + - image: validateschema + context: containers/validate-schema + docker: + dockerfile: Dockerfile deploy: helm: releases: - # argo workflows - - name: eoap-argo-workflows - remoteChart: argo/argo-workflows - namespace: tile-based-training - createNamespace: true - wait: true - setValues: - crds.install: true - server.authModes: [server] - singleNamespace: true - workflow: - serviceAccount: - create: true - name: "argo" - rbac: - create: true - controller: - workflowNamespaces: - - tile-based-training - workflowDefaults: - spec: - serviceAccountName: argo - podGC: - strategy: OnWorkflowCompletion - deleteDelayDuration: 120s - rbac.writeConfigMaps: true - securityContext.runAsNonRoot: false - artifactRepository.s3: - bucket: workflows - endpoint: eoap-event-driven-localstack:4566 - insecure: true - accessKeySecret: - name: localstack-cred - key: accesskey - secretKeySecret: - name: localstack-cred - key: secretkey - # Argo CWL Runner - - name: argo-cwl-runner - chartPath: ./charts/argo-cwl-runner - namespace: tile-based-training - valuesFiles: - - ./charts/argo-cwl-runner/values.yaml - setValueTemplates: - stageOutImage: "{{.IMAGE_NAME_stageout}}:{{.IMAGE_TAG_stageout}}" - setFiles: - userSettings: ./charts/argo-cwl-runner/files/user-settings.json - - # LocalStack - - name: tile-based-training-localstack + - name: tile-based-localstack remoteChart: localstack/localstack namespace: tile-based-training createNamespace: true wait: true setValues: - service.type: ClusterIP + service.type: LoadBalancer # Code-Server with Calrissian - name: tile-based-training-coder @@ -80,9 +30,11 @@ deploy: namespace: tile-based-training createNamespace: true wait: true + setValues: + service.type: LoadBalancer coder.coderImage: eoepca/pde-code-server:1.0.0 - coder.workspace: machine-learning-process + coder.workspace: event-driven-with-argo coderstorageClassName: standard coder.workspaceStorage: 10Gi coderResources.limits.cpu: '2' @@ -95,24 +47,56 @@ deploy: bashrcScript: ./files/bash-rc, bashloginScript: ./files/bash-login } - # MLflow - - name: my-mlflow - remoteChart: community-charts/mlflow - namespace: tile-based-training - createNamespace: true - wait: true - setValues: - service.type: ClusterIP + + # # MLflow + # - name: my-mlflow + # remoteChart: community-charts/mlflow + # namespace: tile-based-training + # createNamespace: true + # wait: true + # setValues: + # service.type: ClusterIP + # service.port: 5000 + # ingress.enabled: true + # ingress.className: standard + # resources.limits.memory: 5Gi + # resources.limits.cpu: 500m + # resources.requests.cpu: 200m + # resources.requests.memory: 3Gi + # backendStore: + # databaseMigration: true # Enable database migration to ensure schema is up-to-date + # postgres: + # enabled: false # Disable PostgreSQL + # mysql: + # enabled: false # Disable MySQL + # artifactRoot: + # artifactRoot: + # proxiedArtifactStorage: false # Disable proxied artifact storage access + # azureBlob: + # enabled: false # Disable Azure Blob Storage + # s3: + # enabled: false # Disable AWS S3 + # gcs: + # enabled: false # Disable Google Cloud Storage + + portForward: - - resourceType: service - resourceName: my-mlflow - namespace: tile-based-training - port: 5000 - localPort: 5000 + # - resourceType: service + # resourceName: my-mlflow + # namespace: tile-based-training + # port: 5000 + # localPort: 5000 + # - resourceType: service + # resourceName: eoap-argo-workflows-server + # namespace: tile-based-training + # address: localhost + # port: 2746 + # localPort: 2746 - resourceType: service - resourceName: code-server-service + resourceName: tile-based-training-coder namespace: tile-based-training + address: localhost port: 8080 - localPort: 8000 + localPort: 8000 From ed511dbae15ba120ac200014d35d048be1b892c7 Mon Sep 17 00:00:00 2001 From: pmembari Date: Mon, 10 Mar 2025 17:28:27 +0100 Subject: [PATCH 15/61] chore: delete redundant charts --- .../charts/argo-cwl-runner/.helmignore | 23 -- .../charts/argo-cwl-runner/Chart.yaml | 24 -- .../argo-cwl-runner/files/user-settings.json | 15 - .../templates/argo-calrissian.yaml | 324 ------------------ .../argo-cwl-runner/templates/semaphore.yaml | 7 - .../templates/user-settings.yaml | 9 - .../charts/argo-cwl-runner/values.yaml | 2 - .../containers/stage-out/Dockerfile | 15 - .../containers/stage-out/README.md | 49 --- .../containers/stage-out/app/__init__.py | 0 .../containers/stage-out/app/main.py | 94 ----- .../containers/stage-out/app/stac.py | 77 ----- .../containers/stage-out/app/usersettings.py | 63 ---- .../containers/stage-out/requirements.txt | 5 - .../containers/stage-out/setup.cfg | 3 - .../containers/stage-out/setup.py | 12 - .../containers/validate-schema/Dockerfile | 5 - 17 files changed, 727 deletions(-) delete mode 100644 machine-learning-process/charts/argo-cwl-runner/.helmignore delete mode 100644 machine-learning-process/charts/argo-cwl-runner/Chart.yaml delete mode 100644 machine-learning-process/charts/argo-cwl-runner/files/user-settings.json delete mode 100644 machine-learning-process/charts/argo-cwl-runner/templates/argo-calrissian.yaml delete mode 100644 machine-learning-process/charts/argo-cwl-runner/templates/semaphore.yaml delete mode 100644 machine-learning-process/charts/argo-cwl-runner/templates/user-settings.yaml delete mode 100644 machine-learning-process/charts/argo-cwl-runner/values.yaml delete mode 100644 machine-learning-process/containers/stage-out/Dockerfile delete mode 100644 machine-learning-process/containers/stage-out/README.md delete mode 100644 machine-learning-process/containers/stage-out/app/__init__.py delete mode 100644 machine-learning-process/containers/stage-out/app/main.py delete mode 100644 machine-learning-process/containers/stage-out/app/stac.py delete mode 100644 machine-learning-process/containers/stage-out/app/usersettings.py delete mode 100644 machine-learning-process/containers/stage-out/requirements.txt delete mode 100644 machine-learning-process/containers/stage-out/setup.cfg delete mode 100644 machine-learning-process/containers/stage-out/setup.py delete mode 100644 machine-learning-process/containers/validate-schema/Dockerfile diff --git a/machine-learning-process/charts/argo-cwl-runner/.helmignore b/machine-learning-process/charts/argo-cwl-runner/.helmignore deleted file mode 100644 index 0e8a0eb..0000000 --- a/machine-learning-process/charts/argo-cwl-runner/.helmignore +++ /dev/null @@ -1,23 +0,0 @@ -# Patterns to ignore when building packages. -# This supports shell glob matching, relative path matching, and -# negation (prefixed with !). Only one pattern per line. -.DS_Store -# Common VCS dirs -.git/ -.gitignore -.bzr/ -.bzrignore -.hg/ -.hgignore -.svn/ -# Common backup files -*.swp -*.bak -*.tmp -*.orig -*~ -# Various IDEs -.project -.idea/ -*.tmproj -.vscode/ diff --git a/machine-learning-process/charts/argo-cwl-runner/Chart.yaml b/machine-learning-process/charts/argo-cwl-runner/Chart.yaml deleted file mode 100644 index ac56d50..0000000 --- a/machine-learning-process/charts/argo-cwl-runner/Chart.yaml +++ /dev/null @@ -1,24 +0,0 @@ -apiVersion: v2 -name: argo-cwl-runner -description: A Helm chart for Kubernetes - -# A chart can be either an 'application' or a 'library' chart. -# -# Application charts are a collection of templates that can be packaged into versioned archives -# to be deployed. -# -# Library charts provide useful utilities or functions for the chart developer. They're included as -# a dependency of application charts to inject those utilities and functions into the rendering -# pipeline. Library charts do not define any templates and therefore cannot be deployed. -type: application - -# This is the chart version. This version number should be incremented each time you make changes -# to the chart and its templates, including the app version. -# Versions are expected to follow Semantic Versioning (https://semver.org/) -version: 0.1.0 - -# This is the version number of the application being deployed. This version number should be -# incremented each time you make changes to the application. Versions are not expected to -# follow Semantic Versioning. They should reflect the version the application is using. -# It is recommended to use it with quotes. -appVersion: "1.16.0" diff --git a/machine-learning-process/charts/argo-cwl-runner/files/user-settings.json b/machine-learning-process/charts/argo-cwl-runner/files/user-settings.json deleted file mode 100644 index cf9eede..0000000 --- a/machine-learning-process/charts/argo-cwl-runner/files/user-settings.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "S3": { - "Services": { - "results": { - "UrlPattern": "s3:\/\/results\/.*", - "Region": "us-east-1", - "AuthenticationRegion": "us-east-1", - "AccessKey": "test", - "SecretKey": "test", - "ServiceURL": "http://eoap-event-driven-localstack.eoap-event-driven.svc.cluster.local:4566", - "ForcePathStyle": "true" - } - } - } -} \ No newline at end of file diff --git a/machine-learning-process/charts/argo-cwl-runner/templates/argo-calrissian.yaml b/machine-learning-process/charts/argo-cwl-runner/templates/argo-calrissian.yaml deleted file mode 100644 index 1aa265a..0000000 --- a/machine-learning-process/charts/argo-cwl-runner/templates/argo-calrissian.yaml +++ /dev/null @@ -1,324 +0,0 @@ -apiVersion: argoproj.io/v1alpha1 -kind: WorkflowTemplate -metadata: - name: argo-cwl-runner - annotations: - workflows.argoproj.io/version: ">= v3.3.0" - workflows.argoproj.io/description: | - This workflow template is a CWL runner for Argo Workflows. -spec: - entrypoint: calrissian-runner - parameters: - - name: parameters - description: "Parameters in JSON format" - value: "" - - name: cwl - description: "CWL document in JSON format" - value: "" - - name: max_ram - description: "Max RAM" - value: "8G" - - name: max_cores - description: "Max cores" - value: "8" - # this is the Workflow throttling to avoid overloading the cluster - # the configMap contains the number of concurrent Workflows allowed - synchronization: - semaphore: - configMapKeyRef: - name: semaphore-argo-cwl-runner - key: workflow - - # volumes - volumes: - - name: usersettings-vol - secret: - secretName: user-settings - - name: calrissian-wdir - persistentVolumeClaim: - claimName: calrissian-wdir - - # Workflow templates are used to define the Workflow steps - templates: - - name: calrissian-runner - # this is the entry point of the Workflow - inputs: - parameters: - - name: parameters - - name: cwl - - name: max_ram - - name: max_cores - outputs: - parameters: - - name: results - valueFrom: - parameter: {{ `"{{steps.get-results.outputs.parameters.calrissian-output}}"` }} - - name: log - valueFrom: - parameter: {{ `"{{steps.get-results.outputs.parameters.calrissian-stderr}}"` }} - - name: usage-report - valueFrom: - parameter: {{ `"{{steps.get-results.outputs.parameters.calrissian-report}}"` }} - - name: stac-catalog - valueFrom: - parameter: {{ `"{{steps.stage-out.outputs.parameters.stac-catalog}}"` }} - artifacts: - - name: tool-logs - from: {{ `"{{steps.get-results.outputs.artifacts.tool-logs}}"` }} - - name: calrissian-output - from: {{ `"{{steps.get-results.outputs.artifacts.calrissian-output}}"` }} - - name: calrissian-stderr - from: {{ `"{{steps.get-results.outputs.artifacts.calrissian-stderr}}"` }} - - name: calrissian-report - from: {{ `"{{steps.get-results.outputs.artifacts.calrissian-report}}"` }} - - steps: - # Workflow steps are defined here - - - name: cwl-prepare - template: cwl-prepare - arguments: - parameters: - - name: cwl - value: {{ `"{{inputs.parameters.cwl}}"` }} - - name: parameters - value: {{ `"{{inputs.parameters.parameters}}"` }} - - - - name: cwl-runner - template: calrissian-tmpl - arguments: - parameters: - - name: max_ram - value: {{ `"{{inputs.parameters.max_ram}}"` }} - - name: max_cores - value: {{ `"{{inputs.parameters.max_cores}}"` }} - - - - name: get-results - # these can run in parallel - # the same template is used for all the steps - template: get-results - arguments: - parameters: - - name: calrissian-output - value: "/calrissian/output.json" - - name: calrissian-stderr - value: "/calrissian/stderr.log" - - name: calrissian-report - value: "/calrissian/report.json" - - - name: stage-out - template: stage-out - arguments: - parameters: - - name: file_path - value: "/calrissian/output.json" - - name: bucket - value: "results" - - name: folder - value: {{ `"{{workflow.name}}-{{workflow.uid}}"` }} - - - name: cwl-prepare - # this steps prepares the CWL inputs - # needed by Calrissian - inputs: - parameters: - - name: cwl - - name: parameters - - script: - image: busybox:1.35.0 - resources: - requests: - memory: 1Gi - cpu: 1 - volumeMounts: - - name: calrissian-wdir - mountPath: /calrissian - env: [] - command: [ash] - source: | - #!/bin/ash - echo {{ `'{{inputs.parameters.cwl}}'` }} >> /calrissian/cwl.json - echo {{ `'{{inputs.parameters.parameters}}'` }} >> /calrissian/input.json - sleep 1 - - - name: calrissian-tmpl - # this step creates the Calrissian Job, Argo creates it as a Kubernetes Job - resource: - action: create - setOwnerReference: true - successCondition: status.succeeded > 0 - failureCondition: status.failed > 3 - manifest: | - apiVersion: batch/v1 - kind: Job - metadata: - generateName: calrissian-water-bodies-detection- - spec: - backoffLimit: 1 - activeDeadlineSeconds: 86400 - ttlSecondsAfterFinished: 120 - template: - metadata: - name: calrissian_pod - spec: - serviceAccountName: argo - containers: - - name: calrissian - image: ghcr.io/duke-gcb/calrissian/calrissian:0.16.0 - imagePullPolicy: IfNotPresent - command: - - calrissian - args: - - --debug - - --pod-serviceaccount - - argo - - --stdout - - /calrissian/output.json - - --stderr - - /calrissian/stderr.log - - --usage-report - - /calrissian/report.json - - --max-ram - - {{ `'{{inputs.parameters.max_ram}}'` }} - - --max-cores - - {{ `'{{inputs.parameters.max_cores}}'` }} - - --tmp-outdir-prefix - - /calrissian/tmp/ - - --outdir - - /calrissian/results/ - - --tool-logs-basepath - - /calrissian/logs - - "/calrissian/cwl.json" - - "/calrissian/input.json" - env: - - name: CALRISSIAN_POD_NAME - valueFrom: - fieldRef: - fieldPath: metadata.name - - name: CALRISSIAN_DELETE_PODS - value: "true" - resources: - limits: - cpu: 2000m - memory: 2G - requests: - cpu: 1000m - memory: 1G - volumeMounts: - - mountPath: /calrissian - name: calrissian-wdir - readOnly: false - restartPolicy: Never - securityContext: - fsGroup: 0 - runAsGroup: 0 - runAsUser: 0 - terminationGracePeriodSeconds: 120 - volumes: - - name: calrissian-wdir - persistentVolumeClaim: - claimName: {{"{{workflow.name}}"}}-calrissian-wdir - readOnly: false - - inputs: - parameters: - - name: max_ram - - name: max_cores - outputs: - parameters: [] - artifacts: - - name: logs - path: /calrissian/logs - s3: - key: {{ `"{{workflow.name}}-{{workflow.uid}}-artifacts/calrissian-logs.tgz"` }} - - - - - name: get-results - # reads the files generated by Calrissian - inputs: - parameters: - - name: calrissian-output - - name: calrissian-stderr - - name: calrissian-report - outputs: - parameters: - - name: calrissian-output - valueFrom: - path: /tmp/calrissian-output.json - - name: calrissian-stderr - valueFrom: - path: /tmp/calrissian-stderr.txt - - name: calrissian-report - valueFrom: - path: /tmp/calrissian-report.json - artifacts: - - name: tool-logs - path: /calrissian/logs - s3: - key: {{ `"{{workflow.name}}-{{workflow.uid}}-artifacts/tool-logs.tgz"` }} - - name: calrissian-output - path: /tmp/calrissian-output.json - s3: - key: {{ `"{{workflow.name}}-{{workflow.uid}}-artifacts/calrissian-output.tgz"` }} - - name: calrissian-stderr - path: /tmp/calrissian-stderr.txt - s3: - key: {{ `"{{workflow.name}}-{{workflow.uid}}-artifacts/calrissian-stderr.tgz"` }} - - name: calrissian-report - path: /tmp/calrissian-report.json - s3: - key: {{ `"{{workflow.name}}-{{workflow.uid}}-artifacts/calrissian-report.tgz"` }} - script: - image: busybox:1.35.0 - resources: - requests: - memory: 1Gi - cpu: 1 - volumeMounts: - - name: calrissian-wdir - mountPath: /calrissian - command: [ash] - source: | - #!/bin/ash - cat "{{"{{inputs.parameters.calrissian-output}}"}}" > /tmp/calrissian-output.json - cat "{{"{{inputs.parameters.calrissian-stderr}}"}}" > /tmp/calrissian-stderr.txt - cat "{{"{{inputs.parameters.calrissian-report}}"}}" > /tmp/calrissian-report.json - - - name: stage-out - inputs: - parameters: - - name: file_path - - name: bucket - - name: folder - outputs: - parameters: - - name: stac-catalog - valueFrom: - path: /tmp/output - script: - image: {{ .Values.stageOutImage }} - resources: - requests: - memory: 1Gi - cpu: 1 - volumeMounts: - - name: calrissian-wdir - mountPath: /calrissian - - name: usersettings-vol - readOnly: true - mountPath: "/etc/secret" - command: [bash] - source: - #!/bin/bash - set -x - - cat /etc/secret/usersettings.json - - stage-out --stac-catalog $( cat {{"{{inputs.parameters.file_path}}"}} | jq -r .stac_catalog.path - ) --user-settings /etc/secret/usersettings.json --bucket "{{"{{inputs.parameters.bucket}}"}}" --subfolder "{{"{{inputs.parameters.folder}}"}}" - - res=$? - - echo "s3://{{"{{inputs.parameters.bucket}}"}}/{{"{{inputs.parameters.folder}}"}}/catalog.json" > /tmp/output - - exit $res \ No newline at end of file diff --git a/machine-learning-process/charts/argo-cwl-runner/templates/semaphore.yaml b/machine-learning-process/charts/argo-cwl-runner/templates/semaphore.yaml deleted file mode 100644 index 7751209..0000000 --- a/machine-learning-process/charts/argo-cwl-runner/templates/semaphore.yaml +++ /dev/null @@ -1,7 +0,0 @@ -apiVersion: v1 -kind: ConfigMap -metadata: - name: semaphore-argo-cwl-runner -data: - workflow: "{{ .Values.semaphore }}" - \ No newline at end of file diff --git a/machine-learning-process/charts/argo-cwl-runner/templates/user-settings.yaml b/machine-learning-process/charts/argo-cwl-runner/templates/user-settings.yaml deleted file mode 100644 index 6bdf160..0000000 --- a/machine-learning-process/charts/argo-cwl-runner/templates/user-settings.yaml +++ /dev/null @@ -1,9 +0,0 @@ -# secret with the user-settings.json, it transforms the value yaml to json -apiVersion: v1 -kind: Secret -metadata: - name: user-settings - labels: - app: argo-water-bodies -data: - usersettings.json: {{ required "A valid .Values.userSettings entry required!" ( tpl ( .Values.userSettings | default ( .Files.Get "files/user-settings.json")) . | b64enc) }} \ No newline at end of file diff --git a/machine-learning-process/charts/argo-cwl-runner/values.yaml b/machine-learning-process/charts/argo-cwl-runner/values.yaml deleted file mode 100644 index 23d307c..0000000 --- a/machine-learning-process/charts/argo-cwl-runner/values.yaml +++ /dev/null @@ -1,2 +0,0 @@ -stageOutImage: stageout -semaphore: 3 \ No newline at end of file diff --git a/machine-learning-process/containers/stage-out/Dockerfile b/machine-learning-process/containers/stage-out/Dockerfile deleted file mode 100644 index 14d3e6e..0000000 --- a/machine-learning-process/containers/stage-out/Dockerfile +++ /dev/null @@ -1,15 +0,0 @@ -FROM python:3.11 - -RUN apt update && apt install -y yq - -WORKDIR /code - -COPY ./requirements.txt /code/requirements.txt - -RUN pip install --no-cache-dir -r /code/requirements.txt - -COPY . /code - -RUN cd /code && python setup.py install - -RUN stage-out --help diff --git a/machine-learning-process/containers/stage-out/README.md b/machine-learning-process/containers/stage-out/README.md deleted file mode 100644 index a7a4564..0000000 --- a/machine-learning-process/containers/stage-out/README.md +++ /dev/null @@ -1,49 +0,0 @@ -# stage-out - - -## For developers - -Create a Python environment with, e.g. `mamba`: - -``` -mamba create -n env_stage_in python -mamba activate env_stage_in -``` - -``` -pip install -r requirements.txt -``` - -## Container - -Build the container with: - -``` -docker build -t stage-out . -``` - -Test the container with: - -``` -docker run --rm docker.io/library/stage-out stage-out --help -``` - -## CWL - -You can use a cwlrunner like `cwltool` to do a stage-in operation. - -Requirement: - -* a built container tagged `docker.io/library/stage-out:latest` - -``` -cwltool stage-out.cwl --bucket iride-sentinel-2 --stac_catalog /data/work/iride-marketplace/stage-in/_ka1p9cp --subfolder pippo --usersettings usersettings.json -``` - -## Run tests - -The unit tests can be run with: - -`nose2` - -TODO: add capture stdout in CWL \ No newline at end of file diff --git a/machine-learning-process/containers/stage-out/app/__init__.py b/machine-learning-process/containers/stage-out/app/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/machine-learning-process/containers/stage-out/app/main.py b/machine-learning-process/containers/stage-out/app/main.py deleted file mode 100644 index 40470e7..0000000 --- a/machine-learning-process/containers/stage-out/app/main.py +++ /dev/null @@ -1,94 +0,0 @@ -""" Main module for the application. """ -import os -import shutil - -import boto3 -import botocore -import click -import pystac -from botocore.client import Config -from loguru import logger -from pystac.stac_io import StacIO -import sys -from app.stac import CustomStacIO, upload_file_with_chunk_size -from app.usersettings import UserSettings - - -@click.command() -@click.option( - "--stac-catalog", help="Local path to a folder containing catalog.json STAC Catalog", required=True -) -@click.option("--user-settings", help="S3 user settings", required=True) -@click.option("--bucket", "bucket", help="S3 bucket", required=True) -@click.option("--subfolder", "subfolder", help="S3 subfolder", required=True) -def main(stac_catalog, user_settings, bucket, subfolder): - user_settings_config = UserSettings.from_file(user_settings) - - s3_settings = user_settings_config.get_s3_settings(f"s3://{bucket}/{subfolder}") - - if not s3_settings: - raise ValueError("No S3 settings found for this bucket") - - # set the environment variables for S3 from the user settings - os.environ["aws_access_key_id"] = s3_settings["aws_access_key_id"] - os.environ["aws_secret_access_key"] = s3_settings["aws_secret_access_key"] - os.environ["aws_region_name"] = s3_settings["region_name"] - os.environ["aws_endpoint_url"] = s3_settings["endpoint_url"] - - client = boto3.client( - "s3", - **s3_settings, - config=Config(s3={"addressing_style": "path", "signature_version": "s3v4"}), - ) - - shutil.copytree(stac_catalog, "/tmp/catalog") - cat = pystac.read_file(os.path.join("/tmp/catalog", "catalog.json")) - - StacIO.set_default(CustomStacIO) - - for item in cat.get_items(): - for key, asset in item.get_assets().items(): - s3_path = os.path.normpath( - os.path.join(os.path.join(subfolder, item.id, asset.href)) - ) - logger.info(f"upload {asset.href} to s3://{bucket}/{s3_path}") - - upload_file_with_chunk_size( - client, - asset.get_absolute_href(), - bucket, - s3_path - ) - - asset.href = f"s3://{bucket}/{s3_path}" - item.add_asset(key, asset) - - cat.normalize_hrefs(f"s3://{bucket}/{subfolder}") - - for item in cat.get_items(): - # upload item to S3 - logger.info(f"upload {item.id} to s3://{bucket}/{subfolder}") - for index, link in enumerate(item.links): - if link.rel in ["collection"]: - logger.info("saving collection.json") - collection = link.target - collection.links = [] - pystac.write_file(link.target, dest_href="./collection.json") - item.links.pop(index) - item.set_collection(None) - if link.rel in ["root"]: - item.links.pop(index) - pystac.write_file(item, item.get_self_href()) - - # upload catalog to S3 - logger.info(f"upload catalog.json to s3://{bucket}/{subfolder}") - for index, link in enumerate(cat.links): - if link.rel in ["root", "collection"]: - cat.links.pop(index) - pystac.write_file(cat, cat.get_self_href()) - - logger.info("Done!") - - print(f"s3://{bucket}/{subfolder}/catalog.json", file=sys.stdout) -if __name__ == "__main__": - main() diff --git a/machine-learning-process/containers/stage-out/app/stac.py b/machine-learning-process/containers/stage-out/app/stac.py deleted file mode 100644 index 66709fc..0000000 --- a/machine-learning-process/containers/stage-out/app/stac.py +++ /dev/null @@ -1,77 +0,0 @@ -import os -from urllib.parse import urlparse - -import boto3 -import botocore -from botocore.client import Config -from pystac.stac_io import DefaultStacIO - - -class CustomStacIO(DefaultStacIO): - """Custom STAC IO class that uses boto3 to read from S3.""" - - def __init__(self): - self.session = botocore.session.Session() - - def write_text(self, dest, txt, *args, **kwargs): - self.s3_client = self.session.create_client( - service_name="s3", - use_ssl=True, - aws_access_key_id=os.environ["aws_access_key_id"], - aws_secret_access_key=os.environ["aws_secret_access_key"], - region_name=os.environ["aws_region_name"], - endpoint_url=os.environ["aws_endpoint_url"], - config=Config(s3={"addressing_style": "path", "signature_version": "s3v4"}), - ) - - parsed = urlparse(dest) - if parsed.scheme == "s3": - self.s3_client.put_object( - Body=txt.encode("UTF-8"), - Bucket=parsed.netloc, - Key=parsed.path[1:], - ContentType="application/geo+json", - ) - else: - super().write_text(dest, txt, *args, **kwargs) - -import boto3 -from boto3.s3.transfer import TransferConfig -import os - -import boto3 -import os -import shutil -import logging -from boto3.s3.transfer import TransferConfig -from botocore.client import Config -from pystac import StacIO -import pystac - -# Assuming CustomStacIO is already defined somewhere -# from your_custom_module import CustomStacIO - -logger = logging.getLogger() -logger.setLevel(logging.INFO) - -def upload_file_with_chunk_size(client, file_path, bucket_name, s3_key, max_chunks=1000, default_chunk_size_mb=25): - # Get the size of the file - file_size = os.path.getsize(file_path) - - # Convert default chunk size from MB to bytes - default_chunk_size = default_chunk_size_mb * 1024 * 1024 - - # Calculate the optimal chunk size to ensure the number of chunks does not exceed max_chunks - optimal_chunk_size = min(default_chunk_size, file_size // max_chunks + 1) - - # Configure the transfer settings - config = TransferConfig(multipart_chunksize=optimal_chunk_size) - - # Upload the file - client.upload_file( - Filename=file_path, - Bucket=bucket_name, - Key=s3_key, - Config=config - ) - diff --git a/machine-learning-process/containers/stage-out/app/usersettings.py b/machine-learning-process/containers/stage-out/app/usersettings.py deleted file mode 100644 index 9f4e8ae..0000000 --- a/machine-learning-process/containers/stage-out/app/usersettings.py +++ /dev/null @@ -1,63 +0,0 @@ -""" -This code defines a UserSettings class that has methods for reading JSON files, -matching regular expressions, and setting environment variables for an S3 service. -The set_s3_environment method sets environment variables for an S3 service based -on a given URL. -""" - -import json -import os -import re -from typing import Dict - - -class UserSettings: - """class for reading JSON files, matching regular expressions, - and setting environment variables for an S3 service""" - - def __init__(self, settings: Dict): - self.settings = settings - - @classmethod - def from_file(cls, file_path): - """create a UserSettings object from a JSON file""" - return cls(cls.read_json_file(file_path)) - - @staticmethod - def read_json_file(file_path): - """read a JSON file and return the contents as a dictionary""" - with open(file_path, "r", encoding="utf-8") as stream: - return json.load(stream) - - @staticmethod - def match_regex(regex, string): - """match a regular expression to a string and return the match object""" - return re.search(regex, string) - - @staticmethod - def set_s3_vars(s3_service): - """set environment variables for an S3 service""" - os.environ["AWS_ACCESS_KEY_ID"] = s3_service["AccessKey"] - os.environ["AWS_SECRET_ACCESS_KEY"] = s3_service["SecretKey"] - os.environ["AWS_DEFAULT_REGION"] = s3_service["Region"] - os.environ["AWS_REGION"] = s3_service["Region"] - os.environ["AWS_S3_ENDPOINT"] = s3_service["ServiceURL"] - - def set_s3_environment(self, url): - """set environment variables for an S3 service based on a given URL""" - for _, s3_service in self.settings["S3"]["Services"].items(): - if self.match_regex(s3_service["UrlPattern"], url): - self.set_s3_vars(s3_service) - break - - def get_s3_settings(self, url): - """return S3 settings based on a given URL""" - for _, s3_service in self.settings["S3"]["Services"].items(): - if self.match_regex(s3_service["UrlPattern"], url): - return { - "region_name": s3_service["Region"], - "endpoint_url": s3_service["ServiceURL"], - "aws_access_key_id": s3_service["AccessKey"], - "aws_secret_access_key": s3_service["SecretKey"], - } - return None diff --git a/machine-learning-process/containers/stage-out/requirements.txt b/machine-learning-process/containers/stage-out/requirements.txt deleted file mode 100644 index 281aba6..0000000 --- a/machine-learning-process/containers/stage-out/requirements.txt +++ /dev/null @@ -1,5 +0,0 @@ -boto3 -pystac -loguru -click -pyyaml \ No newline at end of file diff --git a/machine-learning-process/containers/stage-out/setup.cfg b/machine-learning-process/containers/stage-out/setup.cfg deleted file mode 100644 index 03162a4..0000000 --- a/machine-learning-process/containers/stage-out/setup.cfg +++ /dev/null @@ -1,3 +0,0 @@ -[metadata] -name=stage-out -version=0.3.4 diff --git a/machine-learning-process/containers/stage-out/setup.py b/machine-learning-process/containers/stage-out/setup.py deleted file mode 100644 index 85fdb58..0000000 --- a/machine-learning-process/containers/stage-out/setup.py +++ /dev/null @@ -1,12 +0,0 @@ -from setuptools import setup - -console_scripts = ["stage-out=app.main:main"] - -setup( - name="stage-out", - version="0.1", - description="The funniest joke in the world", - packages=["app"], - entry_points={"console_scripts": console_scripts}, - zip_safe=False, -) diff --git a/machine-learning-process/containers/validate-schema/Dockerfile b/machine-learning-process/containers/validate-schema/Dockerfile deleted file mode 100644 index 5c9c7a4..0000000 --- a/machine-learning-process/containers/validate-schema/Dockerfile +++ /dev/null @@ -1,5 +0,0 @@ -FROM python:3.11-slim - -WORKDIR /app - -RUN pip install jsonschema \ No newline at end of file From a04fe3c313aab0178e16845a16093c03dde656a3 Mon Sep 17 00:00:00 2001 From: pmembari Date: Mon, 10 Mar 2025 17:33:45 +0100 Subject: [PATCH 16/61] feat: edit mlflow artifactRoot --- machine-learning-process/skaffold.yaml | 145 ++++++++++++------------- 1 file changed, 67 insertions(+), 78 deletions(-) diff --git a/machine-learning-process/skaffold.yaml b/machine-learning-process/skaffold.yaml index b79af69..5788b97 100644 --- a/machine-learning-process/skaffold.yaml +++ b/machine-learning-process/skaffold.yaml @@ -2,91 +2,80 @@ apiVersion: skaffold/v4beta9 kind: Config metadata: name: tile-based-training -build: - artifacts: - - image: stageout - context: containers/stage-out - docker: - dockerfile: Dockerfile - - image: validateschema - context: containers/validate-schema - docker: - dockerfile: Dockerfile deploy: helm: releases: - - name: tile-based-localstack - remoteChart: localstack/localstack - namespace: tile-based-training - createNamespace: true - wait: true - setValues: - service.type: LoadBalancer - - # Code-Server with Calrissian - - name: tile-based-training-coder - chartPath: ../charts/coder - namespace: tile-based-training - createNamespace: true - wait: true + # - name: tile-based-localstack + # remoteChart: localstack/localstack + # namespace: tile-based-training + # createNamespace: true + # wait: true + # setValues: + # service.type: LoadBalancer - setValues: - service.type: LoadBalancer - coder.coderImage: eoepca/pde-code-server:1.0.0 - coder.workspace: event-driven-with-argo - coderstorageClassName: standard - coder.workspaceStorage: 10Gi - coderResources.limits.cpu: '2' - coderResources.limits.memory: '6442450944' - coderResources.requests.cpu: '1' - coderResources.requests.memory: '4294967296' - calrissian.enabled: true - setFiles: { - initScript: ./files/init.sh, - bashrcScript: ./files/bash-rc, - bashloginScript: ./files/bash-login - } - - # # MLflow - # - name: my-mlflow - # remoteChart: community-charts/mlflow + # # Code-Server with Calrissian + # - name: tile-based-training-coder + # chartPath: ../charts/coder # namespace: tile-based-training # createNamespace: true # wait: true + # setValues: - # service.type: ClusterIP - # service.port: 5000 - # ingress.enabled: true - # ingress.className: standard - # resources.limits.memory: 5Gi - # resources.limits.cpu: 500m - # resources.requests.cpu: 200m - # resources.requests.memory: 3Gi - # backendStore: - # databaseMigration: true # Enable database migration to ensure schema is up-to-date - # postgres: - # enabled: false # Disable PostgreSQL - # mysql: - # enabled: false # Disable MySQL - # artifactRoot: - # artifactRoot: - # proxiedArtifactStorage: false # Disable proxied artifact storage access - # azureBlob: - # enabled: false # Disable Azure Blob Storage - # s3: - # enabled: false # Disable AWS S3 - # gcs: - # enabled: false # Disable Google Cloud Storage + # service.type: LoadBalancer + # coder.coderImage: eoepca/pde-code-server:1.0.0 + # coder.workspace: event-driven-with-argo + # coderstorageClassName: standard + # coder.workspaceStorage: 10Gi + # coderResources.limits.cpu: '2' + # coderResources.limits.memory: '6442450944' + # coderResources.requests.cpu: '1' + # coderResources.requests.memory: '4294967296' + # calrissian.enabled: true + # setFiles: { + # initScript: ./files/init.sh, + # bashrcScript: ./files/bash-rc, + # bashloginScript: ./files/bash-login + # } + + # MLflow + - name: my-mlflow + remoteChart: community-charts/mlflow + namespace: tile-based-training + createNamespace: true + wait: true + setValues: + service.type: ClusterIP + service.port: 5000 + ingress.enabled: true + ingress.className: standard + resources.limits.memory: 5Gi + resources.limits.cpu: 500m + resources.requests.cpu: 200m + resources.requests.memory: 3Gi + backendStore: + databaseMigration: true # Enable database migration to ensure schema is up-to-date + postgres: + enabled: false # Disable PostgreSQL + mysql: + enabled: false # Disable MySQL + artifactRoot: + proxiedArtifactStorage: false # Disable proxied artifact storage access + azureBlob: + enabled: false # Disable Azure Blob Storage + s3: + enabled: false # Disable AWS S3 + gcs: + enabled: false # Disable Google Cloud Storage portForward: - # - resourceType: service - # resourceName: my-mlflow - # namespace: tile-based-training - # port: 5000 - # localPort: 5000 + - resourceType: service + resourceName: my-mlflow + namespace: tile-based-training + port: 5000 + localPort: 5000 # - resourceType: service # resourceName: eoap-argo-workflows-server @@ -94,9 +83,9 @@ portForward: # address: localhost # port: 2746 # localPort: 2746 - - resourceType: service - resourceName: tile-based-training-coder - namespace: tile-based-training - address: localhost - port: 8080 - localPort: 8000 + # - resourceType: service + # resourceName: tile-based-training-coder + # namespace: tile-based-training + # address: localhost + # port: 8080 + # localPort: 8000 From ca88aec1f0b0cf9eac6f2984f4608768d99eec4c Mon Sep 17 00:00:00 2001 From: pmembari Date: Mon, 10 Mar 2025 18:03:12 +0100 Subject: [PATCH 17/61] feat: deployed code server without notebooks --- charts/coder/templates/coder.yaml | 19 +++++++- charts/coder/values.yaml | 17 +++++-- machine-learning-process/files/init.sh | 11 +++-- machine-learning-process/skaffold.yaml | 64 ++++++++++++-------------- 4 files changed, 67 insertions(+), 44 deletions(-) diff --git a/charts/coder/templates/coder.yaml b/charts/coder/templates/coder.yaml index 5902cd1..fa7524f 100644 --- a/charts/coder/templates/coder.yaml +++ b/charts/coder/templates/coder.yaml @@ -42,7 +42,7 @@ spec: - name: code-server image: {{ .Values.coder.coderImage }} securityContext: - privileged: true + privileged: {{ .Values.coder.securityContext.privileged }} command: ["/bin/bash", "-c"] args: - > @@ -66,6 +66,10 @@ spec: mountPath: /etc/calrissian/pod-node-selector.yaml subPath: pod-node-selector.yaml {{ end }} + {{- if .Values.daskGateway.enabled }} + - name: dask-gateway-config + mountPath: /etc/dask + {{- end }} env: - name: XDG_CONFIG_HOME value: /workspace/.local @@ -86,7 +90,13 @@ spec: valueFrom: fieldRef: fieldPath: metadata.name - {{ end }} + {{- end }} + {{- if .Values.daskGateway.enabled }} + - name: DASK_GATEWAY_ADDRESS + value: "{{ .Values.daskGateway.daskGatewayUrl }}" + - name: DASK_IMAGE + value: "{{ .Values.daskGateway.image }}" + {{- end }} resources: limits: cpu: {{ .Values.coderResources.limits.cpu }} @@ -119,6 +129,11 @@ spec: name: node-selectors defaultMode: 420 {{ end }} + {{- if .Values.daskGateway.enabled }} + - name: dask-gateway-config + configMap: + name: dask-gateway-config + {{- end }} --- apiVersion: v1 kind: PersistentVolumeClaim diff --git a/charts/coder/values.yaml b/charts/coder/values.yaml index 12cf352..2683ce4 100644 --- a/charts/coder/values.yaml +++ b/charts/coder/values.yaml @@ -1,14 +1,17 @@ - coder: coderImage: eoepca/pde-code-server:1.0.0 workspace: quickwin storageClassName: standard workspaceStorage: 20Gi + securityContext: + privileged: true calrissian: - enabled: False + enabled: True storage: 10Gi storageClassName: standard + imagePullSecrets: + - kaniko-secret coderResources: limits: @@ -16,4 +19,12 @@ coderResources: memory: '6442450944' requests: cpu: '1' - memory: '4294967296' \ No newline at end of file + memory: '4294967296' + +skaffold: + enabled: False + +daskGateway: + enabled: False + daskGatewayUrl: "" + image: "" \ No newline at end of file diff --git a/machine-learning-process/files/init.sh b/machine-learning-process/files/init.sh index fe17126..1cf690a 100644 --- a/machine-learning-process/files/init.sh +++ b/machine-learning-process/files/init.sh @@ -1,9 +1,9 @@ cd /workspace #temp -mkdir event-driven-with-argo -git clone https://github.com/eoap/event-driven-with-argo.git -cd event-driven-with-argo +mkdir notebooks +git clone https://github.com/ai-extensions/notebooks.git +cd notebooks git checkout develop code-server --install-extension ms-python.python @@ -15,8 +15,9 @@ echo '{"workbench.colorTheme": "Visual Studio Dark"}' > /workspace/User/settings python -m venv /workspace/.venv source /workspace/.venv/bin/activate -/workspace/.venv/bin/python -m pip install --no-cache-dir ipykernel requests pyyaml boto3==1.35.23 loguru redis pystac_client -/workspace/.venv/bin/python -m ipykernel install --user --name zoo_env --display-name "Python (Event Driven with Argo)" +/workspace/.venv/bin/python -m pip install --no-cache-dir hatch +hatch shell prod +hatch -m ipykernel install --user --name env_5 --display-name "Python (env_5)" export AWS_DEFAULT_REGION="us-east-1" export AWS_ACCESS_KEY_ID="test" diff --git a/machine-learning-process/skaffold.yaml b/machine-learning-process/skaffold.yaml index 5788b97..e9a9446 100644 --- a/machine-learning-process/skaffold.yaml +++ b/machine-learning-process/skaffold.yaml @@ -14,29 +14,29 @@ deploy: # setValues: # service.type: LoadBalancer - # # Code-Server with Calrissian - # - name: tile-based-training-coder - # chartPath: ../charts/coder - # namespace: tile-based-training - # createNamespace: true - # wait: true + # Code-Server with Calrissian + - name: tile-based-training-coder + chartPath: ../charts/coder + namespace: tile-based-training + createNamespace: true + wait: true - # setValues: - # service.type: LoadBalancer - # coder.coderImage: eoepca/pde-code-server:1.0.0 - # coder.workspace: event-driven-with-argo - # coderstorageClassName: standard - # coder.workspaceStorage: 10Gi - # coderResources.limits.cpu: '2' - # coderResources.limits.memory: '6442450944' - # coderResources.requests.cpu: '1' - # coderResources.requests.memory: '4294967296' - # calrissian.enabled: true - # setFiles: { - # initScript: ./files/init.sh, - # bashrcScript: ./files/bash-rc, - # bashloginScript: ./files/bash-login - # } + setValues: + service.type: LoadBalancer + coder.coderImage: eoepca/pde-code-server:1.0.0 + coder.workspace: notebooks + coderstorageClassName: standard + coder.workspaceStorage: 10Gi + coderResources.limits.cpu: '2' + coderResources.limits.memory: '6442450944' + coderResources.requests.cpu: '1' + coderResources.requests.memory: '4294967296' + calrissian.enabled: true + setFiles: { + initScript: ./files/init.sh, + bashrcScript: ./files/bash-rc, + bashloginScript: ./files/bash-login + } # MLflow - name: my-mlflow @@ -73,19 +73,15 @@ deploy: portForward: - resourceType: service resourceName: my-mlflow + address: localhost namespace: tile-based-training port: 5000 localPort: 5000 - # - resourceType: service - # resourceName: eoap-argo-workflows-server - # namespace: tile-based-training - # address: localhost - # port: 2746 - # localPort: 2746 - # - resourceType: service - # resourceName: tile-based-training-coder - # namespace: tile-based-training - # address: localhost - # port: 8080 - # localPort: 8000 + + - resourceType: service + resourceName: code-server-service # code server service name in ../chart/coder/templates/coder.yaml + namespace: tile-based-training + address: localhost + port: 8080 + localPort: 8000 From 6f095a3f42785ea6773ddb34370509a87a0b185f Mon Sep 17 00:00:00 2001 From: Fabrice Brito Date: Tue, 11 Mar 2025 15:09:11 +0100 Subject: [PATCH 18/61] adds dask-gateway and advanced tooling. updates how-to --- advanced-tooling/Dockerfile.coder | 125 +++++++++++++++++++++++ advanced-tooling/entrypoint.sh | 37 +++++++ advanced-tooling/skaffold.yaml | 19 +++- charts/coder/templates/coder.yaml | 2 + charts/coder/templates/kaniko-rbac.yaml | 51 +++++++++ charts/coder/templates/role-binding.yaml | 14 +++ charts/coder/templates/roles.yaml | 15 +++ charts/coder/values.yaml | 5 +- dask-gateway/README.md | 23 +++++ dask-gateway/files/bash-rc | 5 +- dask-gateway/files/init.sh | 23 ++++- dask-gateway/requirements.txt | 3 +- dask-gateway/skaffold.yaml | 26 +++-- how-to/files/bash-rc | 3 + 14 files changed, 335 insertions(+), 16 deletions(-) create mode 100644 advanced-tooling/Dockerfile.coder create mode 100644 advanced-tooling/entrypoint.sh create mode 100644 charts/coder/templates/kaniko-rbac.yaml create mode 100644 dask-gateway/README.md diff --git a/advanced-tooling/Dockerfile.coder b/advanced-tooling/Dockerfile.coder new file mode 100644 index 0000000..c19f367 --- /dev/null +++ b/advanced-tooling/Dockerfile.coder @@ -0,0 +1,125 @@ +FROM docker.io/library/python:3.10.12@sha256:bac3a0e0d16125977e351c861e2f4b12ecafeaa6f72431dc970d0b9155103232 + +RUN \ + apt-get update && \ + apt-get install -y \ + build-essential \ + curl \ + gcc \ + vim \ + tree \ + file + +RUN \ + echo "**** install node repo ****" && \ + curl -s https://deb.nodesource.com/gpgkey/nodesource.gpg.key | apt-key add - && \ + echo 'deb https://deb.nodesource.com/node_14.x jammy main' \ + > /etc/apt/sources.list.d/nodesource.list && \ + echo "**** install build dependencies ****" && \ + apt-get update && \ + apt-get install -y \ + nodejs + +RUN \ + echo "**** install runtime dependencies ****" && \ + apt-get install -y \ + git \ + jq \ + libatomic1 \ + nano \ + net-tools \ + sudo \ + podman \ + wget \ + python3 \ + python3-pip + +RUN \ + echo "**** install code-server ****" && \ + if [ -z ${CODE_RELEASE+x} ]; then \ + CODE_RELEASE=$(curl -sX GET https://api.github.com/repos/coder/code-server/releases/latest \ + | awk '/tag_name/{print $4;exit}' FS='[""]' | sed 's|^v||'); \ + fi && \ + mkdir -p /app/code-server && \ + curl -o \ + /tmp/code-server.tar.gz -L \ + "https://github.com/coder/code-server/releases/download/v${CODE_RELEASE}/code-server-${CODE_RELEASE}-linux-amd64.tar.gz" && \ + tar xf /tmp/code-server.tar.gz -C \ + /app/code-server --strip-components=1 && \ + echo "**** patch 4.0.2 ****" && \ + if [ "${CODE_RELEASE}" = "4.0.2" ] && [ "$(uname -m)" != "x86_64" ]; then \ + cd /app/code-server && \ + npm i --production @node-rs/argon2; \ + fi && \ + echo "**** clean up ****" && \ + apt-get purge --auto-remove -y \ + build-essential \ + nodejs && \ + apt-get clean && \ + rm -rf \ + /config/* \ + /tmp/* \ + /var/lib/apt/lists/* \ + /var/tmp/* \ + /etc/apt/sources.list.d/nodesource.list + +ENV USER=jovyan \ + UID=1001 \ + GID=100 \ + HOME=/workspace \ + PATH=/opt/conda/bin:/app/code-server/bin/:$PATH:/app/code-server/ + + +# RUN \ +# echo "**** install conda ****" && \ +# wget https://repo.anaconda.com/miniconda/Miniconda3-py39_4.12.0-Linux-x86_64.sh -O miniconda.sh -q && \ +# sh miniconda.sh -b -p /opt/conda && \ +# conda install -n base -c conda-forge mamba && \ +# conda config --system --append channels conda-forge && \ +# conda config --system --append channels terradue && \ +# conda config --system --append channels eoepca && \ +# conda config --system --append channels r && \ +# conda config --system --set auto_update_conda false && \ +# conda config --system --set show_channel_urls true && \ +# conda config --system --set channel_priority "flexible" + + +# RUN \ +# mamba install -n base cwltool cwl-wrapper==0.12.2 nodejs && \ +# mamba clean -a + +RUN \ + echo "**** install yq, aws cli ****" && \ + VERSION="v4.12.2" && \ + BINARY="yq_linux_amd64" && \ + wget --quiet https://github.com/mikefarah/yq/releases/download/${VERSION}/${BINARY}.tar.gz -O - |\ + tar xz && mv ${BINARY} /usr/bin/yq && \ + pip install awscli && \ + pip install awscli-plugin-endpoint + +RUN \ + echo "**** install jupyter-hub native proxy ****" && \ + pip install jhsingle-native-proxy>=0.0.9 && \ + echo "**** install bash kernel ****" && \ + pip install bash_kernel && \ + python -m bash_kernel.install + +RUN \ + echo "**** adds user jovyan ****" && \ + useradd -m -s /bin/bash -N -u $UID $USER + +COPY entrypoint.sh /opt/entrypoint.sh + +RUN chmod +x /opt/entrypoint.sh + +# RUN chown -R jovyan:100 /opt/conda + +# RUN \ +# echo "**** required by cwltool docker pull even if running with --podman ****" && \ +# ln -s /usr/bin/podman /usr/bin/docker + +ENTRYPOINT ["/opt/entrypoint.sh"] + +EXPOSE 8888 + +USER jovyan \ No newline at end of file diff --git a/advanced-tooling/entrypoint.sh b/advanced-tooling/entrypoint.sh new file mode 100644 index 0000000..f5b1057 --- /dev/null +++ b/advanced-tooling/entrypoint.sh @@ -0,0 +1,37 @@ +#!/bin/bash + +collect_port=0 +port="8888" +delim='=' + +for var in "$@" +do + echo "$var" + + if [ "$collect_port" == "1" ]; then + echo "Collecting external port $var" + port=$var + collect_port=0 + fi + + splitarg=${var%%$delim*} + + if [ "$splitarg" == "--port" ]; then + if [ ${#splitarg} == ${#var} ]; then + collect_port=1 + else + port=${var#*$delim} + echo "Setting external port $port" + fi + fi +done + +destport=$((port + 1)) + +echo "Using internal port $destport" + +if [ -z "$CODE_SERVER_WS" ]; then + CODE_SERVER_WS="/workspace" +fi + +jhsingle-native-proxy --port $port --destport $destport code-server {--}auth none {--}bind-addr 0.0.0.0:$destport {--}user-data-dir /workspace $CODE_SERVER_WS \ No newline at end of file diff --git a/advanced-tooling/skaffold.yaml b/advanced-tooling/skaffold.yaml index 0b3fe58..22c3433 100644 --- a/advanced-tooling/skaffold.yaml +++ b/advanced-tooling/skaffold.yaml @@ -2,7 +2,18 @@ apiVersion: skaffold/v4beta9 kind: Config metadata: name: eoap-advanced-tooling - + +build: + tagPolicy: + envTemplate: + template: 2h + + artifacts: + - image: ttl.sh/eoap-advanced-tooling-coder + context: . + docker: + dockerfile: Dockerfile.coder + deploy: helm: releases: @@ -11,9 +22,9 @@ deploy: namespace: eoap-advanced-tooling createNamespace: true setValues: - coder.coderImage: eoepca/pde-code-server:1.0.0 + coder.coderImage: ttl.sh/eoap-dask-gateway-coder:2h coder.workspace: advanced-tooling - coderstorageClassName: standard + coder.storageClassName: openebs-kernel-nfs-scw coder.workspaceStorage: 10Gi coderResources.limits.cpu: '2' coderResources.limits.memory: '6442450944' @@ -21,7 +32,9 @@ deploy: coderResources.requests.memory: '4294967296' coder.securityContext.privileged: false calrissian.enabled: true + calrissian.storageClassName: openebs-kernel-nfs-scw skaffold.enabled: true + kaniko.enabled: true setFiles: { initScript: ./files/init.sh, diff --git a/charts/coder/templates/coder.yaml b/charts/coder/templates/coder.yaml index fa7524f..e20e6cf 100644 --- a/charts/coder/templates/coder.yaml +++ b/charts/coder/templates/coder.yaml @@ -97,6 +97,8 @@ spec: - name: DASK_IMAGE value: "{{ .Values.daskGateway.image }}" {{- end }} + {{ if .Values.kaniko.enabled }} + {{- end }} resources: limits: cpu: {{ .Values.coderResources.limits.cpu }} diff --git a/charts/coder/templates/kaniko-rbac.yaml b/charts/coder/templates/kaniko-rbac.yaml new file mode 100644 index 0000000..38ee269 --- /dev/null +++ b/charts/coder/templates/kaniko-rbac.yaml @@ -0,0 +1,51 @@ +{{- if .Values.kaniko.enabled -}} +apiVersion: v1 +kind: Secret +metadata: + name: kaniko-secret +type: kubernetes.io/dockerconfigjson +data: + .dockerconfigjson: ew0KICAgICJhdXRocyI6IHsNCiAgICAgICAgImZha2UucmVnaXN0cnkuaW8iOiB7DQogICAgICAgICAgICAidXNlcm5hbWUiOiAiZmFrZXVzZXIiLA0KICAgICAgICAgICAgInBhc3N3b3JkIjogImZha2VwYXNzd29yZCIsDQogICAgICAgICAgICAiZW1haWwiOiAiZmFrZUBleGFtcGxlLmNvbSIsDQogICAgICAgICAgICAiYXV0aCI6ICIiDQogICAgICAgIH0NCiAgICB9DQp9 +--- +apiVersion: v1 +kind: Secret +metadata: + name: ttl-secret +type: kubernetes.io/dockerconfigjson +data: + .dockerconfigjson: ew0KICAgICJhdXRocyI6IHsNCiAgICAgICAgImZha2UucmVnaXN0cnkuaW8iOiB7DQogICAgICAgICAgICAidXNlcm5hbWUiOiAiZmFrZXVzZXIiLA0KICAgICAgICAgICAgInBhc3N3b3JkIjogImZha2VwYXNzd29yZCIsDQogICAgICAgICAgICAiZW1haWwiOiAiZmFrZUBleGFtcGxlLmNvbSIsDQogICAgICAgICAgICAiYXV0aCI6ICIiDQogICAgICAgIH0NCiAgICB9DQp9 +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: kaniko-sa +imagePullSecrets: + - name: kaniko-secret +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: kaniko-role +rules: + - apiGroups: [""] + resources: ["pods"] + verbs: ["create", "get", "list", "watch", "delete"] + - apiGroups: [""] + resources: ["pods/exec"] + verbs: ["create"] + - apiGroups: [""] + resources: ["pods/log"] + verbs: ["get","list"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: kaniko-rolebinding +subjects: + - kind: ServiceAccount + name: kaniko-sa +roleRef: + kind: Role + name: kaniko-role + apiGroup: rbac.authorization.k8s.io +{{- end -}} \ No newline at end of file diff --git a/charts/coder/templates/role-binding.yaml b/charts/coder/templates/role-binding.yaml index eba5b8d..9b0c330 100644 --- a/charts/coder/templates/role-binding.yaml +++ b/charts/coder/templates/role-binding.yaml @@ -46,6 +46,20 @@ roleRef: kind: Role name: pod-exec-role subjects: +- kind: ServiceAccount + name: calrissian-sa +{{- end }} +{{- if .Values.daskGateway.enabled }} +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: dask-configmap-binding +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: dask-configmap-role +subjects: - kind: ServiceAccount name: calrissian-sa {{- end }} \ No newline at end of file diff --git a/charts/coder/templates/roles.yaml b/charts/coder/templates/roles.yaml index 2b93aa0..5db4adb 100644 --- a/charts/coder/templates/roles.yaml +++ b/charts/coder/templates/roles.yaml @@ -39,4 +39,19 @@ rules: - '' resources: - pods/exec +{{- end }} +{{- if .Values.daskGateway.enabled }} +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: dask-configmap-role +rules: +- verbs: + - create + - delete + apiGroups: + - '' + resources: + - configmaps {{- end }} \ No newline at end of file diff --git a/charts/coder/values.yaml b/charts/coder/values.yaml index a2aae37..f8a0dfa 100644 --- a/charts/coder/values.yaml +++ b/charts/coder/values.yaml @@ -28,4 +28,7 @@ skaffold: daskGateway: enabled: False daskGatewayUrl: "" - image: "" \ No newline at end of file + image: "" + +kaniko: + enabled: False \ No newline at end of file diff --git a/dask-gateway/README.md b/dask-gateway/README.md new file mode 100644 index 0000000..92a48e6 --- /dev/null +++ b/dask-gateway/README.md @@ -0,0 +1,23 @@ +# Dask Gateway + +This folder contains a skaffold configuration to deploy Dask Gateway and Code Server. + +This deployment is used to run the e-learning module [Dask application package](https://github.com/eoap/dask-app-package) + +## Usage + +Use `skaffold` to deploy on minikube: + +``` +skaffold dev +``` + +You can also use a remote cluster with: + +``` +skaffold dev --default-repo +``` + +## Why the deployment is so long? + +THe code server pod has an init container that installs several tools, code extensions and python packages and this takes time. \ No newline at end of file diff --git a/dask-gateway/files/bash-rc b/dask-gateway/files/bash-rc index 00dd390..4c25a44 100644 --- a/dask-gateway/files/bash-rc +++ b/dask-gateway/files/bash-rc @@ -1,5 +1,8 @@ alias ll="ls -l" -alias aws="aws --endpoint-url=http://eoap-mastering-app-package-localstack:4566" +alias aws="aws --endpoint-url=http://eoap-dask-gateway-localstack:4566" + +export PATH=/workspace/.local/bin:/workspace/.venv/bin:$PATH +export TASK_X_REMOTE_TASKFILES=1 source /workspace/.venv/bin/activate \ No newline at end of file diff --git a/dask-gateway/files/init.sh b/dask-gateway/files/init.sh index 1a17a5e..02b5b1c 100644 --- a/dask-gateway/files/init.sh +++ b/dask-gateway/files/init.sh @@ -2,6 +2,8 @@ set -x +export PATH=/workspace/.local/bin:/workspace/.venv/bin:$PATH + cd /workspace git clone 'https://github.com/eoap/dask-app-package.git' @@ -10,6 +12,7 @@ code-server --install-extension ms-python.python code-server --install-extension redhat.vscode-yaml code-server --install-extension sbg-rabix.benten-cwl code-server --install-extension ms-toolsai.jupyter +code-server --install-extension tamasfe.even-better-toml ln -s /workspace/.local/share/code-server/extensions /workspace/extensions @@ -17,9 +20,11 @@ mkdir -p /workspace/User/ echo '{"workbench.colorTheme": "Visual Studio Dark"}' > /workspace/User/settings.json +pip install git+https://github.com/Terradue/calrissian@dask-gateway tomlq # calrissian and tomlq (for task files) + python -m venv /workspace/.venv source /workspace/.venv/bin/activate -/workspace/.venv/bin/python -m pip install --no-cache-dir numpy==1.26.3 dask==2024.1.0 distributed==2024.1.0 dask-gateway==2024.1.0 pystac bokeh rioxarray==0.18.1 loguru==0.7.3 odc-stac[botocore]==0.3.10 stackstac==0.5.1 pystac-client planetary-computer +/workspace/.venv/bin/python -m pip install --no-cache-dir numpy==1.26.3 dask==2024.1.0 distributed==2024.1.0 dask-gateway==2024.1.0 pystac bokeh rioxarray==0.18.1 loguru==0.7.3 odc-stac[botocore]==0.3.10 stackstac==0.5.1 pystac-client planetary-computer xarray-spatial ipykernel res=$? if [ $res -ne 0 ]; then echo "Failed to install packages" @@ -39,4 +44,18 @@ export AWS_ACCESS_KEY_ID="test" export AWS_SECRET_ACCESS_KEY="test" -aws s3 mb s3://results --endpoint-url=http://eoap-dask-gateway-localstack:4566 \ No newline at end of file +aws s3 mb s3://results --endpoint-url=http://eoap-dask-gateway-localstack:4566 + + +curl -s -L https://github.com/pypa/hatch/releases/latest/download/hatch-x86_64-unknown-linux-gnu.tar.gz | tar -xzvf - -C /workspace/.venv/bin/ +chmod +x /workspace/.venv/bin/hatch + +curl -s -L https://github.com/go-task/task/releases/download/v3.41.0/task_linux_amd64.tar.gz | tar -xzvf - -C /workspace/.venv/bin/ +chmod +x /workspace/.venv/bin/task + +curl -s -LO https://github.com/mikefarah/yq/releases/download/v4.45.1/yq_linux_amd64.tar.gz +tar -xvf yq_linux_amd64.tar.gz +mv yq_linux_amd64 /workspace/.venv/bin/yq + +curl -s -L https://storage.googleapis.com/skaffold/releases/latest/skaffold-linux-amd64 > /workspace/.venv/bin/skaffold +chmod +x /workspace/.venv/bin/skaffold diff --git a/dask-gateway/requirements.txt b/dask-gateway/requirements.txt index 0914146..0687b22 100644 --- a/dask-gateway/requirements.txt +++ b/dask-gateway/requirements.txt @@ -9,4 +9,5 @@ loguru==0.7.3 odc-stac[botocore]==0.3.10 stackstac==0.5.1 pystac-client -planetary-computer \ No newline at end of file +planetary-computer +xarray-spatial \ No newline at end of file diff --git a/dask-gateway/skaffold.yaml b/dask-gateway/skaffold.yaml index 6b3162f..21f30f6 100644 --- a/dask-gateway/skaffold.yaml +++ b/dask-gateway/skaffold.yaml @@ -4,16 +4,20 @@ metadata: name: eoap-dask-gateway build: + tagPolicy: + envTemplate: + template: 2h + artifacts: - - image: worker + - image: ttl.sh/eoap-dask-gateway-worker context: . docker: dockerfile: Dockerfile.worker - - image: daskclient + - image: ttl.sh/eoap-dask-gateway-daskclient context: . docker: dockerfile: Dockerfile.client - - image: coder + - image: ttl.sh/eoap-dask-gateway-coder context: . docker: dockerfile: Dockerfile.coder @@ -28,8 +32,10 @@ deploy: valuesFiles: - values.yaml setValueTemplates: - gateway.backend.image.name: "{{.IMAGE_NAME_worker}}" - gateway.backend.image.tag: "{{.IMAGE_TAG_worker}}" + gateway.backend.image.name: ttl.sh/eoap-dask-gateway-worker + gateway.backend.image.tag: 2h + gateway.backend.imagePullSecrets: + - name: kaniko-secret traefik.service.type: "ClusterIP" - name: eoap-dask-gateway @@ -37,19 +43,22 @@ deploy: namespace: eoap-dask-gateway createNamespace: true setValueTemplates: - coder.coderImage: "{{.IMAGE_NAME_coder}}" - daskGateway.image: "{{.IMAGE_NAME_worker}}:{{.IMAGE_TAG_worker}}" + coder.coderImage: ttl.sh/eoap-dask-gateway-coder + daskGateway.image: ttl.sh/eoap-dask-gateway-worker:2h setValues: coder.workspace: dask-app-package - coderstorageClassName: standard + coder.storageClassName: openebs-kernel-nfs-scw coder.workspaceStorage: 10Gi coderResources.limits.cpu: '2' coderResources.limits.memory: '6442450944' coderResources.requests.cpu: '1' coderResources.requests.memory: '4294967296' calrissian.enabled: true + calrissian.storageClassName: openebs-kernel-nfs-scw + skaffold.enabled: true daskGateway.enabled: true daskGateway.daskGatewayUrl: "http://traefik-dask-gateway.eoap-dask-gateway.svc.cluster.local:80" + kaniko.enabled: true setFiles: { initScript: ./files/init.sh, bashrcScript: ./files/bash-rc, @@ -61,6 +70,7 @@ deploy: createNamespace: true setValues: service.type: ClusterIP + hooks: before: [] after: diff --git a/how-to/files/bash-rc b/how-to/files/bash-rc index b4d7f46..00102c4 100644 --- a/how-to/files/bash-rc +++ b/how-to/files/bash-rc @@ -2,4 +2,7 @@ alias ll="ls -l" alias aws="aws --endpoint-url=http://eoap-how-to-localstack:4566" +export PATH=/workspace/.venv/bin:$PATH +export TASK_X_REMOTE_TASKFILES=1 + source /workspace/.venv/bin/activate \ No newline at end of file From a31e875079a4d571bd7c79dd3ab381507140d427 Mon Sep 17 00:00:00 2001 From: pmembari Date: Tue, 11 Mar 2025 17:41:33 +0100 Subject: [PATCH 19/61] feat: remove the configmaps --- machine-learning-process/skaffold.yaml | 27 ++++++++++++-------------- 1 file changed, 12 insertions(+), 15 deletions(-) diff --git a/machine-learning-process/skaffold.yaml b/machine-learning-process/skaffold.yaml index e9a9446..0f31e6a 100644 --- a/machine-learning-process/skaffold.yaml +++ b/machine-learning-process/skaffold.yaml @@ -5,26 +5,24 @@ metadata: deploy: helm: releases: - - # - name: tile-based-localstack - # remoteChart: localstack/localstack - # namespace: tile-based-training - # createNamespace: true - # wait: true - # setValues: - # service.type: LoadBalancer + + - name: tile-based-training-localstack + remoteChart: localstack/localstack + namespace: tile-based-training + createNamespace: true + setValues: + service.type: LoadBalancer # Code-Server with Calrissian - - name: tile-based-training-coder + - name: code-server chartPath: ../charts/coder namespace: tile-based-training createNamespace: true - wait: true - setValues: service.type: LoadBalancer + storageClassRWX: "standard" coder.coderImage: eoepca/pde-code-server:1.0.0 - coder.workspace: notebooks + coder.workspace: machine-learning-process coderstorageClassName: standard coder.workspaceStorage: 10Gi coderResources.limits.cpu: '2' @@ -35,15 +33,14 @@ deploy: setFiles: { initScript: ./files/init.sh, bashrcScript: ./files/bash-rc, - bashloginScript: ./files/bash-login + bashloginScript: ./files/bash-login, } - + # MLflow - name: my-mlflow remoteChart: community-charts/mlflow namespace: tile-based-training createNamespace: true - wait: true setValues: service.type: ClusterIP service.port: 5000 From c0dee430ad65669c6a5b5789a1061ac44d0e9b62 Mon Sep 17 00:00:00 2001 From: pmembari Date: Tue, 11 Mar 2025 17:41:55 +0100 Subject: [PATCH 20/61] feat: change strategy --- .../charts/argo-cwl-runner/.helmignore | 23 -- .../charts/argo-cwl-runner/Chart.yaml | 24 -- .../argo-cwl-runner/files/user-settings.json | 15 - .../templates/argo-calrissian.yaml | 324 ------------------ .../argo-cwl-runner/templates/semaphore.yaml | 7 - .../templates/user-settings.yaml | 9 - .../charts/argo-cwl-runner/values.yaml | 2 - .../charts/event-driven/Chart.yaml | 6 - .../event-driven/templates/event-bus.yaml | 18 - .../templates/event-sensor-log.yaml | 16 - .../event-driven/templates/event-sensor.yaml | 46 --- .../event-driven/templates/event-source.yaml | 13 - .../templates/local-stack-secret.yaml | 10 - .../event-driven/templates/role-binding.yaml | 11 - .../charts/event-driven/templates/roles.yaml | 52 --- .../charts/event-driven/values.yaml | 4 - .../containers/stage-out/Dockerfile | 15 - .../containers/stage-out/README.md | 49 --- .../containers/stage-out/app/__init__.py | 0 .../containers/stage-out/app/main.py | 94 ----- .../containers/stage-out/app/stac.py | 77 ----- .../containers/stage-out/app/usersettings.py | 63 ---- .../containers/stage-out/requirements.txt | 5 - .../containers/stage-out/setup.cfg | 3 - .../containers/stage-out/setup.py | 12 - .../containers/validate-schema/Dockerfile | 5 - machine-learning-process/files/init.sh | 18 +- 27 files changed, 11 insertions(+), 910 deletions(-) delete mode 100644 machine-learning-process/charts/argo-cwl-runner/.helmignore delete mode 100644 machine-learning-process/charts/argo-cwl-runner/Chart.yaml delete mode 100644 machine-learning-process/charts/argo-cwl-runner/files/user-settings.json delete mode 100644 machine-learning-process/charts/argo-cwl-runner/templates/argo-calrissian.yaml delete mode 100644 machine-learning-process/charts/argo-cwl-runner/templates/semaphore.yaml delete mode 100644 machine-learning-process/charts/argo-cwl-runner/templates/user-settings.yaml delete mode 100644 machine-learning-process/charts/argo-cwl-runner/values.yaml delete mode 100644 machine-learning-process/charts/event-driven/Chart.yaml delete mode 100644 machine-learning-process/charts/event-driven/templates/event-bus.yaml delete mode 100644 machine-learning-process/charts/event-driven/templates/event-sensor-log.yaml delete mode 100644 machine-learning-process/charts/event-driven/templates/event-sensor.yaml delete mode 100644 machine-learning-process/charts/event-driven/templates/event-source.yaml delete mode 100644 machine-learning-process/charts/event-driven/templates/local-stack-secret.yaml delete mode 100644 machine-learning-process/charts/event-driven/templates/role-binding.yaml delete mode 100644 machine-learning-process/charts/event-driven/templates/roles.yaml delete mode 100644 machine-learning-process/charts/event-driven/values.yaml delete mode 100644 machine-learning-process/containers/stage-out/Dockerfile delete mode 100644 machine-learning-process/containers/stage-out/README.md delete mode 100644 machine-learning-process/containers/stage-out/app/__init__.py delete mode 100644 machine-learning-process/containers/stage-out/app/main.py delete mode 100644 machine-learning-process/containers/stage-out/app/stac.py delete mode 100644 machine-learning-process/containers/stage-out/app/usersettings.py delete mode 100644 machine-learning-process/containers/stage-out/requirements.txt delete mode 100644 machine-learning-process/containers/stage-out/setup.cfg delete mode 100644 machine-learning-process/containers/stage-out/setup.py delete mode 100644 machine-learning-process/containers/validate-schema/Dockerfile diff --git a/machine-learning-process/charts/argo-cwl-runner/.helmignore b/machine-learning-process/charts/argo-cwl-runner/.helmignore deleted file mode 100644 index 0e8a0eb..0000000 --- a/machine-learning-process/charts/argo-cwl-runner/.helmignore +++ /dev/null @@ -1,23 +0,0 @@ -# Patterns to ignore when building packages. -# This supports shell glob matching, relative path matching, and -# negation (prefixed with !). Only one pattern per line. -.DS_Store -# Common VCS dirs -.git/ -.gitignore -.bzr/ -.bzrignore -.hg/ -.hgignore -.svn/ -# Common backup files -*.swp -*.bak -*.tmp -*.orig -*~ -# Various IDEs -.project -.idea/ -*.tmproj -.vscode/ diff --git a/machine-learning-process/charts/argo-cwl-runner/Chart.yaml b/machine-learning-process/charts/argo-cwl-runner/Chart.yaml deleted file mode 100644 index ac56d50..0000000 --- a/machine-learning-process/charts/argo-cwl-runner/Chart.yaml +++ /dev/null @@ -1,24 +0,0 @@ -apiVersion: v2 -name: argo-cwl-runner -description: A Helm chart for Kubernetes - -# A chart can be either an 'application' or a 'library' chart. -# -# Application charts are a collection of templates that can be packaged into versioned archives -# to be deployed. -# -# Library charts provide useful utilities or functions for the chart developer. They're included as -# a dependency of application charts to inject those utilities and functions into the rendering -# pipeline. Library charts do not define any templates and therefore cannot be deployed. -type: application - -# This is the chart version. This version number should be incremented each time you make changes -# to the chart and its templates, including the app version. -# Versions are expected to follow Semantic Versioning (https://semver.org/) -version: 0.1.0 - -# This is the version number of the application being deployed. This version number should be -# incremented each time you make changes to the application. Versions are not expected to -# follow Semantic Versioning. They should reflect the version the application is using. -# It is recommended to use it with quotes. -appVersion: "1.16.0" diff --git a/machine-learning-process/charts/argo-cwl-runner/files/user-settings.json b/machine-learning-process/charts/argo-cwl-runner/files/user-settings.json deleted file mode 100644 index cf9eede..0000000 --- a/machine-learning-process/charts/argo-cwl-runner/files/user-settings.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "S3": { - "Services": { - "results": { - "UrlPattern": "s3:\/\/results\/.*", - "Region": "us-east-1", - "AuthenticationRegion": "us-east-1", - "AccessKey": "test", - "SecretKey": "test", - "ServiceURL": "http://eoap-event-driven-localstack.eoap-event-driven.svc.cluster.local:4566", - "ForcePathStyle": "true" - } - } - } -} \ No newline at end of file diff --git a/machine-learning-process/charts/argo-cwl-runner/templates/argo-calrissian.yaml b/machine-learning-process/charts/argo-cwl-runner/templates/argo-calrissian.yaml deleted file mode 100644 index 1aa265a..0000000 --- a/machine-learning-process/charts/argo-cwl-runner/templates/argo-calrissian.yaml +++ /dev/null @@ -1,324 +0,0 @@ -apiVersion: argoproj.io/v1alpha1 -kind: WorkflowTemplate -metadata: - name: argo-cwl-runner - annotations: - workflows.argoproj.io/version: ">= v3.3.0" - workflows.argoproj.io/description: | - This workflow template is a CWL runner for Argo Workflows. -spec: - entrypoint: calrissian-runner - parameters: - - name: parameters - description: "Parameters in JSON format" - value: "" - - name: cwl - description: "CWL document in JSON format" - value: "" - - name: max_ram - description: "Max RAM" - value: "8G" - - name: max_cores - description: "Max cores" - value: "8" - # this is the Workflow throttling to avoid overloading the cluster - # the configMap contains the number of concurrent Workflows allowed - synchronization: - semaphore: - configMapKeyRef: - name: semaphore-argo-cwl-runner - key: workflow - - # volumes - volumes: - - name: usersettings-vol - secret: - secretName: user-settings - - name: calrissian-wdir - persistentVolumeClaim: - claimName: calrissian-wdir - - # Workflow templates are used to define the Workflow steps - templates: - - name: calrissian-runner - # this is the entry point of the Workflow - inputs: - parameters: - - name: parameters - - name: cwl - - name: max_ram - - name: max_cores - outputs: - parameters: - - name: results - valueFrom: - parameter: {{ `"{{steps.get-results.outputs.parameters.calrissian-output}}"` }} - - name: log - valueFrom: - parameter: {{ `"{{steps.get-results.outputs.parameters.calrissian-stderr}}"` }} - - name: usage-report - valueFrom: - parameter: {{ `"{{steps.get-results.outputs.parameters.calrissian-report}}"` }} - - name: stac-catalog - valueFrom: - parameter: {{ `"{{steps.stage-out.outputs.parameters.stac-catalog}}"` }} - artifacts: - - name: tool-logs - from: {{ `"{{steps.get-results.outputs.artifacts.tool-logs}}"` }} - - name: calrissian-output - from: {{ `"{{steps.get-results.outputs.artifacts.calrissian-output}}"` }} - - name: calrissian-stderr - from: {{ `"{{steps.get-results.outputs.artifacts.calrissian-stderr}}"` }} - - name: calrissian-report - from: {{ `"{{steps.get-results.outputs.artifacts.calrissian-report}}"` }} - - steps: - # Workflow steps are defined here - - - name: cwl-prepare - template: cwl-prepare - arguments: - parameters: - - name: cwl - value: {{ `"{{inputs.parameters.cwl}}"` }} - - name: parameters - value: {{ `"{{inputs.parameters.parameters}}"` }} - - - - name: cwl-runner - template: calrissian-tmpl - arguments: - parameters: - - name: max_ram - value: {{ `"{{inputs.parameters.max_ram}}"` }} - - name: max_cores - value: {{ `"{{inputs.parameters.max_cores}}"` }} - - - - name: get-results - # these can run in parallel - # the same template is used for all the steps - template: get-results - arguments: - parameters: - - name: calrissian-output - value: "/calrissian/output.json" - - name: calrissian-stderr - value: "/calrissian/stderr.log" - - name: calrissian-report - value: "/calrissian/report.json" - - - name: stage-out - template: stage-out - arguments: - parameters: - - name: file_path - value: "/calrissian/output.json" - - name: bucket - value: "results" - - name: folder - value: {{ `"{{workflow.name}}-{{workflow.uid}}"` }} - - - name: cwl-prepare - # this steps prepares the CWL inputs - # needed by Calrissian - inputs: - parameters: - - name: cwl - - name: parameters - - script: - image: busybox:1.35.0 - resources: - requests: - memory: 1Gi - cpu: 1 - volumeMounts: - - name: calrissian-wdir - mountPath: /calrissian - env: [] - command: [ash] - source: | - #!/bin/ash - echo {{ `'{{inputs.parameters.cwl}}'` }} >> /calrissian/cwl.json - echo {{ `'{{inputs.parameters.parameters}}'` }} >> /calrissian/input.json - sleep 1 - - - name: calrissian-tmpl - # this step creates the Calrissian Job, Argo creates it as a Kubernetes Job - resource: - action: create - setOwnerReference: true - successCondition: status.succeeded > 0 - failureCondition: status.failed > 3 - manifest: | - apiVersion: batch/v1 - kind: Job - metadata: - generateName: calrissian-water-bodies-detection- - spec: - backoffLimit: 1 - activeDeadlineSeconds: 86400 - ttlSecondsAfterFinished: 120 - template: - metadata: - name: calrissian_pod - spec: - serviceAccountName: argo - containers: - - name: calrissian - image: ghcr.io/duke-gcb/calrissian/calrissian:0.16.0 - imagePullPolicy: IfNotPresent - command: - - calrissian - args: - - --debug - - --pod-serviceaccount - - argo - - --stdout - - /calrissian/output.json - - --stderr - - /calrissian/stderr.log - - --usage-report - - /calrissian/report.json - - --max-ram - - {{ `'{{inputs.parameters.max_ram}}'` }} - - --max-cores - - {{ `'{{inputs.parameters.max_cores}}'` }} - - --tmp-outdir-prefix - - /calrissian/tmp/ - - --outdir - - /calrissian/results/ - - --tool-logs-basepath - - /calrissian/logs - - "/calrissian/cwl.json" - - "/calrissian/input.json" - env: - - name: CALRISSIAN_POD_NAME - valueFrom: - fieldRef: - fieldPath: metadata.name - - name: CALRISSIAN_DELETE_PODS - value: "true" - resources: - limits: - cpu: 2000m - memory: 2G - requests: - cpu: 1000m - memory: 1G - volumeMounts: - - mountPath: /calrissian - name: calrissian-wdir - readOnly: false - restartPolicy: Never - securityContext: - fsGroup: 0 - runAsGroup: 0 - runAsUser: 0 - terminationGracePeriodSeconds: 120 - volumes: - - name: calrissian-wdir - persistentVolumeClaim: - claimName: {{"{{workflow.name}}"}}-calrissian-wdir - readOnly: false - - inputs: - parameters: - - name: max_ram - - name: max_cores - outputs: - parameters: [] - artifacts: - - name: logs - path: /calrissian/logs - s3: - key: {{ `"{{workflow.name}}-{{workflow.uid}}-artifacts/calrissian-logs.tgz"` }} - - - - - name: get-results - # reads the files generated by Calrissian - inputs: - parameters: - - name: calrissian-output - - name: calrissian-stderr - - name: calrissian-report - outputs: - parameters: - - name: calrissian-output - valueFrom: - path: /tmp/calrissian-output.json - - name: calrissian-stderr - valueFrom: - path: /tmp/calrissian-stderr.txt - - name: calrissian-report - valueFrom: - path: /tmp/calrissian-report.json - artifacts: - - name: tool-logs - path: /calrissian/logs - s3: - key: {{ `"{{workflow.name}}-{{workflow.uid}}-artifacts/tool-logs.tgz"` }} - - name: calrissian-output - path: /tmp/calrissian-output.json - s3: - key: {{ `"{{workflow.name}}-{{workflow.uid}}-artifacts/calrissian-output.tgz"` }} - - name: calrissian-stderr - path: /tmp/calrissian-stderr.txt - s3: - key: {{ `"{{workflow.name}}-{{workflow.uid}}-artifacts/calrissian-stderr.tgz"` }} - - name: calrissian-report - path: /tmp/calrissian-report.json - s3: - key: {{ `"{{workflow.name}}-{{workflow.uid}}-artifacts/calrissian-report.tgz"` }} - script: - image: busybox:1.35.0 - resources: - requests: - memory: 1Gi - cpu: 1 - volumeMounts: - - name: calrissian-wdir - mountPath: /calrissian - command: [ash] - source: | - #!/bin/ash - cat "{{"{{inputs.parameters.calrissian-output}}"}}" > /tmp/calrissian-output.json - cat "{{"{{inputs.parameters.calrissian-stderr}}"}}" > /tmp/calrissian-stderr.txt - cat "{{"{{inputs.parameters.calrissian-report}}"}}" > /tmp/calrissian-report.json - - - name: stage-out - inputs: - parameters: - - name: file_path - - name: bucket - - name: folder - outputs: - parameters: - - name: stac-catalog - valueFrom: - path: /tmp/output - script: - image: {{ .Values.stageOutImage }} - resources: - requests: - memory: 1Gi - cpu: 1 - volumeMounts: - - name: calrissian-wdir - mountPath: /calrissian - - name: usersettings-vol - readOnly: true - mountPath: "/etc/secret" - command: [bash] - source: - #!/bin/bash - set -x - - cat /etc/secret/usersettings.json - - stage-out --stac-catalog $( cat {{"{{inputs.parameters.file_path}}"}} | jq -r .stac_catalog.path - ) --user-settings /etc/secret/usersettings.json --bucket "{{"{{inputs.parameters.bucket}}"}}" --subfolder "{{"{{inputs.parameters.folder}}"}}" - - res=$? - - echo "s3://{{"{{inputs.parameters.bucket}}"}}/{{"{{inputs.parameters.folder}}"}}/catalog.json" > /tmp/output - - exit $res \ No newline at end of file diff --git a/machine-learning-process/charts/argo-cwl-runner/templates/semaphore.yaml b/machine-learning-process/charts/argo-cwl-runner/templates/semaphore.yaml deleted file mode 100644 index 7751209..0000000 --- a/machine-learning-process/charts/argo-cwl-runner/templates/semaphore.yaml +++ /dev/null @@ -1,7 +0,0 @@ -apiVersion: v1 -kind: ConfigMap -metadata: - name: semaphore-argo-cwl-runner -data: - workflow: "{{ .Values.semaphore }}" - \ No newline at end of file diff --git a/machine-learning-process/charts/argo-cwl-runner/templates/user-settings.yaml b/machine-learning-process/charts/argo-cwl-runner/templates/user-settings.yaml deleted file mode 100644 index 6bdf160..0000000 --- a/machine-learning-process/charts/argo-cwl-runner/templates/user-settings.yaml +++ /dev/null @@ -1,9 +0,0 @@ -# secret with the user-settings.json, it transforms the value yaml to json -apiVersion: v1 -kind: Secret -metadata: - name: user-settings - labels: - app: argo-water-bodies -data: - usersettings.json: {{ required "A valid .Values.userSettings entry required!" ( tpl ( .Values.userSettings | default ( .Files.Get "files/user-settings.json")) . | b64enc) }} \ No newline at end of file diff --git a/machine-learning-process/charts/argo-cwl-runner/values.yaml b/machine-learning-process/charts/argo-cwl-runner/values.yaml deleted file mode 100644 index 23d307c..0000000 --- a/machine-learning-process/charts/argo-cwl-runner/values.yaml +++ /dev/null @@ -1,2 +0,0 @@ -stageOutImage: stageout -semaphore: 3 \ No newline at end of file diff --git a/machine-learning-process/charts/event-driven/Chart.yaml b/machine-learning-process/charts/event-driven/Chart.yaml deleted file mode 100644 index b23913a..0000000 --- a/machine-learning-process/charts/event-driven/Chart.yaml +++ /dev/null @@ -1,6 +0,0 @@ -apiVersion: v2 -name: event-driven -description: Event driven manifests including event bus and source/sensor -type: application -version: 0.1.0 -appVersion: "1.0.0" diff --git a/machine-learning-process/charts/event-driven/templates/event-bus.yaml b/machine-learning-process/charts/event-driven/templates/event-bus.yaml deleted file mode 100644 index ca28d85..0000000 --- a/machine-learning-process/charts/event-driven/templates/event-bus.yaml +++ /dev/null @@ -1,18 +0,0 @@ -apiVersion: argoproj.io/v1alpha1 -kind: EventBus -metadata: - name: jetstream-event-bus -spec: - jetstream: - version: latest - replicas: {{ .Values.jetstreamReplicas }} - persistence: - storageClassName: standard - accessMode: ReadWriteOnce - volumeSize: {{ .Values.jetstreamVolumeSize }} - streamConfig: | - maxAge: 24h - settings: | - max_file_store: 1GB - startArgs: - - "-D" \ No newline at end of file diff --git a/machine-learning-process/charts/event-driven/templates/event-sensor-log.yaml b/machine-learning-process/charts/event-driven/templates/event-sensor-log.yaml deleted file mode 100644 index dd9e8d5..0000000 --- a/machine-learning-process/charts/event-driven/templates/event-sensor-log.yaml +++ /dev/null @@ -1,16 +0,0 @@ -apiVersion: argoproj.io/v1alpha1 -kind: Sensor -metadata: - name: acme-sentinel2-stream-source-logger -spec: - eventBusName: jetstream-event-bus - - dependencies: - - name: acme-sentinel2-stream-source-logger - eventSourceName: acme-sentinel2-stream-source - eventName: acme-sentinel2-stream-source-collected - triggers: - - template: - name: log-event - log: - intervalSeconds: 5 # Frequency of log updates \ No newline at end of file diff --git a/machine-learning-process/charts/event-driven/templates/event-sensor.yaml b/machine-learning-process/charts/event-driven/templates/event-sensor.yaml deleted file mode 100644 index f22053c..0000000 --- a/machine-learning-process/charts/event-driven/templates/event-sensor.yaml +++ /dev/null @@ -1,46 +0,0 @@ -apiVersion: argoproj.io/v1alpha1 -kind: Sensor -metadata: - name: acme-water-bodies-detection-sensor -spec: - eventBusName: jetstream-event-bus - template: - serviceAccountName: argo - dependencies: - - name: acme-sentinel2-stream - eventSourceName: acme-sentinel2-stream-source - eventName: acme-sentinel2-stream-source-collected - transform: - jq: ".values.href = [ .values.href ]" - triggers: - - template: - name: water-bodies-detection-trigger - k8s: - group: argoproj.io - version: v1alpha1 - resource: workflows - operation: create - source: - resource: - apiVersion: argoproj.io/v1alpha1 - kind: Workflow - metadata: - generateName: water-bodies-detection- - namespace: eoap-event-driven - spec: - serviceAccountName: argo - entrypoint: main - arguments: - parameters: - - name: items - - name: aoi - value: "-121.399,39.834,-120.74,40.472" - - name: epsg - value: "EPSG:4326" - workflowTemplateRef: - name: water-bodies-detection - parameters: - - src: - dependencyName: acme-sentinel2-stream - dataKey: values.href - dest: spec.arguments.parameters.0.value diff --git a/machine-learning-process/charts/event-driven/templates/event-source.yaml b/machine-learning-process/charts/event-driven/templates/event-source.yaml deleted file mode 100644 index 4811ea3..0000000 --- a/machine-learning-process/charts/event-driven/templates/event-source.yaml +++ /dev/null @@ -1,13 +0,0 @@ -apiVersion: argoproj.io/v1alpha1 -kind: EventSource -metadata: - name: acme-sentinel2-stream-source -spec: - eventBusName: jetstream-event-bus - redisStream: - acme-sentinel2-stream-source-collected: - hostAddress: redis-service:6379 - db: 0 - streams: - - {{ .Values.streamName }} - consumerGroup: acme-sentinel2-stream-source \ No newline at end of file diff --git a/machine-learning-process/charts/event-driven/templates/local-stack-secret.yaml b/machine-learning-process/charts/event-driven/templates/local-stack-secret.yaml deleted file mode 100644 index b9441fa..0000000 --- a/machine-learning-process/charts/event-driven/templates/local-stack-secret.yaml +++ /dev/null @@ -1,10 +0,0 @@ -apiVersion: v1 -stringData: - accesskey: test - secretkey: test -kind: Secret -metadata: - name: localstack-cred - labels: - app: localstack -type: Opaque \ No newline at end of file diff --git a/machine-learning-process/charts/event-driven/templates/role-binding.yaml b/machine-learning-process/charts/event-driven/templates/role-binding.yaml deleted file mode 100644 index 84d742c..0000000 --- a/machine-learning-process/charts/event-driven/templates/role-binding.yaml +++ /dev/null @@ -1,11 +0,0 @@ -apiVersion: rbac.authorization.k8s.io/v1 -kind: RoleBinding -metadata: - name: argo-extended-role-binding -roleRef: - apiGroup: rbac.authorization.k8s.io - kind: Role - name: argo-extended-role -subjects: -- kind: ServiceAccount - name: argo \ No newline at end of file diff --git a/machine-learning-process/charts/event-driven/templates/roles.yaml b/machine-learning-process/charts/event-driven/templates/roles.yaml deleted file mode 100644 index e55cefe..0000000 --- a/machine-learning-process/charts/event-driven/templates/roles.yaml +++ /dev/null @@ -1,52 +0,0 @@ ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: Role -metadata: - name: argo-extended-role -rules: - - apiGroups: - - "" - resources: - - pods - verbs: - - create - - patch - - delete - - list - - watch - - get - - patch - - apiGroups: - - "" - resources: - - pods/log - verbs: - - get - - list - - apiGroups: - - argoproj.io - resources: - - workflows - verbs: - - create - - get - - list - - watch - - update - - patch - - apiGroups: - - argoproj.io - resources: - - workflowtaskresults - verbs: - - create - - patch - - apiGroups: - - batch - resources: - - jobs - verbs: - - create - - get - - list - - watch \ No newline at end of file diff --git a/machine-learning-process/charts/event-driven/values.yaml b/machine-learning-process/charts/event-driven/values.yaml deleted file mode 100644 index f5789b1..0000000 --- a/machine-learning-process/charts/event-driven/values.yaml +++ /dev/null @@ -1,4 +0,0 @@ -jetstreamReplicas: 2 -jetstreamVolumeSize: 1Gi - -streamName: STREAM \ No newline at end of file diff --git a/machine-learning-process/containers/stage-out/Dockerfile b/machine-learning-process/containers/stage-out/Dockerfile deleted file mode 100644 index 14d3e6e..0000000 --- a/machine-learning-process/containers/stage-out/Dockerfile +++ /dev/null @@ -1,15 +0,0 @@ -FROM python:3.11 - -RUN apt update && apt install -y yq - -WORKDIR /code - -COPY ./requirements.txt /code/requirements.txt - -RUN pip install --no-cache-dir -r /code/requirements.txt - -COPY . /code - -RUN cd /code && python setup.py install - -RUN stage-out --help diff --git a/machine-learning-process/containers/stage-out/README.md b/machine-learning-process/containers/stage-out/README.md deleted file mode 100644 index a7a4564..0000000 --- a/machine-learning-process/containers/stage-out/README.md +++ /dev/null @@ -1,49 +0,0 @@ -# stage-out - - -## For developers - -Create a Python environment with, e.g. `mamba`: - -``` -mamba create -n env_stage_in python -mamba activate env_stage_in -``` - -``` -pip install -r requirements.txt -``` - -## Container - -Build the container with: - -``` -docker build -t stage-out . -``` - -Test the container with: - -``` -docker run --rm docker.io/library/stage-out stage-out --help -``` - -## CWL - -You can use a cwlrunner like `cwltool` to do a stage-in operation. - -Requirement: - -* a built container tagged `docker.io/library/stage-out:latest` - -``` -cwltool stage-out.cwl --bucket iride-sentinel-2 --stac_catalog /data/work/iride-marketplace/stage-in/_ka1p9cp --subfolder pippo --usersettings usersettings.json -``` - -## Run tests - -The unit tests can be run with: - -`nose2` - -TODO: add capture stdout in CWL \ No newline at end of file diff --git a/machine-learning-process/containers/stage-out/app/__init__.py b/machine-learning-process/containers/stage-out/app/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/machine-learning-process/containers/stage-out/app/main.py b/machine-learning-process/containers/stage-out/app/main.py deleted file mode 100644 index 40470e7..0000000 --- a/machine-learning-process/containers/stage-out/app/main.py +++ /dev/null @@ -1,94 +0,0 @@ -""" Main module for the application. """ -import os -import shutil - -import boto3 -import botocore -import click -import pystac -from botocore.client import Config -from loguru import logger -from pystac.stac_io import StacIO -import sys -from app.stac import CustomStacIO, upload_file_with_chunk_size -from app.usersettings import UserSettings - - -@click.command() -@click.option( - "--stac-catalog", help="Local path to a folder containing catalog.json STAC Catalog", required=True -) -@click.option("--user-settings", help="S3 user settings", required=True) -@click.option("--bucket", "bucket", help="S3 bucket", required=True) -@click.option("--subfolder", "subfolder", help="S3 subfolder", required=True) -def main(stac_catalog, user_settings, bucket, subfolder): - user_settings_config = UserSettings.from_file(user_settings) - - s3_settings = user_settings_config.get_s3_settings(f"s3://{bucket}/{subfolder}") - - if not s3_settings: - raise ValueError("No S3 settings found for this bucket") - - # set the environment variables for S3 from the user settings - os.environ["aws_access_key_id"] = s3_settings["aws_access_key_id"] - os.environ["aws_secret_access_key"] = s3_settings["aws_secret_access_key"] - os.environ["aws_region_name"] = s3_settings["region_name"] - os.environ["aws_endpoint_url"] = s3_settings["endpoint_url"] - - client = boto3.client( - "s3", - **s3_settings, - config=Config(s3={"addressing_style": "path", "signature_version": "s3v4"}), - ) - - shutil.copytree(stac_catalog, "/tmp/catalog") - cat = pystac.read_file(os.path.join("/tmp/catalog", "catalog.json")) - - StacIO.set_default(CustomStacIO) - - for item in cat.get_items(): - for key, asset in item.get_assets().items(): - s3_path = os.path.normpath( - os.path.join(os.path.join(subfolder, item.id, asset.href)) - ) - logger.info(f"upload {asset.href} to s3://{bucket}/{s3_path}") - - upload_file_with_chunk_size( - client, - asset.get_absolute_href(), - bucket, - s3_path - ) - - asset.href = f"s3://{bucket}/{s3_path}" - item.add_asset(key, asset) - - cat.normalize_hrefs(f"s3://{bucket}/{subfolder}") - - for item in cat.get_items(): - # upload item to S3 - logger.info(f"upload {item.id} to s3://{bucket}/{subfolder}") - for index, link in enumerate(item.links): - if link.rel in ["collection"]: - logger.info("saving collection.json") - collection = link.target - collection.links = [] - pystac.write_file(link.target, dest_href="./collection.json") - item.links.pop(index) - item.set_collection(None) - if link.rel in ["root"]: - item.links.pop(index) - pystac.write_file(item, item.get_self_href()) - - # upload catalog to S3 - logger.info(f"upload catalog.json to s3://{bucket}/{subfolder}") - for index, link in enumerate(cat.links): - if link.rel in ["root", "collection"]: - cat.links.pop(index) - pystac.write_file(cat, cat.get_self_href()) - - logger.info("Done!") - - print(f"s3://{bucket}/{subfolder}/catalog.json", file=sys.stdout) -if __name__ == "__main__": - main() diff --git a/machine-learning-process/containers/stage-out/app/stac.py b/machine-learning-process/containers/stage-out/app/stac.py deleted file mode 100644 index 66709fc..0000000 --- a/machine-learning-process/containers/stage-out/app/stac.py +++ /dev/null @@ -1,77 +0,0 @@ -import os -from urllib.parse import urlparse - -import boto3 -import botocore -from botocore.client import Config -from pystac.stac_io import DefaultStacIO - - -class CustomStacIO(DefaultStacIO): - """Custom STAC IO class that uses boto3 to read from S3.""" - - def __init__(self): - self.session = botocore.session.Session() - - def write_text(self, dest, txt, *args, **kwargs): - self.s3_client = self.session.create_client( - service_name="s3", - use_ssl=True, - aws_access_key_id=os.environ["aws_access_key_id"], - aws_secret_access_key=os.environ["aws_secret_access_key"], - region_name=os.environ["aws_region_name"], - endpoint_url=os.environ["aws_endpoint_url"], - config=Config(s3={"addressing_style": "path", "signature_version": "s3v4"}), - ) - - parsed = urlparse(dest) - if parsed.scheme == "s3": - self.s3_client.put_object( - Body=txt.encode("UTF-8"), - Bucket=parsed.netloc, - Key=parsed.path[1:], - ContentType="application/geo+json", - ) - else: - super().write_text(dest, txt, *args, **kwargs) - -import boto3 -from boto3.s3.transfer import TransferConfig -import os - -import boto3 -import os -import shutil -import logging -from boto3.s3.transfer import TransferConfig -from botocore.client import Config -from pystac import StacIO -import pystac - -# Assuming CustomStacIO is already defined somewhere -# from your_custom_module import CustomStacIO - -logger = logging.getLogger() -logger.setLevel(logging.INFO) - -def upload_file_with_chunk_size(client, file_path, bucket_name, s3_key, max_chunks=1000, default_chunk_size_mb=25): - # Get the size of the file - file_size = os.path.getsize(file_path) - - # Convert default chunk size from MB to bytes - default_chunk_size = default_chunk_size_mb * 1024 * 1024 - - # Calculate the optimal chunk size to ensure the number of chunks does not exceed max_chunks - optimal_chunk_size = min(default_chunk_size, file_size // max_chunks + 1) - - # Configure the transfer settings - config = TransferConfig(multipart_chunksize=optimal_chunk_size) - - # Upload the file - client.upload_file( - Filename=file_path, - Bucket=bucket_name, - Key=s3_key, - Config=config - ) - diff --git a/machine-learning-process/containers/stage-out/app/usersettings.py b/machine-learning-process/containers/stage-out/app/usersettings.py deleted file mode 100644 index 9f4e8ae..0000000 --- a/machine-learning-process/containers/stage-out/app/usersettings.py +++ /dev/null @@ -1,63 +0,0 @@ -""" -This code defines a UserSettings class that has methods for reading JSON files, -matching regular expressions, and setting environment variables for an S3 service. -The set_s3_environment method sets environment variables for an S3 service based -on a given URL. -""" - -import json -import os -import re -from typing import Dict - - -class UserSettings: - """class for reading JSON files, matching regular expressions, - and setting environment variables for an S3 service""" - - def __init__(self, settings: Dict): - self.settings = settings - - @classmethod - def from_file(cls, file_path): - """create a UserSettings object from a JSON file""" - return cls(cls.read_json_file(file_path)) - - @staticmethod - def read_json_file(file_path): - """read a JSON file and return the contents as a dictionary""" - with open(file_path, "r", encoding="utf-8") as stream: - return json.load(stream) - - @staticmethod - def match_regex(regex, string): - """match a regular expression to a string and return the match object""" - return re.search(regex, string) - - @staticmethod - def set_s3_vars(s3_service): - """set environment variables for an S3 service""" - os.environ["AWS_ACCESS_KEY_ID"] = s3_service["AccessKey"] - os.environ["AWS_SECRET_ACCESS_KEY"] = s3_service["SecretKey"] - os.environ["AWS_DEFAULT_REGION"] = s3_service["Region"] - os.environ["AWS_REGION"] = s3_service["Region"] - os.environ["AWS_S3_ENDPOINT"] = s3_service["ServiceURL"] - - def set_s3_environment(self, url): - """set environment variables for an S3 service based on a given URL""" - for _, s3_service in self.settings["S3"]["Services"].items(): - if self.match_regex(s3_service["UrlPattern"], url): - self.set_s3_vars(s3_service) - break - - def get_s3_settings(self, url): - """return S3 settings based on a given URL""" - for _, s3_service in self.settings["S3"]["Services"].items(): - if self.match_regex(s3_service["UrlPattern"], url): - return { - "region_name": s3_service["Region"], - "endpoint_url": s3_service["ServiceURL"], - "aws_access_key_id": s3_service["AccessKey"], - "aws_secret_access_key": s3_service["SecretKey"], - } - return None diff --git a/machine-learning-process/containers/stage-out/requirements.txt b/machine-learning-process/containers/stage-out/requirements.txt deleted file mode 100644 index 281aba6..0000000 --- a/machine-learning-process/containers/stage-out/requirements.txt +++ /dev/null @@ -1,5 +0,0 @@ -boto3 -pystac -loguru -click -pyyaml \ No newline at end of file diff --git a/machine-learning-process/containers/stage-out/setup.cfg b/machine-learning-process/containers/stage-out/setup.cfg deleted file mode 100644 index 03162a4..0000000 --- a/machine-learning-process/containers/stage-out/setup.cfg +++ /dev/null @@ -1,3 +0,0 @@ -[metadata] -name=stage-out -version=0.3.4 diff --git a/machine-learning-process/containers/stage-out/setup.py b/machine-learning-process/containers/stage-out/setup.py deleted file mode 100644 index 85fdb58..0000000 --- a/machine-learning-process/containers/stage-out/setup.py +++ /dev/null @@ -1,12 +0,0 @@ -from setuptools import setup - -console_scripts = ["stage-out=app.main:main"] - -setup( - name="stage-out", - version="0.1", - description="The funniest joke in the world", - packages=["app"], - entry_points={"console_scripts": console_scripts}, - zip_safe=False, -) diff --git a/machine-learning-process/containers/validate-schema/Dockerfile b/machine-learning-process/containers/validate-schema/Dockerfile deleted file mode 100644 index 5c9c7a4..0000000 --- a/machine-learning-process/containers/validate-schema/Dockerfile +++ /dev/null @@ -1,5 +0,0 @@ -FROM python:3.11-slim - -WORKDIR /app - -RUN pip install jsonschema \ No newline at end of file diff --git a/machine-learning-process/files/init.sh b/machine-learning-process/files/init.sh index f4e30c8..9d0842e 100644 --- a/machine-learning-process/files/init.sh +++ b/machine-learning-process/files/init.sh @@ -1,10 +1,13 @@ +#!/bin/bash + +set -x cd /workspace #temp -mkdir notebooks -git clone https://github.com/ai-extensions/notebooks.git -cd notebooks -git checkout develop +mkdir machine-learning-process +git clone https://github.com/parham-membari-terradue/machine-learning-process.git +cd machine-learning-process + code-server --install-extension ms-python.python code-server --install-extension ms-toolsai.jupyter @@ -17,8 +20,9 @@ python -m venv /workspace/.venv source /workspace/.venv/bin/activate /workspace/.venv/bin/python -m pip install --no-cache-dir hatch -export AWS_DEFAULT_REGION="us-east-1" +export AWS_DEFAULT_REGION="far-par" export AWS_ACCESS_KEY_ID="test" export AWS_SECRET_ACCESS_KEY="test" -aws s3 mb s3://results --endpoint-url=http://eoap-event-driven-localstack:4566 -aws s3 mb s3://workflows --endpoint-url=http://eoap-event-driven-localstack:4566 \ No newline at end of file +export AWS_BUCKET_NAME="ai-ext-bucket-dev" +export AWS_S3_ENDPOINT="https://s3.fr-par.scw.cloud" +export MLFLOW_TRACKING_URI="http://localhost:5000" \ No newline at end of file From 7a49f99b11c60cd344212f35b654b4a5982f59de Mon Sep 17 00:00:00 2001 From: Fabrice Brito Date: Tue, 11 Mar 2025 17:53:03 +0100 Subject: [PATCH 21/61] fixes in mastering skaffold --- mastering-app-package/skaffold.yaml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/mastering-app-package/skaffold.yaml b/mastering-app-package/skaffold.yaml index 7b24805..ceb82f6 100644 --- a/mastering-app-package/skaffold.yaml +++ b/mastering-app-package/skaffold.yaml @@ -13,13 +13,14 @@ deploy: setValues: coder.coderImage: eoepca/pde-code-server:1.0.0 coder.workspace: mastering-app-package - coderstorageClassName: standard + coder.storageClassName: standard coder.workspaceStorage: 10Gi coderResources.limits.cpu: '2' coderResources.limits.memory: '6442450944' coderResources.requests.cpu: '1' coderResources.requests.memory: '4294967296' calrissian.enabled: true + calrissian.imagePullSecrets: [] setFiles: { initScript: ./files/init.sh, bashrcScript: ./files/bash-rc, From 82738742b9e0b9ca96c7428de162c32ec0867345 Mon Sep 17 00:00:00 2001 From: Fabrice Brito Date: Tue, 11 Mar 2025 17:53:57 +0100 Subject: [PATCH 22/61] how-to using ttl.sh --- how-to/skaffold.yaml | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/how-to/skaffold.yaml b/how-to/skaffold.yaml index 1e61065..f7cd71a 100644 --- a/how-to/skaffold.yaml +++ b/how-to/skaffold.yaml @@ -4,8 +4,12 @@ metadata: name: eoap-how-to build: + tagPolicy: + envTemplate: + template: 2h + artifacts: - - image: appimage + - image: ttl.sh/eoap-how-to-coder deploy: helm: @@ -15,15 +19,18 @@ deploy: namespace: eoap-how-to createNamespace: true setValueTemplates: - coder.coderImage: "{{.IMAGE_REPO_appimage}}/{{.IMAGE_TAG_appimage}}@{{.IMAGE_DIGEST_appimage}}" + coder.coderImage: ttl.sh/eoap-how-to-coder:2h setValues: coder.workspace: how-to - coderstorageClassName: standard + coder.storageClassName: openebs-kernel-nfs-scw coder.workspaceStorage: 10Gi coderResources.limits.cpu: '2' coderResources.limits.memory: '6442450944' coderResources.requests.cpu: '1' coderResources.requests.memory: '4294967296' + calrissian.enabled: true + calrissian.storageClassName: openebs-kernel-nfs-scw + calrissian.imagePullSecrets: [] skaffold.enabled: True setFiles: { initScript: ./files/init.sh, From 90e26fea97234c0dc844c368adf0a2ad52e6f6e4 Mon Sep 17 00:00:00 2001 From: pmembari Date: Tue, 11 Mar 2025 18:10:34 +0100 Subject: [PATCH 23/61] chore: must handle error for mlflow write --- machine-learning-process/skaffold.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/machine-learning-process/skaffold.yaml b/machine-learning-process/skaffold.yaml index 0f31e6a..d6d0c28 100644 --- a/machine-learning-process/skaffold.yaml +++ b/machine-learning-process/skaffold.yaml @@ -37,7 +37,7 @@ deploy: } # MLflow - - name: my-mlflow + - name: my-mlflow # OSError: [Errno 30] Read-only file system: '/mlflow' remoteChart: community-charts/mlflow namespace: tile-based-training createNamespace: true From 00affb72004b2a8f7bc91929ebeab989c53b13f5 Mon Sep 17 00:00:00 2001 From: parham-membari-terradue Date: Wed, 12 Mar 2025 18:25:34 +0100 Subject: [PATCH 24/61] fix: mlflow artifact issue --- machine-learning-process/README.md | 25 ++++++++++++++++++------- machine-learning-process/files/init.sh | 4 ++-- machine-learning-process/skaffold.yaml | 7 +++---- 3 files changed, 23 insertions(+), 13 deletions(-) diff --git a/machine-learning-process/README.md b/machine-learning-process/README.md index 9ebffd7..64bffa6 100644 --- a/machine-learning-process/README.md +++ b/machine-learning-process/README.md @@ -7,18 +7,29 @@ kubectl create ns tile-based-training 2- install mlflow using helm: ``` helm repo add community-charts https://community-charts.github.io/helm-charts +helm repo add localstack https://helm.localstack.cloud helm repo update -helm install my-mlflow community-charts/mlflow --namespace tile-based-training ``` -3- edit [init.sh](./files/init.sh) with a correct repository: -ex: +3- Deploy everything using command below: ``` -git clone https://{your_git_account}:{your_access_token}@github.com/ai-extensions/notebooks.git +skaffold dev ``` - -4- Deploy everything using command below: +4- Open the code-server and mlflow on your browser: ``` -skaffold dev +code-server: http://localhost:8000 +mlflow: http://localhost:5000 +``` +5- Follow the documentation of both `training` and `inference`. You must build the docker image for each of them which are located under `training/make-ml-model` and `inference/make-inference`. +6- Run the application packages with calrissian(documentation is provided for each module). + + +### **Troubleshooting** + +If the **Code Server** and **Mlflow** are unreachable at `http://127.0.0.1:8000` and `http://127.0.0.1:5000` respectively, there may be an existing process occupying those port. You can terminate the process using the command below, allowing Kubernetes to automatically re-establish port forwarding for traffic on port `8000` and `5000`: + +```sh +sudo fuser -k 8000/tcp +sudo fuser -k 5000/tcp ``` \ No newline at end of file diff --git a/machine-learning-process/files/init.sh b/machine-learning-process/files/init.sh index 9d0842e..427c5d4 100644 --- a/machine-learning-process/files/init.sh +++ b/machine-learning-process/files/init.sh @@ -18,11 +18,11 @@ echo '{"workbench.colorTheme": "Visual Studio Dark"}' > /workspace/User/settings python -m venv /workspace/.venv source /workspace/.venv/bin/activate -/workspace/.venv/bin/python -m pip install --no-cache-dir hatch +/workspace/.venv/bin/python -m pip install --no-cache-dir hatch calrissian export AWS_DEFAULT_REGION="far-par" export AWS_ACCESS_KEY_ID="test" export AWS_SECRET_ACCESS_KEY="test" export AWS_BUCKET_NAME="ai-ext-bucket-dev" export AWS_S3_ENDPOINT="https://s3.fr-par.scw.cloud" -export MLFLOW_TRACKING_URI="http://localhost:5000" \ No newline at end of file +export MLFLOW_TRACKING_URI="http://my-mlflow:5000" \ No newline at end of file diff --git a/machine-learning-process/skaffold.yaml b/machine-learning-process/skaffold.yaml index d6d0c28..2f28fbe 100644 --- a/machine-learning-process/skaffold.yaml +++ b/machine-learning-process/skaffold.yaml @@ -19,11 +19,10 @@ deploy: namespace: tile-based-training createNamespace: true setValues: - service.type: LoadBalancer - storageClassRWX: "standard" + coder.serviceType: LoadBalancer + coder.storageClassName: standard coder.coderImage: eoepca/pde-code-server:1.0.0 coder.workspace: machine-learning-process - coderstorageClassName: standard coder.workspaceStorage: 10Gi coderResources.limits.cpu: '2' coderResources.limits.memory: '6442450944' @@ -57,7 +56,7 @@ deploy: mysql: enabled: false # Disable MySQL artifactRoot: - proxiedArtifactStorage: false # Disable proxied artifact storage access + proxiedArtifactStorage: true # Disable proxied artifact storage access azureBlob: enabled: false # Disable Azure Blob Storage s3: From 8163dfd0483b333269c5344abf3025a2d2fd1b6e Mon Sep 17 00:00:00 2001 From: parham-membari-terradue Date: Wed, 12 Mar 2025 18:26:28 +0100 Subject: [PATCH 25/61] feat: add dynamic code-server service type --- charts/coder/templates/coder.yaml | 3 ++- charts/coder/values.yaml | 5 ++++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/charts/coder/templates/coder.yaml b/charts/coder/templates/coder.yaml index fa7524f..37c61f9 100644 --- a/charts/coder/templates/coder.yaml +++ b/charts/coder/templates/coder.yaml @@ -146,6 +146,7 @@ spec: requests: storage: {{ .Values.coder.workspaceStorage }} storageClassName: {{ .Values.coder.storageClassName}} + --- apiVersion: v1 kind: Service @@ -158,7 +159,7 @@ spec: - name: web port: 8080 targetPort: 8080 - type: ClusterIP + type: {{.Values.coder.serviceType}} --- apiVersion: v1 kind: ConfigMap diff --git a/charts/coder/values.yaml b/charts/coder/values.yaml index 2683ce4..962eb0e 100644 --- a/charts/coder/values.yaml +++ b/charts/coder/values.yaml @@ -1,11 +1,14 @@ coder: coderImage: eoepca/pde-code-server:1.0.0 workspace: quickwin + serviceType: ClusterIP storageClassName: standard workspaceStorage: 20Gi securityContext: privileged: true + + calrissian: enabled: True storage: 10Gi @@ -27,4 +30,4 @@ skaffold: daskGateway: enabled: False daskGatewayUrl: "" - image: "" \ No newline at end of file + image: "" From d4d772802ac13341f08b70a9dfa34ee882c0405f Mon Sep 17 00:00:00 2001 From: pmembari Date: Thu, 13 Mar 2025 17:52:30 +0100 Subject: [PATCH 26/61] feat: add cluster conf --- .../charts/cluster-config/Chart.yaml | 6 ++++ .../charts/cluster-config/templates/role.yaml | 14 +++++++++ .../cluster-config/templates/rolebinding.yaml | 11 +++++++ .../cluster-config/templates/sercret.yaml | 7 +++++ .../templates/serviceaccount.yaml | 6 ++++ .../charts/cluster-config/values.yaml | 2 ++ machine-learning-process/files/init.sh | 30 ++++++++++++++----- machine-learning-process/skaffold.yaml | 6 +++- 8 files changed, 73 insertions(+), 9 deletions(-) create mode 100644 machine-learning-process/charts/cluster-config/Chart.yaml create mode 100644 machine-learning-process/charts/cluster-config/templates/role.yaml create mode 100644 machine-learning-process/charts/cluster-config/templates/rolebinding.yaml create mode 100644 machine-learning-process/charts/cluster-config/templates/sercret.yaml create mode 100644 machine-learning-process/charts/cluster-config/templates/serviceaccount.yaml create mode 100644 machine-learning-process/charts/cluster-config/values.yaml diff --git a/machine-learning-process/charts/cluster-config/Chart.yaml b/machine-learning-process/charts/cluster-config/Chart.yaml new file mode 100644 index 0000000..c2ff6f5 --- /dev/null +++ b/machine-learning-process/charts/cluster-config/Chart.yaml @@ -0,0 +1,6 @@ +apiVersion: v2 +name: cluster-config +description: simple cluster-config chart +type: application +version: 0.1.0 +appVersion: "1.0.0" diff --git a/machine-learning-process/charts/cluster-config/templates/role.yaml b/machine-learning-process/charts/cluster-config/templates/role.yaml new file mode 100644 index 0000000..cb64d4a --- /dev/null +++ b/machine-learning-process/charts/cluster-config/templates/role.yaml @@ -0,0 +1,14 @@ +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: kaniko-role +rules: + - apiGroups: [""] + resources: ["pods"] + verbs: ["create", "get", "list", "watch", "delete"] + - apiGroups: [""] + resources: ["pods/exec"] + verbs: ["create"] + - apiGroups: [""] + resources: ["pods/log"] + verbs: ["get","list"] \ No newline at end of file diff --git a/machine-learning-process/charts/cluster-config/templates/rolebinding.yaml b/machine-learning-process/charts/cluster-config/templates/rolebinding.yaml new file mode 100644 index 0000000..703c0dc --- /dev/null +++ b/machine-learning-process/charts/cluster-config/templates/rolebinding.yaml @@ -0,0 +1,11 @@ +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: kaniko-rolebinding +subjects: + - kind: ServiceAccount + name: kaniko-sa +roleRef: + kind: Role + name: kaniko-role + apiGroup: rbac.authorization.k8s.io \ No newline at end of file diff --git a/machine-learning-process/charts/cluster-config/templates/sercret.yaml b/machine-learning-process/charts/cluster-config/templates/sercret.yaml new file mode 100644 index 0000000..8adb2fd --- /dev/null +++ b/machine-learning-process/charts/cluster-config/templates/sercret.yaml @@ -0,0 +1,7 @@ +apiVersion: v1 +kind: Secret +metadata: + name: kaniko-secret +data: + .dockerconfigjson: {{.Values.secret.dockerconfigjson}} +type: kubernetes.io/dockerconfigjson \ No newline at end of file diff --git a/machine-learning-process/charts/cluster-config/templates/serviceaccount.yaml b/machine-learning-process/charts/cluster-config/templates/serviceaccount.yaml new file mode 100644 index 0000000..fb4ef8f --- /dev/null +++ b/machine-learning-process/charts/cluster-config/templates/serviceaccount.yaml @@ -0,0 +1,6 @@ +apiVersion: v1 +kind: ServiceAccount +metadata: + name: kaniko-sa +imagePullSecrets: + - name: kaniko-secret \ No newline at end of file diff --git a/machine-learning-process/charts/cluster-config/values.yaml b/machine-learning-process/charts/cluster-config/values.yaml new file mode 100644 index 0000000..006849e --- /dev/null +++ b/machine-learning-process/charts/cluster-config/values.yaml @@ -0,0 +1,2 @@ +secret: + dockerconfigjson: "" \ No newline at end of file diff --git a/machine-learning-process/files/init.sh b/machine-learning-process/files/init.sh index 427c5d4..bed559d 100644 --- a/machine-learning-process/files/init.sh +++ b/machine-learning-process/files/init.sh @@ -3,26 +3,40 @@ set -x cd /workspace -#temp -mkdir machine-learning-process -git clone https://github.com/parham-membari-terradue/machine-learning-process.git -cd machine-learning-process +# Install Skaffold +curl -Lo skaffold https://storage.googleapis.com/skaffold/releases/latest/skaffold-linux-amd64 && \ +sudo install skaffold /usr/local/bin/ +# Install yq (YAML processor) +wget https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -O ~/.local/bin/yq +chmod +x ~/.local/bin/yq +# Setup workspace +mkdir machine-learning-process-new +git clone https://github.com/parham-membari-terradue/machine-learning-process-new.git +cd machine-learning-process-new + +# Install VS Code Extensions code-server --install-extension ms-python.python code-server --install-extension ms-toolsai.jupyter ln -s /workspace/.local/share/code-server/extensions /workspace/extensions +# Setup user settings mkdir -p /workspace/User/ echo '{"workbench.colorTheme": "Visual Studio Dark"}' > /workspace/User/settings.json -python -m venv /workspace/.venv -source /workspace/.venv/bin/activate -/workspace/.venv/bin/python -m pip install --no-cache-dir hatch calrissian +# Setup Python virtual environment +python -m pip install --no-cache-dir hatch tomlq calrissian go-task-bin + +# Add ~/.local/bin to PATH for current session +echo 'export PATH=$PATH:$HOME/.local/bin' >> ~/.bashrc +source ~/.bashrc +# AWS environment variables export AWS_DEFAULT_REGION="far-par" export AWS_ACCESS_KEY_ID="test" export AWS_SECRET_ACCESS_KEY="test" export AWS_BUCKET_NAME="ai-ext-bucket-dev" export AWS_S3_ENDPOINT="https://s3.fr-par.scw.cloud" -export MLFLOW_TRACKING_URI="http://my-mlflow:5000" \ No newline at end of file +export MLFLOW_TRACKING_URI="http://my-mlflow:5000" +export TASK_X_REMOTE_TASKFILES="1" diff --git a/machine-learning-process/skaffold.yaml b/machine-learning-process/skaffold.yaml index 2f28fbe..29c723f 100644 --- a/machine-learning-process/skaffold.yaml +++ b/machine-learning-process/skaffold.yaml @@ -5,6 +5,10 @@ metadata: deploy: helm: releases: + - name: cluster-config + chartPath: ./charts/cluster-config + namespace: tile-based-training + createNamespace: true - name: tile-based-training-localstack remoteChart: localstack/localstack @@ -22,7 +26,7 @@ deploy: coder.serviceType: LoadBalancer coder.storageClassName: standard coder.coderImage: eoepca/pde-code-server:1.0.0 - coder.workspace: machine-learning-process + coder.workspace: machine-learning-process-new coder.workspaceStorage: 10Gi coderResources.limits.cpu: '2' coderResources.limits.memory: '6442450944' From 92c1100a71103adfa70f822438cc1e297ffc61b0 Mon Sep 17 00:00:00 2001 From: pmembari Date: Fri, 14 Mar 2025 12:17:55 +0100 Subject: [PATCH 27/61] feat: enable skaffold and update init --- machine-learning-process/files/bash-rc | 5 +++- machine-learning-process/files/init.sh | 36 ++++++++++++++++++-------- machine-learning-process/skaffold.yaml | 8 +++--- 3 files changed, 32 insertions(+), 17 deletions(-) diff --git a/machine-learning-process/files/bash-rc b/machine-learning-process/files/bash-rc index ec5a11e..5b4e6a5 100644 --- a/machine-learning-process/files/bash-rc +++ b/machine-learning-process/files/bash-rc @@ -1,5 +1,8 @@ alias ll="ls -l" alias aws="aws --endpoint-url=http://eoap-event-driven-localstack:4566" - +export PATH=/workspace/.venv/bin:$PATH +export TASK_X_REMOTE_TASKFILES=1 +export MLFLOW_TRACKING_URI="http://my-mlflow:5000" +export PATH=$PATH:/workspace/.venv/bin:/workspace/.venv/bin/yq source /workspace/.venv/bin/activate \ No newline at end of file diff --git a/machine-learning-process/files/init.sh b/machine-learning-process/files/init.sh index bed559d..8313e3a 100644 --- a/machine-learning-process/files/init.sh +++ b/machine-learning-process/files/init.sh @@ -3,13 +3,6 @@ set -x cd /workspace -# Install Skaffold -curl -Lo skaffold https://storage.googleapis.com/skaffold/releases/latest/skaffold-linux-amd64 && \ -sudo install skaffold /usr/local/bin/ - -# Install yq (YAML processor) -wget https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -O ~/.local/bin/yq -chmod +x ~/.local/bin/yq # Setup workspace mkdir machine-learning-process-new @@ -18,6 +11,8 @@ cd machine-learning-process-new # Install VS Code Extensions code-server --install-extension ms-python.python +code-server --install-extension redhat.vscode-yaml +code-server --install-extension sbg-rabix.benten-cwl code-server --install-extension ms-toolsai.jupyter ln -s /workspace/.local/share/code-server/extensions /workspace/extensions @@ -26,11 +21,29 @@ mkdir -p /workspace/User/ echo '{"workbench.colorTheme": "Visual Studio Dark"}' > /workspace/User/settings.json # Setup Python virtual environment -python -m pip install --no-cache-dir hatch tomlq calrissian go-task-bin +python -m venv /workspace/.venv +source /workspace/.venv/bin/activate +/workspace/.venv/bin/python -m pip install --no-cache-dir tomlq calrissian + + +echo "**** install kubectl ****" +curl -s -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" +chmod +x kubectl +mkdir -p /workspace/.venv/bin +mv ./kubectl /workspace/.venv/bin/kubectl + + +curl -s -L https://github.com/pypa/hatch/releases/latest/download/hatch-x86_64-unknown-linux-gnu.tar.gz | tar -xzvf - -C /workspace/.venv/bin/ +chmod +x /workspace/.venv/bin/hatch +curl -s -L https://github.com/go-task/task/releases/download/v3.41.0/task_linux_amd64.tar.gz | tar -xzvf - -C /workspace/.venv/bin/ +chmod +x /workspace/.venv/bin/task + +curl -s -L https://storage.googleapis.com/skaffold/releases/latest/skaffold-linux-amd64 > /workspace/.venv/bin/skaffold +chmod +x /workspace/.venv/bin/skaffold -# Add ~/.local/bin to PATH for current session -echo 'export PATH=$PATH:$HOME/.local/bin' >> ~/.bashrc -source ~/.bashrc +curl -s -LO https://github.com/mikefarah/yq/releases/download/v4.45.1/yq_linux_amd64.tar.gz +tar -xvf yq_linux_amd64.tar.gz +mv yq_linux_amd64 /workspace/.venv/bin/yq # AWS environment variables export AWS_DEFAULT_REGION="far-par" @@ -40,3 +53,4 @@ export AWS_BUCKET_NAME="ai-ext-bucket-dev" export AWS_S3_ENDPOINT="https://s3.fr-par.scw.cloud" export MLFLOW_TRACKING_URI="http://my-mlflow:5000" export TASK_X_REMOTE_TASKFILES="1" +export PATH=$PATH:/workspace/.venv/bin:/workspace/.venv/bin/yq diff --git a/machine-learning-process/skaffold.yaml b/machine-learning-process/skaffold.yaml index 29c723f..86cc5bc 100644 --- a/machine-learning-process/skaffold.yaml +++ b/machine-learning-process/skaffold.yaml @@ -5,11 +5,7 @@ metadata: deploy: helm: releases: - - name: cluster-config - chartPath: ./charts/cluster-config - namespace: tile-based-training - createNamespace: true - + - name: tile-based-training-localstack remoteChart: localstack/localstack namespace: tile-based-training @@ -23,6 +19,7 @@ deploy: namespace: tile-based-training createNamespace: true setValues: + skaffold.enabled: true coder.serviceType: LoadBalancer coder.storageClassName: standard coder.coderImage: eoepca/pde-code-server:1.0.0 @@ -33,6 +30,7 @@ deploy: coderResources.requests.cpu: '1' coderResources.requests.memory: '4294967296' calrissian.enabled: true + coder.securityContext.privileged: false setFiles: { initScript: ./files/init.sh, bashrcScript: ./files/bash-rc, From a0fc2e548c00697e7c96f43a460201006bf86d92 Mon Sep 17 00:00:00 2001 From: pmembari Date: Fri, 14 Mar 2025 12:18:31 +0100 Subject: [PATCH 28/61] feat: add kaniko secret --- charts/coder/templates/coder.yaml | 18 +----------------- charts/coder/templates/kinko-secret.yaml | 7 +++++++ charts/coder/templates/role-binding.yaml | 18 ++++++++++++++++-- charts/coder/templates/roles.yaml | 18 ++++++++++++++++-- charts/coder/templates/service-account.yaml | 8 ++++++++ charts/coder/values.yaml | 9 ++++++++- 6 files changed, 56 insertions(+), 22 deletions(-) create mode 100644 charts/coder/templates/kinko-secret.yaml diff --git a/charts/coder/templates/coder.yaml b/charts/coder/templates/coder.yaml index 37c61f9..e6084ab 100644 --- a/charts/coder/templates/coder.yaml +++ b/charts/coder/templates/coder.yaml @@ -66,10 +66,6 @@ spec: mountPath: /etc/calrissian/pod-node-selector.yaml subPath: pod-node-selector.yaml {{ end }} - {{- if .Values.daskGateway.enabled }} - - name: dask-gateway-config - mountPath: /etc/dask - {{- end }} env: - name: XDG_CONFIG_HOME value: /workspace/.local @@ -90,13 +86,7 @@ spec: valueFrom: fieldRef: fieldPath: metadata.name - {{- end }} - {{- if .Values.daskGateway.enabled }} - - name: DASK_GATEWAY_ADDRESS - value: "{{ .Values.daskGateway.daskGatewayUrl }}" - - name: DASK_IMAGE - value: "{{ .Values.daskGateway.image }}" - {{- end }} + {{ end }} resources: limits: cpu: {{ .Values.coderResources.limits.cpu }} @@ -129,11 +119,6 @@ spec: name: node-selectors defaultMode: 420 {{ end }} - {{- if .Values.daskGateway.enabled }} - - name: dask-gateway-config - configMap: - name: dask-gateway-config - {{- end }} --- apiVersion: v1 kind: PersistentVolumeClaim @@ -146,7 +131,6 @@ spec: requests: storage: {{ .Values.coder.workspaceStorage }} storageClassName: {{ .Values.coder.storageClassName}} - --- apiVersion: v1 kind: Service diff --git a/charts/coder/templates/kinko-secret.yaml b/charts/coder/templates/kinko-secret.yaml new file mode 100644 index 0000000..9bbd223 --- /dev/null +++ b/charts/coder/templates/kinko-secret.yaml @@ -0,0 +1,7 @@ +apiVersion: v1 +kind: Secret +metadata: + name: kaniko-secret +data: + .dockerconfigjson: ewogI...p9Cg== +type: kubernetes.io/dockerconfigjson \ No newline at end of file diff --git a/charts/coder/templates/role-binding.yaml b/charts/coder/templates/role-binding.yaml index e7192a7..eba5b8d 100644 --- a/charts/coder/templates/role-binding.yaml +++ b/charts/coder/templates/role-binding.yaml @@ -1,4 +1,4 @@ -{{ if .Values.calrissian.enabled }} +{{- if .Values.calrissian.enabled }} apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: @@ -34,4 +34,18 @@ roleRef: subjects: - kind: ServiceAccount name: calrissian-sa -{{ end }} \ No newline at end of file +{{- end }} +--- +{{- if .Values.skaffold.enabled }} +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: pod-exec-binding +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: pod-exec-role +subjects: +- kind: ServiceAccount + name: calrissian-sa +{{- end }} \ No newline at end of file diff --git a/charts/coder/templates/roles.yaml b/charts/coder/templates/roles.yaml index 06e8e26..2b93aa0 100644 --- a/charts/coder/templates/roles.yaml +++ b/charts/coder/templates/roles.yaml @@ -1,4 +1,4 @@ -{{ if .Values.calrissian.enabled }} +{{- if .Values.calrissian.enabled }} apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: @@ -25,4 +25,18 @@ rules: - apiGroups: ["batch"] resources: ["jobs"] verbs: ["create","patch","delete","list","watch","get"] -{{ end }} \ No newline at end of file +{{- end }} +{{- if .Values.skaffold.enabled }} +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: pod-exec-role +rules: +- verbs: + - '*' + apiGroups: + - '' + resources: + - pods/exec +{{- end }} \ No newline at end of file diff --git a/charts/coder/templates/service-account.yaml b/charts/coder/templates/service-account.yaml index 3b46632..c48ec2d 100644 --- a/charts/coder/templates/service-account.yaml +++ b/charts/coder/templates/service-account.yaml @@ -3,4 +3,12 @@ apiVersion: v1 kind: ServiceAccount metadata: name: calrissian-sa +{{- if .Values.calrissian.imagePullSecrets }} +imagePullSecrets: +{{- range .Values.calrissian.imagePullSecrets }} +- name: {{ . }} +{{- end }} +{{- end }} + {{ end }} + diff --git a/charts/coder/values.yaml b/charts/coder/values.yaml index 962eb0e..29eab7d 100644 --- a/charts/coder/values.yaml +++ b/charts/coder/values.yaml @@ -15,7 +15,7 @@ calrissian: storageClassName: standard imagePullSecrets: - kaniko-secret - + coderResources: limits: cpu: '2' @@ -31,3 +31,10 @@ daskGateway: enabled: False daskGatewayUrl: "" image: "" + + + + + + + From ff24f7a27ee44b21fa99f161f46ae1dbcfb90bde Mon Sep 17 00:00:00 2001 From: pmembari Date: Fri, 14 Mar 2025 15:50:05 +0100 Subject: [PATCH 29/61] add kaniko --- .pre-commit-config.yaml | 7 ++- charts/coder/templates/coder.yaml | 19 +++++++- charts/coder/templates/kinko-rabc.yaml | 51 +++++++++++++++++++++ charts/coder/templates/kinko-secret.yaml | 7 --- charts/coder/templates/role-binding.yaml | 14 ++++++ charts/coder/templates/roles.yaml | 15 ++++++ charts/coder/templates/service-account.yaml | 3 +- charts/coder/values.yaml | 15 ++---- 8 files changed, 107 insertions(+), 24 deletions(-) create mode 100644 charts/coder/templates/kinko-rabc.yaml delete mode 100644 charts/coder/templates/kinko-secret.yaml diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 39749b2..e2102e8 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -6,12 +6,11 @@ repos: - id: check-merge-conflict - id: debug-statements - id: detect-private-key - - id: end-of-file-fixer - id: no-commit-to-branch args: - --pattern=master|main - id: requirements-txt-fixer - - id: trailing-whitespace + - repo: https://github.com/myint/autoflake rev: v1.4 hooks: @@ -25,8 +24,8 @@ repos: hooks: - id: flake8 args: - - --max-line-length=88 - - --max-doc-length=90 + - --max-line-length=188 + - --max-doc-length=190 - repo: https://github.com/psf/black rev: 22.3.0 diff --git a/charts/coder/templates/coder.yaml b/charts/coder/templates/coder.yaml index e6084ab..69a7dae 100644 --- a/charts/coder/templates/coder.yaml +++ b/charts/coder/templates/coder.yaml @@ -66,6 +66,10 @@ spec: mountPath: /etc/calrissian/pod-node-selector.yaml subPath: pod-node-selector.yaml {{ end }} + {{- if .Values.daskGateway.enabled }} + - name: dask-gateway-config + mountPath: /etc/dask + {{- end }} env: - name: XDG_CONFIG_HOME value: /workspace/.local @@ -86,7 +90,15 @@ spec: valueFrom: fieldRef: fieldPath: metadata.name - {{ end }} + {{- end }} + {{- if .Values.daskGateway.enabled }} + - name: DASK_GATEWAY_ADDRESS + value: "{{ .Values.daskGateway.daskGatewayUrl }}" + - name: DASK_IMAGE + value: "{{ .Values.daskGateway.image }}" + {{- end }} + {{ if .Values.kaniko.enabled }} + {{- end }} resources: limits: cpu: {{ .Values.coderResources.limits.cpu }} @@ -119,6 +131,11 @@ spec: name: node-selectors defaultMode: 420 {{ end }} + {{- if .Values.daskGateway.enabled }} + - name: dask-gateway-config + configMap: + name: dask-gateway-config + {{- end }} --- apiVersion: v1 kind: PersistentVolumeClaim diff --git a/charts/coder/templates/kinko-rabc.yaml b/charts/coder/templates/kinko-rabc.yaml new file mode 100644 index 0000000..38ee269 --- /dev/null +++ b/charts/coder/templates/kinko-rabc.yaml @@ -0,0 +1,51 @@ +{{- if .Values.kaniko.enabled -}} +apiVersion: v1 +kind: Secret +metadata: + name: kaniko-secret +type: kubernetes.io/dockerconfigjson +data: + .dockerconfigjson: ew0KICAgICJhdXRocyI6IHsNCiAgICAgICAgImZha2UucmVnaXN0cnkuaW8iOiB7DQogICAgICAgICAgICAidXNlcm5hbWUiOiAiZmFrZXVzZXIiLA0KICAgICAgICAgICAgInBhc3N3b3JkIjogImZha2VwYXNzd29yZCIsDQogICAgICAgICAgICAiZW1haWwiOiAiZmFrZUBleGFtcGxlLmNvbSIsDQogICAgICAgICAgICAiYXV0aCI6ICIiDQogICAgICAgIH0NCiAgICB9DQp9 +--- +apiVersion: v1 +kind: Secret +metadata: + name: ttl-secret +type: kubernetes.io/dockerconfigjson +data: + .dockerconfigjson: ew0KICAgICJhdXRocyI6IHsNCiAgICAgICAgImZha2UucmVnaXN0cnkuaW8iOiB7DQogICAgICAgICAgICAidXNlcm5hbWUiOiAiZmFrZXVzZXIiLA0KICAgICAgICAgICAgInBhc3N3b3JkIjogImZha2VwYXNzd29yZCIsDQogICAgICAgICAgICAiZW1haWwiOiAiZmFrZUBleGFtcGxlLmNvbSIsDQogICAgICAgICAgICAiYXV0aCI6ICIiDQogICAgICAgIH0NCiAgICB9DQp9 +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: kaniko-sa +imagePullSecrets: + - name: kaniko-secret +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: kaniko-role +rules: + - apiGroups: [""] + resources: ["pods"] + verbs: ["create", "get", "list", "watch", "delete"] + - apiGroups: [""] + resources: ["pods/exec"] + verbs: ["create"] + - apiGroups: [""] + resources: ["pods/log"] + verbs: ["get","list"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: kaniko-rolebinding +subjects: + - kind: ServiceAccount + name: kaniko-sa +roleRef: + kind: Role + name: kaniko-role + apiGroup: rbac.authorization.k8s.io +{{- end -}} \ No newline at end of file diff --git a/charts/coder/templates/kinko-secret.yaml b/charts/coder/templates/kinko-secret.yaml deleted file mode 100644 index 9bbd223..0000000 --- a/charts/coder/templates/kinko-secret.yaml +++ /dev/null @@ -1,7 +0,0 @@ -apiVersion: v1 -kind: Secret -metadata: - name: kaniko-secret -data: - .dockerconfigjson: ewogI...p9Cg== -type: kubernetes.io/dockerconfigjson \ No newline at end of file diff --git a/charts/coder/templates/role-binding.yaml b/charts/coder/templates/role-binding.yaml index eba5b8d..9b0c330 100644 --- a/charts/coder/templates/role-binding.yaml +++ b/charts/coder/templates/role-binding.yaml @@ -46,6 +46,20 @@ roleRef: kind: Role name: pod-exec-role subjects: +- kind: ServiceAccount + name: calrissian-sa +{{- end }} +{{- if .Values.daskGateway.enabled }} +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: dask-configmap-binding +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: dask-configmap-role +subjects: - kind: ServiceAccount name: calrissian-sa {{- end }} \ No newline at end of file diff --git a/charts/coder/templates/roles.yaml b/charts/coder/templates/roles.yaml index 2b93aa0..5db4adb 100644 --- a/charts/coder/templates/roles.yaml +++ b/charts/coder/templates/roles.yaml @@ -39,4 +39,19 @@ rules: - '' resources: - pods/exec +{{- end }} +{{- if .Values.daskGateway.enabled }} +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: dask-configmap-role +rules: +- verbs: + - create + - delete + apiGroups: + - '' + resources: + - configmaps {{- end }} \ No newline at end of file diff --git a/charts/coder/templates/service-account.yaml b/charts/coder/templates/service-account.yaml index c48ec2d..8fadc1e 100644 --- a/charts/coder/templates/service-account.yaml +++ b/charts/coder/templates/service-account.yaml @@ -10,5 +10,4 @@ imagePullSecrets: {{- end }} {{- end }} -{{ end }} - +{{ end }} \ No newline at end of file diff --git a/charts/coder/values.yaml b/charts/coder/values.yaml index 29eab7d..11338c3 100644 --- a/charts/coder/values.yaml +++ b/charts/coder/values.yaml @@ -1,13 +1,12 @@ + coder: coderImage: eoepca/pde-code-server:1.0.0 workspace: quickwin - serviceType: ClusterIP storageClassName: standard workspaceStorage: 20Gi securityContext: privileged: true - - + serviceType: ClusterIP calrissian: enabled: True @@ -15,7 +14,7 @@ calrissian: storageClassName: standard imagePullSecrets: - kaniko-secret - + coderResources: limits: cpu: '2' @@ -32,9 +31,5 @@ daskGateway: daskGatewayUrl: "" image: "" - - - - - - +kaniko: + enabled: False \ No newline at end of file From 97c5c60448cac8407978685236383f4dbe982bef Mon Sep 17 00:00:00 2001 From: pmembari Date: Fri, 14 Mar 2025 17:37:20 +0100 Subject: [PATCH 30/61] feat: edit securityContext.privileged --- charts/coder/templates/coder.yaml | 12 ++++++------ charts/coder/templates/kinko-rabc.yaml | 2 +- charts/coder/templates/role-binding.yaml | 2 +- charts/coder/templates/roles.yaml | 2 +- charts/coder/templates/service-account.yaml | 2 +- charts/coder/values.yaml | 6 +++--- machine-learning-process/files/init.sh | 2 +- machine-learning-process/skaffold.yaml | 11 ++++++----- 8 files changed, 20 insertions(+), 19 deletions(-) diff --git a/charts/coder/templates/coder.yaml b/charts/coder/templates/coder.yaml index 69a7dae..55ca6c1 100644 --- a/charts/coder/templates/coder.yaml +++ b/charts/coder/templates/coder.yaml @@ -63,8 +63,8 @@ spec: - mountPath: /calrissian name: calrissian-volume - name: node-selectors - mountPath: /etc/calrissian/pod-node-selector.yaml - subPath: pod-node-selector.yaml + mountPath: /etc/calrissian/pod-node-selector.yaml + subPath: pod-node-selector.yaml {{ end }} {{- if .Values.daskGateway.enabled }} - name: dask-gateway-config @@ -168,7 +168,7 @@ metadata: name: init data: init: | - {{ required "A valid .Values.initScript entry required!" ( tpl ( .Values.initScript | default ( .Files.Get "files/init.sh")) . | nindent 10 ) }} + {{ required "A valid .Values.initScript entry required!" ( tpl ( .Values.initScript | default ( .Files.Get "files/init.sh")) . | nindent 10 ) }} --- apiVersion: v1 kind: ConfigMap @@ -176,7 +176,7 @@ metadata: name: bash-rc data: bash-rc: | - {{ required "A valid .Values.bashrcScript entry required!" ( tpl ( .Values.bashrcScript | default ( .Files.Get "files/bash-rc")) . | nindent 10 ) }} + {{ required "A valid .Values.bashrcScript entry required!" ( tpl ( .Values.bashrcScript | default ( .Files.Get "files/bash-rc")) . | nindent 10 ) }} --- apiVersion: v1 kind: ConfigMap @@ -184,7 +184,7 @@ metadata: name: bash-login data: bash-login: | - {{ required "A valid .Values.bashloginScript entry required!" ( tpl ( .Values.bashloginScript | default ( .Files.Get "files/bash-login")) . | nindent 10 ) }} + {{ required "A valid .Values.bashloginScript entry required!" ( tpl ( .Values.bashloginScript | default ( .Files.Get "files/bash-login")) . | nindent 10 ) }} --- {{ if .Values.calrissian.enabled }} apiVersion: v1 @@ -194,4 +194,4 @@ metadata: data: pod-node-selector.yaml: | {} -{{ end }} \ No newline at end of file +{{ end }} diff --git a/charts/coder/templates/kinko-rabc.yaml b/charts/coder/templates/kinko-rabc.yaml index 38ee269..0264eb4 100644 --- a/charts/coder/templates/kinko-rabc.yaml +++ b/charts/coder/templates/kinko-rabc.yaml @@ -48,4 +48,4 @@ roleRef: kind: Role name: kaniko-role apiGroup: rbac.authorization.k8s.io -{{- end -}} \ No newline at end of file +{{- end -}} diff --git a/charts/coder/templates/role-binding.yaml b/charts/coder/templates/role-binding.yaml index 9b0c330..df02c12 100644 --- a/charts/coder/templates/role-binding.yaml +++ b/charts/coder/templates/role-binding.yaml @@ -62,4 +62,4 @@ roleRef: subjects: - kind: ServiceAccount name: calrissian-sa -{{- end }} \ No newline at end of file +{{- end }} diff --git a/charts/coder/templates/roles.yaml b/charts/coder/templates/roles.yaml index 5db4adb..2039c21 100644 --- a/charts/coder/templates/roles.yaml +++ b/charts/coder/templates/roles.yaml @@ -54,4 +54,4 @@ rules: - '' resources: - configmaps -{{- end }} \ No newline at end of file +{{- end }} diff --git a/charts/coder/templates/service-account.yaml b/charts/coder/templates/service-account.yaml index 8fadc1e..3096892 100644 --- a/charts/coder/templates/service-account.yaml +++ b/charts/coder/templates/service-account.yaml @@ -10,4 +10,4 @@ imagePullSecrets: {{- end }} {{- end }} -{{ end }} \ No newline at end of file +{{ end }} diff --git a/charts/coder/values.yaml b/charts/coder/values.yaml index 11338c3..875339e 100644 --- a/charts/coder/values.yaml +++ b/charts/coder/values.yaml @@ -10,7 +10,7 @@ coder: calrissian: enabled: True - storage: 10Gi + storage: 20Gi storageClassName: standard imagePullSecrets: - kaniko-secret @@ -26,10 +26,10 @@ coderResources: skaffold: enabled: False -daskGateway: +daskGateway: enabled: False daskGatewayUrl: "" image: "" kaniko: - enabled: False \ No newline at end of file + enabled: False diff --git a/machine-learning-process/files/init.sh b/machine-learning-process/files/init.sh index 8313e3a..0fa9bbc 100644 --- a/machine-learning-process/files/init.sh +++ b/machine-learning-process/files/init.sh @@ -53,4 +53,4 @@ export AWS_BUCKET_NAME="ai-ext-bucket-dev" export AWS_S3_ENDPOINT="https://s3.fr-par.scw.cloud" export MLFLOW_TRACKING_URI="http://my-mlflow:5000" export TASK_X_REMOTE_TASKFILES="1" -export PATH=$PATH:/workspace/.venv/bin:/workspace/.venv/bin/yq +export PATH=$PATH:/workspace/.venv/bin diff --git a/machine-learning-process/skaffold.yaml b/machine-learning-process/skaffold.yaml index 86cc5bc..86e8975 100644 --- a/machine-learning-process/skaffold.yaml +++ b/machine-learning-process/skaffold.yaml @@ -18,19 +18,20 @@ deploy: chartPath: ../charts/coder namespace: tile-based-training createNamespace: true - setValues: - skaffold.enabled: true + setValues: coder.serviceType: LoadBalancer coder.storageClassName: standard coder.coderImage: eoepca/pde-code-server:1.0.0 coder.workspace: machine-learning-process-new - coder.workspaceStorage: 10Gi + coder.workspaceStorage: 20Gi coderResources.limits.cpu: '2' coderResources.limits.memory: '6442450944' coderResources.requests.cpu: '1' coderResources.requests.memory: '4294967296' calrissian.enabled: true - coder.securityContext.privileged: false + coder.securityContext.privileged: true + kaniko.enabled: true + skaffold.enabled: true setFiles: { initScript: ./files/init.sh, bashrcScript: ./files/bash-rc, @@ -38,7 +39,7 @@ deploy: } # MLflow - - name: my-mlflow # OSError: [Errno 30] Read-only file system: '/mlflow' + - name: my-mlflow remoteChart: community-charts/mlflow namespace: tile-based-training createNamespace: true From ef7f819c2effe38e95ac36de6d2da812d5ac5690 Mon Sep 17 00:00:00 2001 From: pmembari Date: Mon, 24 Mar 2025 14:56:23 +0100 Subject: [PATCH 31/61] feat: update git repo --- machine-learning-process/README.md | 2 +- machine-learning-process/files/init.sh | 8 ++++---- machine-learning-process/skaffold.yaml | 8 ++++---- 3 files changed, 9 insertions(+), 9 deletions(-) diff --git a/machine-learning-process/README.md b/machine-learning-process/README.md index 64bffa6..07db6c2 100644 --- a/machine-learning-process/README.md +++ b/machine-learning-process/README.md @@ -21,7 +21,7 @@ skaffold dev code-server: http://localhost:8000 mlflow: http://localhost:5000 ``` -5- Follow the documentation of both `training` and `inference`. You must build the docker image for each of them which are located under `training/make-ml-model` and `inference/make-inference`. +5- Follow the documentations of both `training` and `inference`. 6- Run the application packages with calrissian(documentation is provided for each module). diff --git a/machine-learning-process/files/init.sh b/machine-learning-process/files/init.sh index 0fa9bbc..4d115d8 100644 --- a/machine-learning-process/files/init.sh +++ b/machine-learning-process/files/init.sh @@ -5,9 +5,9 @@ cd /workspace # Setup workspace -mkdir machine-learning-process-new -git clone https://github.com/parham-membari-terradue/machine-learning-process-new.git -cd machine-learning-process-new +mkdir machine-learning-process +git clone https://github.com/parham-membari-terradue/machine-learning-process.git +cd machine-learning-process # Install VS Code Extensions code-server --install-extension ms-python.python @@ -44,7 +44,7 @@ chmod +x /workspace/.venv/bin/skaffold curl -s -LO https://github.com/mikefarah/yq/releases/download/v4.45.1/yq_linux_amd64.tar.gz tar -xvf yq_linux_amd64.tar.gz mv yq_linux_amd64 /workspace/.venv/bin/yq - +chmod +x /workspace/.venv/bin/yq # AWS environment variables export AWS_DEFAULT_REGION="far-par" export AWS_ACCESS_KEY_ID="test" diff --git a/machine-learning-process/skaffold.yaml b/machine-learning-process/skaffold.yaml index 86e8975..2c24768 100644 --- a/machine-learning-process/skaffold.yaml +++ b/machine-learning-process/skaffold.yaml @@ -11,7 +11,7 @@ deploy: namespace: tile-based-training createNamespace: true setValues: - service.type: LoadBalancer + service.type: NodePort # Code-Server with Calrissian - name: code-server @@ -19,11 +19,11 @@ deploy: namespace: tile-based-training createNamespace: true setValues: - coder.serviceType: LoadBalancer + coder.serviceType: NodePort coder.storageClassName: standard coder.coderImage: eoepca/pde-code-server:1.0.0 - coder.workspace: machine-learning-process-new - coder.workspaceStorage: 20Gi + coder.workspace: machine-learning-process + coder.workspaceStorage: 30Gi coderResources.limits.cpu: '2' coderResources.limits.memory: '6442450944' coderResources.requests.cpu: '1' From ba8cc3e26d58a507e32ce0ea527c2402b7976c9b Mon Sep 17 00:00:00 2001 From: parham-membari-terradue Date: Mon, 7 Apr 2025 00:31:42 +0200 Subject: [PATCH 32/61] chore: edit git repo --- machine-learning-process/files/init.sh | 6 +++--- machine-learning-process/skaffold.yaml | 8 ++++---- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/machine-learning-process/files/init.sh b/machine-learning-process/files/init.sh index 0fa9bbc..9e326c5 100644 --- a/machine-learning-process/files/init.sh +++ b/machine-learning-process/files/init.sh @@ -5,9 +5,9 @@ cd /workspace # Setup workspace -mkdir machine-learning-process-new -git clone https://github.com/parham-membari-terradue/machine-learning-process-new.git -cd machine-learning-process-new +mkdir machine-learning-process +git clone https://github.com/parham-membari-terradue/machine-learning-process.git +cd machine-learning-process # Install VS Code Extensions code-server --install-extension ms-python.python diff --git a/machine-learning-process/skaffold.yaml b/machine-learning-process/skaffold.yaml index 86e8975..2c24768 100644 --- a/machine-learning-process/skaffold.yaml +++ b/machine-learning-process/skaffold.yaml @@ -11,7 +11,7 @@ deploy: namespace: tile-based-training createNamespace: true setValues: - service.type: LoadBalancer + service.type: NodePort # Code-Server with Calrissian - name: code-server @@ -19,11 +19,11 @@ deploy: namespace: tile-based-training createNamespace: true setValues: - coder.serviceType: LoadBalancer + coder.serviceType: NodePort coder.storageClassName: standard coder.coderImage: eoepca/pde-code-server:1.0.0 - coder.workspace: machine-learning-process-new - coder.workspaceStorage: 20Gi + coder.workspace: machine-learning-process + coder.workspaceStorage: 30Gi coderResources.limits.cpu: '2' coderResources.limits.memory: '6442450944' coderResources.requests.cpu: '1' From df7c3dcfad063afe1a4cf5151c7c14788c985425 Mon Sep 17 00:00:00 2001 From: parham-membari-terradue Date: Mon, 7 Apr 2025 00:53:57 +0200 Subject: [PATCH 33/61] feat: edit yq installation --- machine-learning-process/files/init.sh | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/machine-learning-process/files/init.sh b/machine-learning-process/files/init.sh index 4d115d8..d631c41 100644 --- a/machine-learning-process/files/init.sh +++ b/machine-learning-process/files/init.sh @@ -23,7 +23,7 @@ echo '{"workbench.colorTheme": "Visual Studio Dark"}' > /workspace/User/settings # Setup Python virtual environment python -m venv /workspace/.venv source /workspace/.venv/bin/activate -/workspace/.venv/bin/python -m pip install --no-cache-dir tomlq calrissian +/workspace/.venv/bin/python -m pip install --no-cache-dir tomlq calrissian yq echo "**** install kubectl ****" @@ -41,9 +41,9 @@ chmod +x /workspace/.venv/bin/task curl -s -L https://storage.googleapis.com/skaffold/releases/latest/skaffold-linux-amd64 > /workspace/.venv/bin/skaffold chmod +x /workspace/.venv/bin/skaffold -curl -s -LO https://github.com/mikefarah/yq/releases/download/v4.45.1/yq_linux_amd64.tar.gz -tar -xvf yq_linux_amd64.tar.gz -mv yq_linux_amd64 /workspace/.venv/bin/yq +# curl -s -LO https://github.com/mikefarah/yq/releases/download/v4.45.1/yq_linux_amd64.tar.gz +# tar -xvf yq_linux_amd64.tar.gz +# mv yq_linux_amd64 /workspace/.venv/bin/yq chmod +x /workspace/.venv/bin/yq # AWS environment variables export AWS_DEFAULT_REGION="far-par" From 0bb9f4aa84969036e8e15beb36715d7bf72aab2f Mon Sep 17 00:00:00 2001 From: pmembari Date: Tue, 8 Apr 2025 12:17:11 +0200 Subject: [PATCH 34/61] feat: change repo reference --- machine-learning-process/files/init.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/machine-learning-process/files/init.sh b/machine-learning-process/files/init.sh index d631c41..07d7dbc 100644 --- a/machine-learning-process/files/init.sh +++ b/machine-learning-process/files/init.sh @@ -6,7 +6,7 @@ cd /workspace # Setup workspace mkdir machine-learning-process -git clone https://github.com/parham-membari-terradue/machine-learning-process.git +git clone https://github.com/eoap/machine-learning-process.git cd machine-learning-process # Install VS Code Extensions From e7768c942e05c5ad3c5e5c9aac1003ea61686c0e Mon Sep 17 00:00:00 2001 From: pmembari Date: Tue, 8 Apr 2025 16:01:47 +0200 Subject: [PATCH 35/61] feat: remove redundant sa --- charts/coder/templates/kinko-rabc.yaml | 51 -------------------------- 1 file changed, 51 deletions(-) delete mode 100644 charts/coder/templates/kinko-rabc.yaml diff --git a/charts/coder/templates/kinko-rabc.yaml b/charts/coder/templates/kinko-rabc.yaml deleted file mode 100644 index 0264eb4..0000000 --- a/charts/coder/templates/kinko-rabc.yaml +++ /dev/null @@ -1,51 +0,0 @@ -{{- if .Values.kaniko.enabled -}} -apiVersion: v1 -kind: Secret -metadata: - name: kaniko-secret -type: kubernetes.io/dockerconfigjson -data: - .dockerconfigjson: ew0KICAgICJhdXRocyI6IHsNCiAgICAgICAgImZha2UucmVnaXN0cnkuaW8iOiB7DQogICAgICAgICAgICAidXNlcm5hbWUiOiAiZmFrZXVzZXIiLA0KICAgICAgICAgICAgInBhc3N3b3JkIjogImZha2VwYXNzd29yZCIsDQogICAgICAgICAgICAiZW1haWwiOiAiZmFrZUBleGFtcGxlLmNvbSIsDQogICAgICAgICAgICAiYXV0aCI6ICIiDQogICAgICAgIH0NCiAgICB9DQp9 ---- -apiVersion: v1 -kind: Secret -metadata: - name: ttl-secret -type: kubernetes.io/dockerconfigjson -data: - .dockerconfigjson: ew0KICAgICJhdXRocyI6IHsNCiAgICAgICAgImZha2UucmVnaXN0cnkuaW8iOiB7DQogICAgICAgICAgICAidXNlcm5hbWUiOiAiZmFrZXVzZXIiLA0KICAgICAgICAgICAgInBhc3N3b3JkIjogImZha2VwYXNzd29yZCIsDQogICAgICAgICAgICAiZW1haWwiOiAiZmFrZUBleGFtcGxlLmNvbSIsDQogICAgICAgICAgICAiYXV0aCI6ICIiDQogICAgICAgIH0NCiAgICB9DQp9 ---- -apiVersion: v1 -kind: ServiceAccount -metadata: - name: kaniko-sa -imagePullSecrets: - - name: kaniko-secret ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: Role -metadata: - name: kaniko-role -rules: - - apiGroups: [""] - resources: ["pods"] - verbs: ["create", "get", "list", "watch", "delete"] - - apiGroups: [""] - resources: ["pods/exec"] - verbs: ["create"] - - apiGroups: [""] - resources: ["pods/log"] - verbs: ["get","list"] ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: RoleBinding -metadata: - name: kaniko-rolebinding -subjects: - - kind: ServiceAccount - name: kaniko-sa -roleRef: - kind: Role - name: kaniko-role - apiGroup: rbac.authorization.k8s.io -{{- end -}} From 88e19b09df30ebd205f20b0ea7e9112a6973d475 Mon Sep 17 00:00:00 2001 From: pmembari Date: Tue, 8 Apr 2025 18:13:00 +0200 Subject: [PATCH 36/61] feat: specify mlflow's chart version --- machine-learning-process/skaffold.yaml | 1 + 1 file changed, 1 insertion(+) diff --git a/machine-learning-process/skaffold.yaml b/machine-learning-process/skaffold.yaml index 2c24768..612ddeb 100644 --- a/machine-learning-process/skaffold.yaml +++ b/machine-learning-process/skaffold.yaml @@ -41,6 +41,7 @@ deploy: # MLflow - name: my-mlflow remoteChart: community-charts/mlflow + version: 0.15.0 namespace: tile-based-training createNamespace: true setValues: From bd325057674ddb6bb2cb9c5206cedaba88c26cde Mon Sep 17 00:00:00 2001 From: pmembari Date: Tue, 8 Apr 2025 18:17:35 +0200 Subject: [PATCH 37/61] docs: improve docs --- machine-learning-process/README.md | 13 +++---------- 1 file changed, 3 insertions(+), 10 deletions(-) diff --git a/machine-learning-process/README.md b/machine-learning-process/README.md index 07db6c2..bbf546b 100644 --- a/machine-learning-process/README.md +++ b/machine-learning-process/README.md @@ -1,10 +1,6 @@ # Installation -1- Create namespace: -``` -kubectl create ns tile-based-training -``` -2- install mlflow using helm: +1- install mlflow using helm: ``` helm repo add community-charts https://community-charts.github.io/helm-charts helm repo add localstack https://helm.localstack.cloud @@ -12,18 +8,15 @@ helm repo update ``` -3- Deploy everything using command below: +2- Deploy everything using command below: ``` skaffold dev ``` -4- Open the code-server and mlflow on your browser: +3- Open the code-server and mlflow on your browser: ``` code-server: http://localhost:8000 mlflow: http://localhost:5000 ``` -5- Follow the documentations of both `training` and `inference`. -6- Run the application packages with calrissian(documentation is provided for each module). - ### **Troubleshooting** From eec794daec571c8ea665fdc4312403c3bf75ad2f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?G=C3=A9rald=20Fenoy?= Date: Fri, 11 Apr 2025 12:56:19 +0200 Subject: [PATCH 38/61] Use patches in the macos profile and remove the standard one Helm chart version bump to 0.4.14 Closes #8 --- ogc-api-processes-with-zoo/skaffold.yaml | 186 ++++++++++------------- 1 file changed, 77 insertions(+), 109 deletions(-) diff --git a/ogc-api-processes-with-zoo/skaffold.yaml b/ogc-api-processes-with-zoo/skaffold.yaml index 266e2b6..71c9e5b 100644 --- a/ogc-api-processes-with-zoo/skaffold.yaml +++ b/ogc-api-processes-with-zoo/skaffold.yaml @@ -6,120 +6,88 @@ metadata: build: platforms: ["linux/amd64"] -profiles: - - name: macos - - deploy: - helm: - releases: - - name: zoo-project-dru - remoteChart: zoo-project/zoo-project-dru - namespace: eoap-zoo-project - createNamespace: true - version: 0.3.36 - valuesFiles: - - values.yaml - setValues: - iam.enabled: false - cookiecutter.templateUrl: https://github.com/eoap/zoo-service-template.git - cookiecutter.templateBranch: feature-secrets-nodeselector-imagepullsecrets - filter_in.enabled: true - filter_out.enabled: true - zoofpm.image.pullPolicy: Never - zookernel.image.pullPolicy: Never - workflow.storageClass: hostpath - workflow.argo.storageClass: hostpath - persistence.storageClass: hostpath - persistence.procServicesStorageClass: hostpath - persistence.tmpStorageClass: hostpath - persistence.procServicesAccessMode: ReadWriteMany - workflow.additionalInputs: - s3_bucket: results - region_name: us-east-1 - aws_secret_access_key: test - aws_access_key_id: test - endpoint_url: http://eoap-zoo-project-localstack.eoap-zoo-project.svc.cluster.local:4566 +deploy: + helm: + releases: + - name: zoo-project-dru + remoteChart: zoo-project/zoo-project-dru + namespace: eoap-zoo-project + createNamespace: true + version: 0.4.14 + valuesFiles: + - values.yaml + setValues: + iam.enabled: false + cookiecutter.templateUrl: https://github.com/eoap/zoo-service-template.git + cookiecutter.templateBranch: feature-secrets-nodeselector-imagepullsecrets + filter_in.enabled: true + filter_out.enabled: true + persistence.procServicesAccessMode: ReadWriteMany + workflow.additionalInputs: + s3_bucket: results + region_name: us-east-1 + aws_secret_access_key: test + aws_access_key_id: test + endpoint_url: http://eoap-zoo-project-localstack.eoap-zoo-project.svc.cluster.local:4566 - - name: eoap-zoo-project-coder - chartPath: ../charts/coder - namespace: eoap-zoo-project - createNamespace: true - setValues: - coder.coderImage: eoepca/pde-code-server:1.0.0 - coder.workspace: ogc-api-processes-with-zoo - #coder.storageClassName: standard - calrissian.storageClassName: hostpath - coder.storageClassName: hostpath - coder.workspaceStorage: 10Gi - coderResources.limits.cpu: '2' - coderResources.limits.memory: '6442450944' - coderResources.requests.cpu: '1' - coderResources.requests.memory: '4294967296' - calrissian.enabled: true + - name: eoap-zoo-project-coder + chartPath: ../charts/coder + namespace: eoap-zoo-project + createNamespace: true + setValues: + coder.coderImage: eoepca/pde-code-server:1.0.0 + coder.workspace: ogc-api-processes-with-zoo + coder.workspaceStorage: 10Gi + coderResources.limits.cpu: '2' + coderResources.limits.memory: '6442450944' + coderResources.requests.cpu: '1' + coderResources.requests.memory: '4294967296' + calrissian.enabled: true - setFiles: { - initScript: ./files/init.sh, - bashrcScript: ./files/bash-rc, - bashloginScript: ./files/bash-login - } - - name: eoap-zoo-project-localstack - remoteChart: localstack/localstack - namespace: eoap-zoo-project - createNamespace: true - setValues: - service.type: ClusterIP + setFiles: { + initScript: ./files/init.sh, + bashrcScript: ./files/bash-rc, + bashloginScript: ./files/bash-login + } + - name: eoap-zoo-project-localstack + remoteChart: localstack/localstack + namespace: eoap-zoo-project + createNamespace: true + setValues: + service.type: ClusterIP - - name: standard - - deploy: - helm: - releases: - - name: zoo-project-dru - remoteChart: zoo-project/zoo-project-dru - namespace: eoap-zoo-project - createNamespace: true - version: 0.3.36 - valuesFiles: - - values.yaml - setValues: - iam.enabled: false - cookiecutter.templateUrl: https://github.com/eoap/zoo-service-template.git - cookiecutter.templateBranch: feature-secrets-nodeselector-imagepullsecrets - filter_in.enabled: true - filter_out.enabled: true - persistence.procServicesAccessMode: ReadWriteMany - workflow.additionalInputs: - s3_bucket: results - region_name: us-east-1 - aws_secret_access_key: test - aws_access_key_id: test - endpoint_url: http://eoap-zoo-project-localstack.eoap-zoo-project.svc.cluster.local:4566 +profiles: - - name: eoap-zoo-project-coder - chartPath: ../charts/coder - namespace: eoap-zoo-project - createNamespace: true - setValues: - coder.coderImage: eoepca/pde-code-server:1.0.0 - coder.workspace: ogc-api-processes-with-zoo - coder.workspaceStorage: 10Gi - coderResources.limits.cpu: '2' - coderResources.limits.memory: '6442450944' - coderResources.requests.cpu: '1' - coderResources.requests.memory: '4294967296' - calrissian.enabled: true + - name: macos - setFiles: { - initScript: ./files/init.sh, - bashrcScript: ./files/bash-rc, - bashloginScript: ./files/bash-login - } - - name: eoap-zoo-project-localstack - remoteChart: localstack/localstack - namespace: eoap-zoo-project - createNamespace: true - setValues: - service.type: ClusterIP + patches: + - op: add + path: /deploy/helm/releases/0/setValues/zoofpm.image.pullPolicy + value: Never + - op: add + path: /deploy/helm/releases/0/setValues/zookernel.image.pullPolicy + value: Never + - op: add + path: /deploy/helm/releases/0/setValues/workflow.storageClass + value: hostpath + - op: add + path: /deploy/helm/releases/0/setValues/workflow.argo.storageClass + value: hostpath + - op: add + path: /deploy/helm/releases/0/setValues/persistence.storageClass + value: hostpath + - op: add + path: /deploy/helm/releases/0/setValues/persistence.procServicesStorageClass + value: hostpath + - op: add + path: /deploy/helm/releases/0/setValues/persistence.tmpStorageClass + value: hostpath + - op: add + path: /deploy/helm/releases/1/setValues/coder.storageClassName + value: hostpath + - op: add + path: /deploy/helm/releases/1/setValues/calrissian.storageClassName + value: hostpath portForward: - resourceType: service From de78200b35accd72328de5e68fcc324be18b395b Mon Sep 17 00:00:00 2001 From: pmembari Date: Tue, 29 Apr 2025 16:15:41 +0200 Subject: [PATCH 39/61] feat: checkout to develop in init --- machine-learning-process/files/init.sh | 4 +--- machine-learning-process/skaffold.yaml | 9 +++++---- 2 files changed, 6 insertions(+), 7 deletions(-) diff --git a/machine-learning-process/files/init.sh b/machine-learning-process/files/init.sh index 07d7dbc..ace5161 100644 --- a/machine-learning-process/files/init.sh +++ b/machine-learning-process/files/init.sh @@ -8,6 +8,7 @@ cd /workspace mkdir machine-learning-process git clone https://github.com/eoap/machine-learning-process.git cd machine-learning-process +git checkout develop # Install VS Code Extensions code-server --install-extension ms-python.python @@ -46,11 +47,8 @@ chmod +x /workspace/.venv/bin/skaffold # mv yq_linux_amd64 /workspace/.venv/bin/yq chmod +x /workspace/.venv/bin/yq # AWS environment variables -export AWS_DEFAULT_REGION="far-par" export AWS_ACCESS_KEY_ID="test" export AWS_SECRET_ACCESS_KEY="test" -export AWS_BUCKET_NAME="ai-ext-bucket-dev" -export AWS_S3_ENDPOINT="https://s3.fr-par.scw.cloud" export MLFLOW_TRACKING_URI="http://my-mlflow:5000" export TASK_X_REMOTE_TASKFILES="1" export PATH=$PATH:/workspace/.venv/bin diff --git a/machine-learning-process/skaffold.yaml b/machine-learning-process/skaffold.yaml index 612ddeb..d1f67c7 100644 --- a/machine-learning-process/skaffold.yaml +++ b/machine-learning-process/skaffold.yaml @@ -23,14 +23,15 @@ deploy: coder.storageClassName: standard coder.coderImage: eoepca/pde-code-server:1.0.0 coder.workspace: machine-learning-process - coder.workspaceStorage: 30Gi - coderResources.limits.cpu: '2' - coderResources.limits.memory: '6442450944' + coder.workspaceStorage: 40Gi + coderResources.limits.cpu: '3' + coderResources.limits.memory: '10737418240' coderResources.requests.cpu: '1' coderResources.requests.memory: '4294967296' calrissian.enabled: true + #calrissian.storage: 30Gi coder.securityContext.privileged: true - kaniko.enabled: true + kaniko.enabled: false skaffold.enabled: true setFiles: { initScript: ./files/init.sh, From 5c9de908f87d470b476cf78ecb997fa998c6eeac Mon Sep 17 00:00:00 2001 From: pmembari Date: Mon, 12 May 2025 16:24:48 +0200 Subject: [PATCH 40/61] feat: update code server with tzdata --- machine-learning-process/files/bash-rc | 4 +++- machine-learning-process/files/init.sh | 16 +++++++++++----- machine-learning-process/skaffold.yaml | 2 +- 3 files changed, 15 insertions(+), 7 deletions(-) diff --git a/machine-learning-process/files/bash-rc b/machine-learning-process/files/bash-rc index 5b4e6a5..0d43c96 100644 --- a/machine-learning-process/files/bash-rc +++ b/machine-learning-process/files/bash-rc @@ -1,8 +1,10 @@ alias ll="ls -l" - +DEBIAN_FRONTEND=noninteractive apt-get install -y tzdata +dpkg-reconfigure --frontend noninteractive tzdata alias aws="aws --endpoint-url=http://eoap-event-driven-localstack:4566" export PATH=/workspace/.venv/bin:$PATH export TASK_X_REMOTE_TASKFILES=1 export MLFLOW_TRACKING_URI="http://my-mlflow:5000" +export ARROW_TZDATA_DIR=/usr/share/zoneinfo export PATH=$PATH:/workspace/.venv/bin:/workspace/.venv/bin/yq source /workspace/.venv/bin/activate \ No newline at end of file diff --git a/machine-learning-process/files/init.sh b/machine-learning-process/files/init.sh index ace5161..248077e 100644 --- a/machine-learning-process/files/init.sh +++ b/machine-learning-process/files/init.sh @@ -8,7 +8,6 @@ cd /workspace mkdir machine-learning-process git clone https://github.com/eoap/machine-learning-process.git cd machine-learning-process -git checkout develop # Install VS Code Extensions code-server --install-extension ms-python.python @@ -24,7 +23,7 @@ echo '{"workbench.colorTheme": "Visual Studio Dark"}' > /workspace/User/settings # Setup Python virtual environment python -m venv /workspace/.venv source /workspace/.venv/bin/activate -/workspace/.venv/bin/python -m pip install --no-cache-dir tomlq calrissian yq +/workspace/.venv/bin/python -m pip install --no-cache-dir tomlq calrissian yq backports.zoneinfo echo "**** install kubectl ****" @@ -42,13 +41,20 @@ chmod +x /workspace/.venv/bin/task curl -s -L https://storage.googleapis.com/skaffold/releases/latest/skaffold-linux-amd64 > /workspace/.venv/bin/skaffold chmod +x /workspace/.venv/bin/skaffold -# curl -s -LO https://github.com/mikefarah/yq/releases/download/v4.45.1/yq_linux_amd64.tar.gz -# tar -xvf yq_linux_amd64.tar.gz -# mv yq_linux_amd64 /workspace/.venv/bin/yq +# install tzdata + +wget https://github.com/eggert/tz/archive/refs/tags/2024a.tar.gz +tar -xzf 2024a.tar.gz +mkdir -p /usr/share/zoneinfo +cp -r tz-2024a/** /usr/share/zoneinfo + + + chmod +x /workspace/.venv/bin/yq # AWS environment variables export AWS_ACCESS_KEY_ID="test" export AWS_SECRET_ACCESS_KEY="test" export MLFLOW_TRACKING_URI="http://my-mlflow:5000" +export ARROW_TZDATA_DIR=/usr/share/zoneinfo export TASK_X_REMOTE_TASKFILES="1" export PATH=$PATH:/workspace/.venv/bin diff --git a/machine-learning-process/skaffold.yaml b/machine-learning-process/skaffold.yaml index d1f67c7..a6fc264 100644 --- a/machine-learning-process/skaffold.yaml +++ b/machine-learning-process/skaffold.yaml @@ -21,7 +21,7 @@ deploy: setValues: coder.serviceType: NodePort coder.storageClassName: standard - coder.coderImage: eoepca/pde-code-server:1.0.0 + coder.coderImage: eoepca/pde-code-server:1.0.1 coder.workspace: machine-learning-process coder.workspaceStorage: 40Gi coderResources.limits.cpu: '3' From da380158d1e36e0d562a24f97ffa82f1cb976f36 Mon Sep 17 00:00:00 2001 From: Fabrice Brito Date: Mon, 12 May 2025 16:40:53 +0200 Subject: [PATCH 41/61] updates requirements --- dask-gateway/files/init.sh | 2 +- dask-gateway/requirements.txt | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/dask-gateway/files/init.sh b/dask-gateway/files/init.sh index 02b5b1c..84c40d0 100644 --- a/dask-gateway/files/init.sh +++ b/dask-gateway/files/init.sh @@ -24,7 +24,7 @@ pip install git+https://github.com/Terradue/calrissian@dask-gateway tomlq # calr python -m venv /workspace/.venv source /workspace/.venv/bin/activate -/workspace/.venv/bin/python -m pip install --no-cache-dir numpy==1.26.3 dask==2024.1.0 distributed==2024.1.0 dask-gateway==2024.1.0 pystac bokeh rioxarray==0.18.1 loguru==0.7.3 odc-stac[botocore]==0.3.10 stackstac==0.5.1 pystac-client planetary-computer xarray-spatial ipykernel +/workspace/.venv/bin/python -m pip install --no-cache-dir numpy==1.26.3 dask==2024.1.0 distributed==2024.1.0 dask-gateway==2024.1.0 pystac bokeh rioxarray==0.18.1 loguru==0.7.3 odc-stac[botocore]==0.3.10 stackstac==0.5.1 pystac-client planetary-computer xarray-spatial "xarray[complete]" ipykernel rio-color res=$? if [ $res -ne 0 ]; then echo "Failed to install packages" diff --git a/dask-gateway/requirements.txt b/dask-gateway/requirements.txt index 0687b22..057de8f 100644 --- a/dask-gateway/requirements.txt +++ b/dask-gateway/requirements.txt @@ -10,4 +10,6 @@ odc-stac[botocore]==0.3.10 stackstac==0.5.1 pystac-client planetary-computer -xarray-spatial \ No newline at end of file +xarray-spatial +rio-color +xarray[complete] \ No newline at end of file From 99793c2af345b2cb0fba278a9d88af7304f985d6 Mon Sep 17 00:00:00 2001 From: parham-membari-terradue Date: Tue, 13 May 2025 15:22:25 +0200 Subject: [PATCH 42/61] feat: edit coder docker reference --- machine-learning-process/skaffold.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/machine-learning-process/skaffold.yaml b/machine-learning-process/skaffold.yaml index a6fc264..a61f634 100644 --- a/machine-learning-process/skaffold.yaml +++ b/machine-learning-process/skaffold.yaml @@ -21,7 +21,7 @@ deploy: setValues: coder.serviceType: NodePort coder.storageClassName: standard - coder.coderImage: eoepca/pde-code-server:1.0.1 + coder.coderImage: eoepca/pde-code-server:devlatest coder.workspace: machine-learning-process coder.workspaceStorage: 40Gi coderResources.limits.cpu: '3' From 105a2b6a0b63618ac9044a0a1033376d78ef7e50 Mon Sep 17 00:00:00 2001 From: parham-membari-terradue Date: Tue, 13 May 2025 15:33:13 +0200 Subject: [PATCH 43/61] feat: zoneinfo handeled --- machine-learning-process/files/init.sh | 6 ------ 1 file changed, 6 deletions(-) diff --git a/machine-learning-process/files/init.sh b/machine-learning-process/files/init.sh index 248077e..c92175a 100644 --- a/machine-learning-process/files/init.sh +++ b/machine-learning-process/files/init.sh @@ -41,12 +41,6 @@ chmod +x /workspace/.venv/bin/task curl -s -L https://storage.googleapis.com/skaffold/releases/latest/skaffold-linux-amd64 > /workspace/.venv/bin/skaffold chmod +x /workspace/.venv/bin/skaffold -# install tzdata - -wget https://github.com/eggert/tz/archive/refs/tags/2024a.tar.gz -tar -xzf 2024a.tar.gz -mkdir -p /usr/share/zoneinfo -cp -r tz-2024a/** /usr/share/zoneinfo From ae40984252cddad8ed5bc2bd5e6300cc09383203 Mon Sep 17 00:00:00 2001 From: Fabrice Brito Date: Tue, 20 May 2025 17:32:06 +0200 Subject: [PATCH 44/61] adds CI for building dask and how-to --- .github/workflows/build.yaml | 37 +++++++++++++++++++++ dask-gateway/Dockerfile.coder | 43 ++++++++++++++++++++----- dask-gateway/codemeta.json | 3 ++ dask-gateway/requirements.txt | 18 +++++------ dask-gateway/skaffold.yaml | 3 +- how-to/{Dockerfile => Dockerfile.coder} | 0 how-to/codemeta.json | 3 ++ how-to/skaffold.yaml | 9 ++++-- 8 files changed, 95 insertions(+), 21 deletions(-) create mode 100644 .github/workflows/build.yaml create mode 100644 dask-gateway/codemeta.json rename how-to/{Dockerfile => Dockerfile.coder} (100%) create mode 100644 how-to/codemeta.json diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml new file mode 100644 index 0000000..35e3f8d --- /dev/null +++ b/.github/workflows/build.yaml @@ -0,0 +1,37 @@ +name: build +on: + push: + branches: + - main + - develop + + paths: + # Only rebuild website when apps have changed + - 'dask-gateway/**' + - .github/** + +jobs: + + + container-build: + + #needs: [] + + runs-on: ubuntu-latest + + strategy: + matrix: + step: ["dask-gateway", "how-to"] + + steps: + - uses: actions/checkout@v2 + - run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin + - name: build & push image + run: | + version=$(cat ${{ matrix.step }}/codemeta.json | jq -r .version )" + IMAGE_ID=ghcr.io/eoap/dev-platform-eoap/${{ matrix.step }}-coder + docker build dev-platform-eoap/${{ matrix.step }} --file dev-platform-eoap/${{ matrix.step }}/Dockerfile.coder --tag ${{ matrix.step }} + docker tag ${{ matrix.step }} $IMAGE_ID:${version} + docker push $IMAGE_ID:${version} + + \ No newline at end of file diff --git a/dask-gateway/Dockerfile.coder b/dask-gateway/Dockerfile.coder index c19f367..f04c9a8 100644 --- a/dask-gateway/Dockerfile.coder +++ b/dask-gateway/Dockerfile.coder @@ -18,13 +18,13 @@ RUN \ echo "**** install build dependencies ****" && \ apt-get update && \ apt-get install -y \ - nodejs + nodejs && \ + apt remove yq jq -y RUN \ echo "**** install runtime dependencies ****" && \ apt-get install -y \ git \ - jq \ libatomic1 \ nano \ net-tools \ @@ -89,12 +89,8 @@ ENV USER=jovyan \ # mamba clean -a RUN \ - echo "**** install yq, aws cli ****" && \ - VERSION="v4.12.2" && \ - BINARY="yq_linux_amd64" && \ - wget --quiet https://github.com/mikefarah/yq/releases/download/${VERSION}/${BINARY}.tar.gz -O - |\ - tar xz && mv ${BINARY} /usr/bin/yq && \ - pip install awscli && \ + echo "**** install aws cli ****" && \ + pip install awscli && \ pip install awscli-plugin-endpoint RUN \ @@ -122,4 +118,35 @@ ENTRYPOINT ["/opt/entrypoint.sh"] EXPOSE 8888 +RUN echo "**** install kubectl ****" && \ + curl -s -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" && \ + chmod +x kubectl && \ + mkdir -p /workspace/.venv/bin && \ + mv ./kubectl /usr/local/bin/kubectl + + +RUN curl -s -L https://github.com/pypa/hatch/releases/latest/download/hatch-x86_64-unknown-linux-gnu.tar.gz | tar -xzvf - -C /usr/local/bin/ && \ + chmod +x /usr/local/bin/hatch + +RUN curl -s -L https://github.com/go-task/task/releases/download/v3.41.0/task_linux_amd64.tar.gz | tar -xzvf - -C /usr/local/bin/ && \ + chmod +x /usr/local/bin/task + +RUN curl -s -L https://storage.googleapis.com/skaffold/releases/latest/skaffold-linux-amd64 > /usr/local/bin/skaffold && \ + chmod +x /usr/local/bin/skaffold + +RUN pip3 install tomlq && tomlq --version + +RUN rm -f /usr/bin/yq && \ + rm -f /usr/local/bin/yq && \ + curl -s -LO https://github.com/mikefarah/yq/releases/download/v4.45.1/yq_linux_amd64.tar.gz && \ + tar -xvf yq_linux_amd64.tar.gz && \ + mv yq_linux_amd64 /usr/local/bin/yq && \ + cp -v /usr/local/bin/yq /usr/bin/yq && \ + chmod +x /usr/bin/yq /usr/local/bin/yq + +RUN curl -s -L https://github.com/jqlang/jq/releases/download/jq-1.7.1/jq-linux-amd64 > /usr/bin/jq && \ + chmod +x /usr/bin/jq + +RUN pip install --upgrade uv + USER jovyan \ No newline at end of file diff --git a/dask-gateway/codemeta.json b/dask-gateway/codemeta.json new file mode 100644 index 0000000..dfba51c --- /dev/null +++ b/dask-gateway/codemeta.json @@ -0,0 +1,3 @@ +{ + "version": "0.1.0" +} \ No newline at end of file diff --git a/dask-gateway/requirements.txt b/dask-gateway/requirements.txt index 057de8f..431358a 100644 --- a/dask-gateway/requirements.txt +++ b/dask-gateway/requirements.txt @@ -1,15 +1,15 @@ numpy==1.26.3 -dask==2024.1.0 -distributed==2024.1.0 -dask-gateway==2024.1.0 -pystac +dask==2025.4.0 +distributed==2025.4.0 +dask-gateway==2025.4.0 +pystac==1.13.0 bokeh==3.6.2 rioxarray==0.18.1 loguru==0.7.3 odc-stac[botocore]==0.3.10 stackstac==0.5.1 -pystac-client -planetary-computer -xarray-spatial -rio-color -xarray[complete] \ No newline at end of file +pystac-client==0.8.6 +planetary-computer==1.0.0 +xarray-spatial==0.4.0 +rio-color==2.0.1 +xarray[complete]==2025.4.0 \ No newline at end of file diff --git a/dask-gateway/skaffold.yaml b/dask-gateway/skaffold.yaml index 21f30f6..7773e08 100644 --- a/dask-gateway/skaffold.yaml +++ b/dask-gateway/skaffold.yaml @@ -2,7 +2,6 @@ apiVersion: skaffold/v4beta9 kind: Config metadata: name: eoap-dask-gateway - build: tagPolicy: envTemplate: @@ -23,7 +22,9 @@ build: dockerfile: Dockerfile.coder deploy: + statusCheckDeadlineSeconds: 2400 helm: + releases: - name: dask-gateway remoteChart: https://helm.dask.org/dask-gateway-2024.1.0.tgz diff --git a/how-to/Dockerfile b/how-to/Dockerfile.coder similarity index 100% rename from how-to/Dockerfile rename to how-to/Dockerfile.coder diff --git a/how-to/codemeta.json b/how-to/codemeta.json new file mode 100644 index 0000000..dfba51c --- /dev/null +++ b/how-to/codemeta.json @@ -0,0 +1,3 @@ +{ + "version": "0.1.0" +} \ No newline at end of file diff --git a/how-to/skaffold.yaml b/how-to/skaffold.yaml index f7cd71a..b9aafba 100644 --- a/how-to/skaffold.yaml +++ b/how-to/skaffold.yaml @@ -10,7 +10,10 @@ build: artifacts: - image: ttl.sh/eoap-how-to-coder - + context: . + docker: + dockerfile: Dockerfile.coder + deploy: helm: releases: @@ -22,14 +25,14 @@ deploy: coder.coderImage: ttl.sh/eoap-how-to-coder:2h setValues: coder.workspace: how-to - coder.storageClassName: openebs-kernel-nfs-scw + coder.storageClassName: standard #openebs-kernel-nfs-scw coder.workspaceStorage: 10Gi coderResources.limits.cpu: '2' coderResources.limits.memory: '6442450944' coderResources.requests.cpu: '1' coderResources.requests.memory: '4294967296' calrissian.enabled: true - calrissian.storageClassName: openebs-kernel-nfs-scw + calrissian.storageClassName: standard #openebs-kernel-nfs-scw calrissian.imagePullSecrets: [] skaffold.enabled: True setFiles: { From e9b6bb58589b83bfbd41bfdf793468de848dcd2a Mon Sep 17 00:00:00 2001 From: Fabrice Brito Date: Tue, 20 May 2025 17:34:45 +0200 Subject: [PATCH 45/61] fixes CI --- .github/workflows/build.yaml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index 35e3f8d..7160fe2 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -8,8 +8,8 @@ on: paths: # Only rebuild website when apps have changed - 'dask-gateway/**' - - .github/** - + - '.github/**' + - 'how-to/**' jobs: @@ -28,7 +28,7 @@ jobs: - run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin - name: build & push image run: | - version=$(cat ${{ matrix.step }}/codemeta.json | jq -r .version )" + version=$(cat ${{ matrix.step }}/codemeta.json | jq -r .version ) IMAGE_ID=ghcr.io/eoap/dev-platform-eoap/${{ matrix.step }}-coder docker build dev-platform-eoap/${{ matrix.step }} --file dev-platform-eoap/${{ matrix.step }}/Dockerfile.coder --tag ${{ matrix.step }} docker tag ${{ matrix.step }} $IMAGE_ID:${version} From 3e6364d524a3670d27a7b439b0555354cf9ed254 Mon Sep 17 00:00:00 2001 From: Fabrice Brito Date: Tue, 20 May 2025 17:36:52 +0200 Subject: [PATCH 46/61] fixes CI --- .github/workflows/build.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index 7160fe2..dab4ba5 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -30,7 +30,7 @@ jobs: run: | version=$(cat ${{ matrix.step }}/codemeta.json | jq -r .version ) IMAGE_ID=ghcr.io/eoap/dev-platform-eoap/${{ matrix.step }}-coder - docker build dev-platform-eoap/${{ matrix.step }} --file dev-platform-eoap/${{ matrix.step }}/Dockerfile.coder --tag ${{ matrix.step }} + docker build ${{ matrix.step }} --file ${{ matrix.step }}/Dockerfile.coder --tag ${{ matrix.step }} docker tag ${{ matrix.step }} $IMAGE_ID:${version} docker push $IMAGE_ID:${version} From 92964c8d923f0fcdbf85e43d23ca705982bf7777 Mon Sep 17 00:00:00 2001 From: Fabrice Brito Date: Tue, 20 May 2025 17:45:52 +0200 Subject: [PATCH 47/61] add advanced-tooling to CI --- .github/workflows/build.yaml | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index dab4ba5..b76d08d 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -7,8 +7,9 @@ on: paths: # Only rebuild website when apps have changed - - 'dask-gateway/**' - '.github/**' + - 'advanced-tooling/**' + - 'dask-gateway/**' - 'how-to/**' jobs: @@ -21,7 +22,7 @@ jobs: strategy: matrix: - step: ["dask-gateway", "how-to"] + step: ["advanced-tooling", "dask-gateway", "how-to"] steps: - uses: actions/checkout@v2 From e376f09edc4b431b995cf3efd3fc920a3d995f98 Mon Sep 17 00:00:00 2001 From: Fabrice Brito Date: Tue, 20 May 2025 17:48:25 +0200 Subject: [PATCH 48/61] adds codemeta.json to advanced-tooling --- advanced-tooling/codemeta.json | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 advanced-tooling/codemeta.json diff --git a/advanced-tooling/codemeta.json b/advanced-tooling/codemeta.json new file mode 100644 index 0000000..dfba51c --- /dev/null +++ b/advanced-tooling/codemeta.json @@ -0,0 +1,3 @@ +{ + "version": "0.1.0" +} \ No newline at end of file From cc21d51e92c2b8f7eec5959605deb78aa7444beb Mon Sep 17 00:00:00 2001 From: Fabrice Brito Date: Tue, 20 May 2025 17:52:33 +0200 Subject: [PATCH 49/61] removes tooling from init.sh and adds tooling in dockerfile --- advanced-tooling/Dockerfile.coder | 35 +++++++++++++++++++++++++++++-- advanced-tooling/files/init.sh | 21 +------------------ 2 files changed, 34 insertions(+), 22 deletions(-) diff --git a/advanced-tooling/Dockerfile.coder b/advanced-tooling/Dockerfile.coder index c19f367..b9be26d 100644 --- a/advanced-tooling/Dockerfile.coder +++ b/advanced-tooling/Dockerfile.coder @@ -18,13 +18,13 @@ RUN \ echo "**** install build dependencies ****" && \ apt-get update && \ apt-get install -y \ - nodejs + nodejs && \ + apt remove yq jq -y RUN \ echo "**** install runtime dependencies ****" && \ apt-get install -y \ git \ - jq \ libatomic1 \ nano \ net-tools \ @@ -122,4 +122,35 @@ ENTRYPOINT ["/opt/entrypoint.sh"] EXPOSE 8888 +RUN echo "**** install kubectl ****" && \ + curl -s -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" && \ + chmod +x kubectl && \ + mkdir -p /workspace/.venv/bin && \ + mv ./kubectl /usr/local/bin/kubectl + + +RUN curl -s -L https://github.com/pypa/hatch/releases/latest/download/hatch-x86_64-unknown-linux-gnu.tar.gz | tar -xzvf - -C /usr/local/bin/ && \ + chmod +x /usr/local/bin/hatch + +RUN curl -s -L https://github.com/go-task/task/releases/download/v3.41.0/task_linux_amd64.tar.gz | tar -xzvf - -C /usr/local/bin/ && \ + chmod +x /usr/local/bin/task + +RUN curl -s -L https://storage.googleapis.com/skaffold/releases/latest/skaffold-linux-amd64 > /usr/local/bin/skaffold && \ + chmod +x /usr/local/bin/skaffold + +RUN pip3 install tomlq && tomlq --version + +RUN rm -f /usr/bin/yq && \ + rm -f /usr/local/bin/yq && \ + curl -s -LO https://github.com/mikefarah/yq/releases/download/v4.45.1/yq_linux_amd64.tar.gz && \ + tar -xvf yq_linux_amd64.tar.gz && \ + mv yq_linux_amd64 /usr/local/bin/yq && \ + cp -v /usr/local/bin/yq /usr/bin/yq && \ + chmod +x /usr/bin/yq /usr/local/bin/yq + +RUN curl -s -L https://github.com/jqlang/jq/releases/download/jq-1.7.1/jq-linux-amd64 > /usr/bin/jq && \ + chmod +x /usr/bin/jq + +RUN pip install --upgrade uv + USER jovyan \ No newline at end of file diff --git a/advanced-tooling/files/init.sh b/advanced-tooling/files/init.sh index 6973b4c..9f49192 100644 --- a/advanced-tooling/files/init.sh +++ b/advanced-tooling/files/init.sh @@ -20,26 +20,7 @@ echo '{"workbench.colorTheme": "Visual Studio Dark"}' > /workspace/User/settings python -m venv /workspace/.venv source /workspace/.venv/bin/activate /workspace/.venv/bin/python -m pip install --no-cache-dir tomlq calrissian -# /workspace/.venv/bin/python -m ipykernel install --user --name mastering_env --display-name "Python (Mastering Application Package)" - -echo "**** install kubectl ****" -curl -s -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" -chmod +x kubectl -mkdir -p /workspace/.venv/bin -mv ./kubectl /workspace/.venv/bin/kubectl - - -curl -s -L https://github.com/pypa/hatch/releases/latest/download/hatch-x86_64-unknown-linux-gnu.tar.gz | tar -xzvf - -C /workspace/.venv/bin/ -chmod +x /workspace/.venv/bin/hatch -curl -s -L https://github.com/go-task/task/releases/download/v3.41.0/task_linux_amd64.tar.gz | tar -xzvf - -C /workspace/.venv/bin/ -chmod +x /workspace/.venv/bin/task - -curl -s -L https://storage.googleapis.com/skaffold/releases/latest/skaffold-linux-amd64 > /workspace/.venv/bin/skaffold -chmod +x /workspace/.venv/bin/skaffold - -curl -s -LO https://github.com/mikefarah/yq/releases/download/v4.45.1/yq_linux_amd64.tar.gz -tar -xvf yq_linux_amd64.tar.gz -mv yq_linux_amd64 /workspace/.venv/bin/yq +/workspace/.venv/bin/python -m ipykernel install --user --name advanced_tooling --display-name "Python (Advanced Tooling)" export AWS_DEFAULT_REGION="us-east-1" From 12a3f551df5ddded20d23ead6eb6c46adea8907f Mon Sep 17 00:00:00 2001 From: Fabrice Brito Date: Tue, 10 Jun 2025 16:14:40 +0200 Subject: [PATCH 50/61] adds application-package-patterns --- application-package-patterns/Dockerfile.coder | 7 +++ application-package-patterns/codemeta.json | 3 ++ application-package-patterns/files/bash-login | 1 + application-package-patterns/files/bash-rc | 8 +++ application-package-patterns/files/init.sh | 32 ++++++++++++ application-package-patterns/skaffold.yaml | 49 +++++++++++++++++++ 6 files changed, 100 insertions(+) create mode 100644 application-package-patterns/Dockerfile.coder create mode 100644 application-package-patterns/codemeta.json create mode 100644 application-package-patterns/files/bash-login create mode 100644 application-package-patterns/files/bash-rc create mode 100644 application-package-patterns/files/init.sh create mode 100644 application-package-patterns/skaffold.yaml diff --git a/application-package-patterns/Dockerfile.coder b/application-package-patterns/Dockerfile.coder new file mode 100644 index 0000000..a6a1564 --- /dev/null +++ b/application-package-patterns/Dockerfile.coder @@ -0,0 +1,7 @@ +FROM eoepca/pde-code-server:1.0.0 + +USER root + +RUN apt update && apt install -y graphviz + +USER jovyan \ No newline at end of file diff --git a/application-package-patterns/codemeta.json b/application-package-patterns/codemeta.json new file mode 100644 index 0000000..dfba51c --- /dev/null +++ b/application-package-patterns/codemeta.json @@ -0,0 +1,3 @@ +{ + "version": "0.1.0" +} \ No newline at end of file diff --git a/application-package-patterns/files/bash-login b/application-package-patterns/files/bash-login new file mode 100644 index 0000000..5becb33 --- /dev/null +++ b/application-package-patterns/files/bash-login @@ -0,0 +1 @@ +source /workspace/.bashrc \ No newline at end of file diff --git a/application-package-patterns/files/bash-rc b/application-package-patterns/files/bash-rc new file mode 100644 index 0000000..00102c4 --- /dev/null +++ b/application-package-patterns/files/bash-rc @@ -0,0 +1,8 @@ +alias ll="ls -l" + +alias aws="aws --endpoint-url=http://eoap-how-to-localstack:4566" + +export PATH=/workspace/.venv/bin:$PATH +export TASK_X_REMOTE_TASKFILES=1 + +source /workspace/.venv/bin/activate \ No newline at end of file diff --git a/application-package-patterns/files/init.sh b/application-package-patterns/files/init.sh new file mode 100644 index 0000000..8b92a62 --- /dev/null +++ b/application-package-patterns/files/init.sh @@ -0,0 +1,32 @@ +#!/bin/bash + +cd /workspace + +git clone 'https://github.com/eoap/application-package-patterns.git' + +code-server --install-extension ms-python.python +code-server --install-extension redhat.vscode-yaml +code-server --install-extension sbg-rabix.benten-cwl +code-server --install-extension ms-toolsai.jupyter + +ln -s /workspace/.local/share/code-server/extensions /workspace/extensions + +mkdir -p /workspace/User/ + +echo '{"workbench.colorTheme": "Visual Studio Dark"}' > /workspace/User/settings.json + +python -m venv /workspace/.venv +source /workspace/.venv/bin/activate +/workspace/.venv/bin/python -m pip install --no-cache-dir stactools rasterio requests stac-asset click-logging tabulate tqdm pystac-client ipykernel loguru scikit-image rio_stac calrissian tomlq + +/workspace/.venv/bin/python -m ipykernel install --user --name cwl_how_to_env --display-name "Python (CWL How-To's)" + +curl -s -L https://github.com/go-task/task/releases/download/v3.41.0/task_linux_amd64.tar.gz | tar -xzvf - -C /workspace/.venv/bin/ +chmod +x /workspace/.venv/bin/task + +curl -s -L https://storage.googleapis.com/skaffold/releases/latest/skaffold-linux-amd64 > /workspace/.venv/bin/skaffold +chmod +x /workspace/.venv/bin/skaffold + +curl -s -LO https://github.com/mikefarah/yq/releases/download/v4.45.1/yq_linux_amd64.tar.gz +tar -xvf yq_linux_amd64.tar.gz +mv yq_linux_amd64 /workspace/.venv/bin/yq \ No newline at end of file diff --git a/application-package-patterns/skaffold.yaml b/application-package-patterns/skaffold.yaml new file mode 100644 index 0000000..eafe30b --- /dev/null +++ b/application-package-patterns/skaffold.yaml @@ -0,0 +1,49 @@ +apiVersion: skaffold/v4beta9 +kind: Config +metadata: + name: eoap-application-package-patterns + +build: + tagPolicy: + envTemplate: + template: 2h + + artifacts: + - image: ttl.sh/eoap-application-package-patterns-coder + context: . + docker: + dockerfile: Dockerfile.coder + +deploy: + helm: + releases: + - name: eoap-application-package-patterns + chartPath: ../charts/coder + namespace: eoap-application-package-patterns + createNamespace: true + setValueTemplates: + coder.coderImage: ttl.sh/eoap-application-package-patterns-coder:2h + setValues: + coder.workspace: application-package-patterns + coder.storageClassName: standard #openebs-kernel-nfs-scw + coder.workspaceStorage: 10Gi + coderResources.limits.cpu: '2' + coderResources.limits.memory: '6442450944' + coderResources.requests.cpu: '1' + coderResources.requests.memory: '4294967296' + calrissian.enabled: true + calrissian.storageClassName: standard #openebs-kernel-nfs-scw + calrissian.imagePullSecrets: [] + skaffold.enabled: True + setFiles: { + initScript: ./files/init.sh, + bashrcScript: ./files/bash-rc, + bashloginScript: ./files/bash-login + } + +portForward: + - resourceType: service + resourceName: code-server-service + namespace: eoap-application-package-patterns + port: 8080 + localPort: 8000 \ No newline at end of file From fd9e49a57c51c261239b7667c0dab7e53b6a6629 Mon Sep 17 00:00:00 2001 From: Fabrice Brito Date: Fri, 20 Jun 2025 10:36:21 +0200 Subject: [PATCH 51/61] updates app-package-patterns config --- application-package-patterns/Dockerfile.coder | 151 +++++++++++++++++- application-package-patterns/entrypoint.sh | 37 +++++ application-package-patterns/files/init.sh | 16 +- application-package-patterns/skaffold.yaml | 9 ++ 4 files changed, 203 insertions(+), 10 deletions(-) create mode 100644 application-package-patterns/entrypoint.sh diff --git a/application-package-patterns/Dockerfile.coder b/application-package-patterns/Dockerfile.coder index a6a1564..f04c9a8 100644 --- a/application-package-patterns/Dockerfile.coder +++ b/application-package-patterns/Dockerfile.coder @@ -1,7 +1,152 @@ -FROM eoepca/pde-code-server:1.0.0 +FROM docker.io/library/python:3.10.12@sha256:bac3a0e0d16125977e351c861e2f4b12ecafeaa6f72431dc970d0b9155103232 -USER root +RUN \ + apt-get update && \ + apt-get install -y \ + build-essential \ + curl \ + gcc \ + vim \ + tree \ + file -RUN apt update && apt install -y graphviz +RUN \ + echo "**** install node repo ****" && \ + curl -s https://deb.nodesource.com/gpgkey/nodesource.gpg.key | apt-key add - && \ + echo 'deb https://deb.nodesource.com/node_14.x jammy main' \ + > /etc/apt/sources.list.d/nodesource.list && \ + echo "**** install build dependencies ****" && \ + apt-get update && \ + apt-get install -y \ + nodejs && \ + apt remove yq jq -y + +RUN \ + echo "**** install runtime dependencies ****" && \ + apt-get install -y \ + git \ + libatomic1 \ + nano \ + net-tools \ + sudo \ + podman \ + wget \ + python3 \ + python3-pip + +RUN \ + echo "**** install code-server ****" && \ + if [ -z ${CODE_RELEASE+x} ]; then \ + CODE_RELEASE=$(curl -sX GET https://api.github.com/repos/coder/code-server/releases/latest \ + | awk '/tag_name/{print $4;exit}' FS='[""]' | sed 's|^v||'); \ + fi && \ + mkdir -p /app/code-server && \ + curl -o \ + /tmp/code-server.tar.gz -L \ + "https://github.com/coder/code-server/releases/download/v${CODE_RELEASE}/code-server-${CODE_RELEASE}-linux-amd64.tar.gz" && \ + tar xf /tmp/code-server.tar.gz -C \ + /app/code-server --strip-components=1 && \ + echo "**** patch 4.0.2 ****" && \ + if [ "${CODE_RELEASE}" = "4.0.2" ] && [ "$(uname -m)" != "x86_64" ]; then \ + cd /app/code-server && \ + npm i --production @node-rs/argon2; \ + fi && \ + echo "**** clean up ****" && \ + apt-get purge --auto-remove -y \ + build-essential \ + nodejs && \ + apt-get clean && \ + rm -rf \ + /config/* \ + /tmp/* \ + /var/lib/apt/lists/* \ + /var/tmp/* \ + /etc/apt/sources.list.d/nodesource.list + +ENV USER=jovyan \ + UID=1001 \ + GID=100 \ + HOME=/workspace \ + PATH=/opt/conda/bin:/app/code-server/bin/:$PATH:/app/code-server/ + + +# RUN \ +# echo "**** install conda ****" && \ +# wget https://repo.anaconda.com/miniconda/Miniconda3-py39_4.12.0-Linux-x86_64.sh -O miniconda.sh -q && \ +# sh miniconda.sh -b -p /opt/conda && \ +# conda install -n base -c conda-forge mamba && \ +# conda config --system --append channels conda-forge && \ +# conda config --system --append channels terradue && \ +# conda config --system --append channels eoepca && \ +# conda config --system --append channels r && \ +# conda config --system --set auto_update_conda false && \ +# conda config --system --set show_channel_urls true && \ +# conda config --system --set channel_priority "flexible" + + +# RUN \ +# mamba install -n base cwltool cwl-wrapper==0.12.2 nodejs && \ +# mamba clean -a + +RUN \ + echo "**** install aws cli ****" && \ + pip install awscli && \ + pip install awscli-plugin-endpoint + +RUN \ + echo "**** install jupyter-hub native proxy ****" && \ + pip install jhsingle-native-proxy>=0.0.9 && \ + echo "**** install bash kernel ****" && \ + pip install bash_kernel && \ + python -m bash_kernel.install + +RUN \ + echo "**** adds user jovyan ****" && \ + useradd -m -s /bin/bash -N -u $UID $USER + +COPY entrypoint.sh /opt/entrypoint.sh + +RUN chmod +x /opt/entrypoint.sh + +# RUN chown -R jovyan:100 /opt/conda + +# RUN \ +# echo "**** required by cwltool docker pull even if running with --podman ****" && \ +# ln -s /usr/bin/podman /usr/bin/docker + +ENTRYPOINT ["/opt/entrypoint.sh"] + +EXPOSE 8888 + +RUN echo "**** install kubectl ****" && \ + curl -s -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" && \ + chmod +x kubectl && \ + mkdir -p /workspace/.venv/bin && \ + mv ./kubectl /usr/local/bin/kubectl + + +RUN curl -s -L https://github.com/pypa/hatch/releases/latest/download/hatch-x86_64-unknown-linux-gnu.tar.gz | tar -xzvf - -C /usr/local/bin/ && \ + chmod +x /usr/local/bin/hatch + +RUN curl -s -L https://github.com/go-task/task/releases/download/v3.41.0/task_linux_amd64.tar.gz | tar -xzvf - -C /usr/local/bin/ && \ + chmod +x /usr/local/bin/task + +RUN curl -s -L https://storage.googleapis.com/skaffold/releases/latest/skaffold-linux-amd64 > /usr/local/bin/skaffold && \ + chmod +x /usr/local/bin/skaffold + +RUN pip3 install tomlq && tomlq --version + +RUN rm -f /usr/bin/yq && \ + rm -f /usr/local/bin/yq && \ + curl -s -LO https://github.com/mikefarah/yq/releases/download/v4.45.1/yq_linux_amd64.tar.gz && \ + tar -xvf yq_linux_amd64.tar.gz && \ + mv yq_linux_amd64 /usr/local/bin/yq && \ + cp -v /usr/local/bin/yq /usr/bin/yq && \ + chmod +x /usr/bin/yq /usr/local/bin/yq + +RUN curl -s -L https://github.com/jqlang/jq/releases/download/jq-1.7.1/jq-linux-amd64 > /usr/bin/jq && \ + chmod +x /usr/bin/jq + +RUN pip install --upgrade uv USER jovyan \ No newline at end of file diff --git a/application-package-patterns/entrypoint.sh b/application-package-patterns/entrypoint.sh new file mode 100644 index 0000000..f5b1057 --- /dev/null +++ b/application-package-patterns/entrypoint.sh @@ -0,0 +1,37 @@ +#!/bin/bash + +collect_port=0 +port="8888" +delim='=' + +for var in "$@" +do + echo "$var" + + if [ "$collect_port" == "1" ]; then + echo "Collecting external port $var" + port=$var + collect_port=0 + fi + + splitarg=${var%%$delim*} + + if [ "$splitarg" == "--port" ]; then + if [ ${#splitarg} == ${#var} ]; then + collect_port=1 + else + port=${var#*$delim} + echo "Setting external port $port" + fi + fi +done + +destport=$((port + 1)) + +echo "Using internal port $destport" + +if [ -z "$CODE_SERVER_WS" ]; then + CODE_SERVER_WS="/workspace" +fi + +jhsingle-native-proxy --port $port --destport $destport code-server {--}auth none {--}bind-addr 0.0.0.0:$destport {--}user-data-dir /workspace $CODE_SERVER_WS \ No newline at end of file diff --git a/application-package-patterns/files/init.sh b/application-package-patterns/files/init.sh index 8b92a62..2edf680 100644 --- a/application-package-patterns/files/init.sh +++ b/application-package-patterns/files/init.sh @@ -21,12 +21,14 @@ source /workspace/.venv/bin/activate /workspace/.venv/bin/python -m ipykernel install --user --name cwl_how_to_env --display-name "Python (CWL How-To's)" -curl -s -L https://github.com/go-task/task/releases/download/v3.41.0/task_linux_amd64.tar.gz | tar -xzvf - -C /workspace/.venv/bin/ -chmod +x /workspace/.venv/bin/task +cd /workspace +git clone https://github.com/EOEPCA/eoap-cwlwrap.git +cd eoap-cwlwrap +/workspace/.venv/bin/pip install --no-cache-dir -e . + -curl -s -L https://storage.googleapis.com/skaffold/releases/latest/skaffold-linux-amd64 > /workspace/.venv/bin/skaffold -chmod +x /workspace/.venv/bin/skaffold +export AWS_DEFAULT_REGION="us-east-1" +export AWS_ACCESS_KEY_ID="test" +export AWS_SECRET_ACCESS_KEY="test" -curl -s -LO https://github.com/mikefarah/yq/releases/download/v4.45.1/yq_linux_amd64.tar.gz -tar -xvf yq_linux_amd64.tar.gz -mv yq_linux_amd64 /workspace/.venv/bin/yq \ No newline at end of file +aws s3 mb s3://results --endpoint-url=http://eoap-application-package-patterns-localstack:4566 diff --git a/application-package-patterns/skaffold.yaml b/application-package-patterns/skaffold.yaml index eafe30b..1c2b4fd 100644 --- a/application-package-patterns/skaffold.yaml +++ b/application-package-patterns/skaffold.yaml @@ -35,12 +35,21 @@ deploy: calrissian.storageClassName: standard #openebs-kernel-nfs-scw calrissian.imagePullSecrets: [] skaffold.enabled: True + kaniko.enabled: true + setFiles: { initScript: ./files/init.sh, bashrcScript: ./files/bash-rc, bashloginScript: ./files/bash-login } + - name: eoap-application-package-patterns-localstack + remoteChart: localstack/localstack + namespace: eoap-application-package-patterns + createNamespace: true + setValues: + service.type: ClusterIP + portForward: - resourceType: service resourceName: code-server-service From b193cf841e798278c8147bfd934fe1277b984331 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?G=C3=A9rald=20Fenoy?= Date: Mon, 23 Jun 2025 12:47:12 +0200 Subject: [PATCH 52/61] Update ZOO-Project-DRU Helm chart to the latest released version --- ogc-api-processes-with-zoo/skaffold.yaml | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/ogc-api-processes-with-zoo/skaffold.yaml b/ogc-api-processes-with-zoo/skaffold.yaml index 71c9e5b..c657b4b 100644 --- a/ogc-api-processes-with-zoo/skaffold.yaml +++ b/ogc-api-processes-with-zoo/skaffold.yaml @@ -13,7 +13,8 @@ deploy: remoteChart: zoo-project/zoo-project-dru namespace: eoap-zoo-project createNamespace: true - version: 0.4.14 + #version: 0.4.14 + version: 0.5.0 valuesFiles: - values.yaml setValues: @@ -23,6 +24,12 @@ deploy: filter_in.enabled: true filter_out.enabled: true persistence.procServicesAccessMode: ReadWriteMany + websocketd.enabled: true + websocketd.port: 8888 + websocketd.image.repository: zooproject/websocketd + websocketd.image.tag: 67449315857b54bbc970f02c7aa4fd10a94721f0 + websocketd.image.pullPolicy: IfNotPresent + redis.enabled: true workflow.additionalInputs: s3_bucket: results region_name: us-east-1 @@ -95,7 +102,12 @@ portForward: namespace: eoap-zoo-project address: localhost port: 8080 - localPort: 8000 + localPort: 8000 + - resourceType: service + resourceName: zoo-project-dru-websocketd + namespace: eoap-zoo-project + port: 8888 + localPort: 8888 - resourceType: service resourceName: zoo-project-dru-service namespace: eoap-zoo-project From 66884c2f0883e80cf0ccf19889ef5fa0e1084495 Mon Sep 17 00:00:00 2001 From: Fabrice Brito Date: Thu, 3 Jul 2025 15:22:37 +0200 Subject: [PATCH 53/61] fixes and CI --- .github/workflows/build.yaml | 21 ++++++++++++++-- advanced-tooling/files/init.sh | 5 ++-- advanced-tooling/skaffold.yaml | 6 ++--- application-package-patterns/files/bash-rc | 2 +- application-package-patterns/files/init.sh | 10 ++++++-- dask-gateway/files/init.sh | 28 ++++------------------ 6 files changed, 39 insertions(+), 33 deletions(-) diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index b76d08d..d9ab7ab 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -22,7 +22,7 @@ jobs: strategy: matrix: - step: ["advanced-tooling", "dask-gateway", "how-to"] + step: ["application-package-patterns", "advanced-tooling", "dask-gateway", "how-to"] steps: - uses: actions/checkout@v2 @@ -35,4 +35,21 @@ jobs: docker tag ${{ matrix.step }} $IMAGE_ID:${version} docker push $IMAGE_ID:${version} - \ No newline at end of file + container-dask: + + runs-on: ubuntu-latest + + strategy: + matrix: + step: ["client", "coder", "worker"] + + steps: + - uses: actions/checkout@v2 + - run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin + - name: build & push image + run: | + version=$(cat dask-gateway/codemeta.json | jq -r .version ) + IMAGE_ID=ghcr.io/eoap/dev-platform-eoap/dask-gateway-${{ matrix.step }} + docker build dask-gateway --file dask-gateway/Dockerfile.${{ matrix.step }} --tag eoap-dask-gateway-${{ matrix.step }} + docker tag eoap-dask-gateway-${{ matrix.step }} $IMAGE_ID:${version} + docker push $IMAGE_ID:${version} \ No newline at end of file diff --git a/advanced-tooling/files/init.sh b/advanced-tooling/files/init.sh index 9f49192..4775bde 100644 --- a/advanced-tooling/files/init.sh +++ b/advanced-tooling/files/init.sh @@ -19,8 +19,9 @@ echo '{"workbench.colorTheme": "Visual Studio Dark"}' > /workspace/User/settings python -m venv /workspace/.venv source /workspace/.venv/bin/activate -/workspace/.venv/bin/python -m pip install --no-cache-dir tomlq calrissian -/workspace/.venv/bin/python -m ipykernel install --user --name advanced_tooling --display-name "Python (Advanced Tooling)" +/workspace/.venv/bin/pip install --upgrade uv +/workspace/.venv/bin/python -m uv pip install --no-cache-dir tomlq calrissian +/workspace/.venv/bin/python -m ipykernel install --user --name advanced_tooling --display-name "Python (Application Package Patterns)" export AWS_DEFAULT_REGION="us-east-1" diff --git a/advanced-tooling/skaffold.yaml b/advanced-tooling/skaffold.yaml index 22c3433..5b23f4b 100644 --- a/advanced-tooling/skaffold.yaml +++ b/advanced-tooling/skaffold.yaml @@ -22,9 +22,9 @@ deploy: namespace: eoap-advanced-tooling createNamespace: true setValues: - coder.coderImage: ttl.sh/eoap-dask-gateway-coder:2h + coder.coderImage: ttl.sh/eoap-advanced-tooling-coder:2h coder.workspace: advanced-tooling - coder.storageClassName: openebs-kernel-nfs-scw + coder.storageClassName: standard #openebs-kernel-nfs-scw coder.workspaceStorage: 10Gi coderResources.limits.cpu: '2' coderResources.limits.memory: '6442450944' @@ -32,7 +32,7 @@ deploy: coderResources.requests.memory: '4294967296' coder.securityContext.privileged: false calrissian.enabled: true - calrissian.storageClassName: openebs-kernel-nfs-scw + calrissian.storageClassName: standard #openebs-kernel-nfs-scw skaffold.enabled: true kaniko.enabled: true diff --git a/application-package-patterns/files/bash-rc b/application-package-patterns/files/bash-rc index 00102c4..a13eb1d 100644 --- a/application-package-patterns/files/bash-rc +++ b/application-package-patterns/files/bash-rc @@ -1,6 +1,6 @@ alias ll="ls -l" -alias aws="aws --endpoint-url=http://eoap-how-to-localstack:4566" +alias aws="aws --endpoint-url=http://eoap-application-package-patterns-localstack:4566" export PATH=/workspace/.venv/bin:$PATH export TASK_X_REMOTE_TASKFILES=1 diff --git a/application-package-patterns/files/init.sh b/application-package-patterns/files/init.sh index 2edf680..941ff94 100644 --- a/application-package-patterns/files/init.sh +++ b/application-package-patterns/files/init.sh @@ -17,9 +17,10 @@ echo '{"workbench.colorTheme": "Visual Studio Dark"}' > /workspace/User/settings python -m venv /workspace/.venv source /workspace/.venv/bin/activate -/workspace/.venv/bin/python -m pip install --no-cache-dir stactools rasterio requests stac-asset click-logging tabulate tqdm pystac-client ipykernel loguru scikit-image rio_stac calrissian tomlq +/workspace/.venv/bin/pip install --upgrade uv +/workspace/.venv/bin/python -m uv pip install --no-cache-dir stactools calrissian tomlq -/workspace/.venv/bin/python -m ipykernel install --user --name cwl_how_to_env --display-name "Python (CWL How-To's)" +/workspace/.venv/bin/python -m ipykernel install --user --name cwl_how_to_env --display-name "Python (Application Package patterns)" cd /workspace git clone https://github.com/EOEPCA/eoap-cwlwrap.git @@ -32,3 +33,8 @@ export AWS_ACCESS_KEY_ID="test" export AWS_SECRET_ACCESS_KEY="test" aws s3 mb s3://results --endpoint-url=http://eoap-application-package-patterns-localstack:4566 + +rm -f /workspace/.venv/bin/yq + +# for calrissian (eoap-cwlwrap uses a recent version of cwltool) +/workspace/.venv/bin/python -m uv pip install cwltool==3.1.20240708091337 \ No newline at end of file diff --git a/dask-gateway/files/init.sh b/dask-gateway/files/init.sh index 84c40d0..3d30b9b 100644 --- a/dask-gateway/files/init.sh +++ b/dask-gateway/files/init.sh @@ -2,15 +2,13 @@ set -x -export PATH=/workspace/.local/bin:/workspace/.venv/bin:$PATH - cd /workspace git clone 'https://github.com/eoap/dask-app-package.git' code-server --install-extension ms-python.python -code-server --install-extension redhat.vscode-yaml -code-server --install-extension sbg-rabix.benten-cwl +#code-server --install-extension redhat.vscode-yaml +#code-server --install-extension sbg-rabix.benten-cwl code-server --install-extension ms-toolsai.jupyter code-server --install-extension tamasfe.even-better-toml @@ -20,11 +18,12 @@ mkdir -p /workspace/User/ echo '{"workbench.colorTheme": "Visual Studio Dark"}' > /workspace/User/settings.json -pip install git+https://github.com/Terradue/calrissian@dask-gateway tomlq # calrissian and tomlq (for task files) +pip install --no-warn-script-location git+https://github.com/Terradue/calrissian@dask-gateway # calrissian python -m venv /workspace/.venv source /workspace/.venv/bin/activate -/workspace/.venv/bin/python -m pip install --no-cache-dir numpy==1.26.3 dask==2024.1.0 distributed==2024.1.0 dask-gateway==2024.1.0 pystac bokeh rioxarray==0.18.1 loguru==0.7.3 odc-stac[botocore]==0.3.10 stackstac==0.5.1 pystac-client planetary-computer xarray-spatial "xarray[complete]" ipykernel rio-color +/workspace/.venv/bin/pip install --upgrade uv +/workspace/.venv/bin/python -m uv pip install --no-cache-dir numpy==1.26.3 dask==2025.4.0 distributed==2025.4.0 dask-gateway==2025.4.0 pystac bokeh rioxarray==0.18.1 loguru==0.7.3 odc-stac[botocore]==0.3.10 stackstac==0.5.1 pystac-client==0.8.6 planetary-computer==1.0.0 xarray-spatial==0.4.0 "xarray[complete]==2025.4.0" ipykernel rio-color==2.0.1 res=$? if [ $res -ne 0 ]; then echo "Failed to install packages" @@ -33,29 +32,12 @@ fi /workspace/.venv/bin/python -m ipykernel install --user --name dask-gateway --display-name "Python (Dask Application Package)" -echo "**** install kubectl ****" -curl -s -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" -chmod +x kubectl -mv ./kubectl /workspace/.venv/bin/kubectl export AWS_DEFAULT_REGION="us-east-1" - export AWS_ACCESS_KEY_ID="test" - export AWS_SECRET_ACCESS_KEY="test" aws s3 mb s3://results --endpoint-url=http://eoap-dask-gateway-localstack:4566 -curl -s -L https://github.com/pypa/hatch/releases/latest/download/hatch-x86_64-unknown-linux-gnu.tar.gz | tar -xzvf - -C /workspace/.venv/bin/ -chmod +x /workspace/.venv/bin/hatch - -curl -s -L https://github.com/go-task/task/releases/download/v3.41.0/task_linux_amd64.tar.gz | tar -xzvf - -C /workspace/.venv/bin/ -chmod +x /workspace/.venv/bin/task - -curl -s -LO https://github.com/mikefarah/yq/releases/download/v4.45.1/yq_linux_amd64.tar.gz -tar -xvf yq_linux_amd64.tar.gz -mv yq_linux_amd64 /workspace/.venv/bin/yq -curl -s -L https://storage.googleapis.com/skaffold/releases/latest/skaffold-linux-amd64 > /workspace/.venv/bin/skaffold -chmod +x /workspace/.venv/bin/skaffold From cd8588a3a15bd036047c8bb579b5f919faa85e37 Mon Sep 17 00:00:00 2001 From: Fabrice Brito Date: Thu, 3 Jul 2025 15:22:54 +0200 Subject: [PATCH 54/61] dockerfile --- application-package-patterns/Dockerfile.coder | 28 +------------------ 1 file changed, 1 insertion(+), 27 deletions(-) diff --git a/application-package-patterns/Dockerfile.coder b/application-package-patterns/Dockerfile.coder index f04c9a8..0462763 100644 --- a/application-package-patterns/Dockerfile.coder +++ b/application-package-patterns/Dockerfile.coder @@ -53,8 +53,7 @@ RUN \ fi && \ echo "**** clean up ****" && \ apt-get purge --auto-remove -y \ - build-essential \ - nodejs && \ + build-essential && \ apt-get clean && \ rm -rf \ /config/* \ @@ -69,25 +68,6 @@ ENV USER=jovyan \ HOME=/workspace \ PATH=/opt/conda/bin:/app/code-server/bin/:$PATH:/app/code-server/ - -# RUN \ -# echo "**** install conda ****" && \ -# wget https://repo.anaconda.com/miniconda/Miniconda3-py39_4.12.0-Linux-x86_64.sh -O miniconda.sh -q && \ -# sh miniconda.sh -b -p /opt/conda && \ -# conda install -n base -c conda-forge mamba && \ -# conda config --system --append channels conda-forge && \ -# conda config --system --append channels terradue && \ -# conda config --system --append channels eoepca && \ -# conda config --system --append channels r && \ -# conda config --system --set auto_update_conda false && \ -# conda config --system --set show_channel_urls true && \ -# conda config --system --set channel_priority "flexible" - - -# RUN \ -# mamba install -n base cwltool cwl-wrapper==0.12.2 nodejs && \ -# mamba clean -a - RUN \ echo "**** install aws cli ****" && \ pip install awscli && \ @@ -108,12 +88,6 @@ COPY entrypoint.sh /opt/entrypoint.sh RUN chmod +x /opt/entrypoint.sh -# RUN chown -R jovyan:100 /opt/conda - -# RUN \ -# echo "**** required by cwltool docker pull even if running with --podman ****" && \ -# ln -s /usr/bin/podman /usr/bin/docker - ENTRYPOINT ["/opt/entrypoint.sh"] EXPOSE 8888 From 6f1d589616687657f6f2502afe025b80821d3238 Mon Sep 17 00:00:00 2001 From: Fabrice Brito Date: Wed, 17 Sep 2025 15:06:34 +0200 Subject: [PATCH 55/61] adds coder container --- .github/workflows/build.yaml | 19 ++++- container/coder/Dockerfile.coder | 126 +++++++++++++++++++++++++++++++ container/coder/codemeta.json | 6 ++ 3 files changed, 150 insertions(+), 1 deletion(-) create mode 100644 container/coder/Dockerfile.coder create mode 100644 container/coder/codemeta.json diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index d9ab7ab..5e34ac9 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -11,9 +11,26 @@ on: - 'advanced-tooling/**' - 'dask-gateway/**' - 'how-to/**' + - 'container/**' jobs: + container-coder: + + runs-on: ubuntu-latest + + steps: + - uses: actions/checkout@v2 + - run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin + - name: build & push image + run: | + cd container/coder + version=$(cat codemeta.json | jq -r .version ) + IMAGE_ID=ghcr.io/eoap/dev-platform-eoap/eoap-coder + docker build . --file Dockerfile.coder --tag eoap-coder + docker tag eoap-coder $IMAGE_ID:${version} + docker push $IMAGE_ID:${version} + container-build: #needs: [] @@ -22,7 +39,7 @@ jobs: strategy: matrix: - step: ["application-package-patterns", "advanced-tooling", "dask-gateway", "how-to"] + step: ["application-package-patterns", "advanced-tooling", "how-to"] steps: - uses: actions/checkout@v2 diff --git a/container/coder/Dockerfile.coder b/container/coder/Dockerfile.coder new file mode 100644 index 0000000..0462763 --- /dev/null +++ b/container/coder/Dockerfile.coder @@ -0,0 +1,126 @@ +FROM docker.io/library/python:3.10.12@sha256:bac3a0e0d16125977e351c861e2f4b12ecafeaa6f72431dc970d0b9155103232 + +RUN \ + apt-get update && \ + apt-get install -y \ + build-essential \ + curl \ + gcc \ + vim \ + tree \ + file + +RUN \ + echo "**** install node repo ****" && \ + curl -s https://deb.nodesource.com/gpgkey/nodesource.gpg.key | apt-key add - && \ + echo 'deb https://deb.nodesource.com/node_14.x jammy main' \ + > /etc/apt/sources.list.d/nodesource.list && \ + echo "**** install build dependencies ****" && \ + apt-get update && \ + apt-get install -y \ + nodejs && \ + apt remove yq jq -y + +RUN \ + echo "**** install runtime dependencies ****" && \ + apt-get install -y \ + git \ + libatomic1 \ + nano \ + net-tools \ + sudo \ + podman \ + wget \ + python3 \ + python3-pip + +RUN \ + echo "**** install code-server ****" && \ + if [ -z ${CODE_RELEASE+x} ]; then \ + CODE_RELEASE=$(curl -sX GET https://api.github.com/repos/coder/code-server/releases/latest \ + | awk '/tag_name/{print $4;exit}' FS='[""]' | sed 's|^v||'); \ + fi && \ + mkdir -p /app/code-server && \ + curl -o \ + /tmp/code-server.tar.gz -L \ + "https://github.com/coder/code-server/releases/download/v${CODE_RELEASE}/code-server-${CODE_RELEASE}-linux-amd64.tar.gz" && \ + tar xf /tmp/code-server.tar.gz -C \ + /app/code-server --strip-components=1 && \ + echo "**** patch 4.0.2 ****" && \ + if [ "${CODE_RELEASE}" = "4.0.2" ] && [ "$(uname -m)" != "x86_64" ]; then \ + cd /app/code-server && \ + npm i --production @node-rs/argon2; \ + fi && \ + echo "**** clean up ****" && \ + apt-get purge --auto-remove -y \ + build-essential && \ + apt-get clean && \ + rm -rf \ + /config/* \ + /tmp/* \ + /var/lib/apt/lists/* \ + /var/tmp/* \ + /etc/apt/sources.list.d/nodesource.list + +ENV USER=jovyan \ + UID=1001 \ + GID=100 \ + HOME=/workspace \ + PATH=/opt/conda/bin:/app/code-server/bin/:$PATH:/app/code-server/ + +RUN \ + echo "**** install aws cli ****" && \ + pip install awscli && \ + pip install awscli-plugin-endpoint + +RUN \ + echo "**** install jupyter-hub native proxy ****" && \ + pip install jhsingle-native-proxy>=0.0.9 && \ + echo "**** install bash kernel ****" && \ + pip install bash_kernel && \ + python -m bash_kernel.install + +RUN \ + echo "**** adds user jovyan ****" && \ + useradd -m -s /bin/bash -N -u $UID $USER + +COPY entrypoint.sh /opt/entrypoint.sh + +RUN chmod +x /opt/entrypoint.sh + +ENTRYPOINT ["/opt/entrypoint.sh"] + +EXPOSE 8888 + +RUN echo "**** install kubectl ****" && \ + curl -s -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" && \ + chmod +x kubectl && \ + mkdir -p /workspace/.venv/bin && \ + mv ./kubectl /usr/local/bin/kubectl + + +RUN curl -s -L https://github.com/pypa/hatch/releases/latest/download/hatch-x86_64-unknown-linux-gnu.tar.gz | tar -xzvf - -C /usr/local/bin/ && \ + chmod +x /usr/local/bin/hatch + +RUN curl -s -L https://github.com/go-task/task/releases/download/v3.41.0/task_linux_amd64.tar.gz | tar -xzvf - -C /usr/local/bin/ && \ + chmod +x /usr/local/bin/task + +RUN curl -s -L https://storage.googleapis.com/skaffold/releases/latest/skaffold-linux-amd64 > /usr/local/bin/skaffold && \ + chmod +x /usr/local/bin/skaffold + +RUN pip3 install tomlq && tomlq --version + +RUN rm -f /usr/bin/yq && \ + rm -f /usr/local/bin/yq && \ + curl -s -LO https://github.com/mikefarah/yq/releases/download/v4.45.1/yq_linux_amd64.tar.gz && \ + tar -xvf yq_linux_amd64.tar.gz && \ + mv yq_linux_amd64 /usr/local/bin/yq && \ + cp -v /usr/local/bin/yq /usr/bin/yq && \ + chmod +x /usr/bin/yq /usr/local/bin/yq + +RUN curl -s -L https://github.com/jqlang/jq/releases/download/jq-1.7.1/jq-linux-amd64 > /usr/bin/jq && \ + chmod +x /usr/bin/jq + +RUN pip install --upgrade uv + +USER jovyan \ No newline at end of file diff --git a/container/coder/codemeta.json b/container/coder/codemeta.json new file mode 100644 index 0000000..b107aae --- /dev/null +++ b/container/coder/codemeta.json @@ -0,0 +1,6 @@ +{ + "@context": "https://doi.org/10.5063/schema/codemeta-2.0", + "type": "SoftwareSourceCode", + "name": "container-coder", + "version": "0.1.0" +} \ No newline at end of file From 0b9589345251dd4347a2a82f235c01a301df1326 Mon Sep 17 00:00:00 2001 From: Fabrice Brito Date: Wed, 17 Sep 2025 15:18:35 +0200 Subject: [PATCH 56/61] docker-coder - update --- container/coder/Dockerfile.coder | 6 ++---- container/coder/entrypoint.sh | 37 ++++++++++++++++++++++++++++++++ 2 files changed, 39 insertions(+), 4 deletions(-) create mode 100644 container/coder/entrypoint.sh diff --git a/container/coder/Dockerfile.coder b/container/coder/Dockerfile.coder index 0462763..4346cbd 100644 --- a/container/coder/Dockerfile.coder +++ b/container/coder/Dockerfile.coder @@ -1,4 +1,4 @@ -FROM docker.io/library/python:3.10.12@sha256:bac3a0e0d16125977e351c861e2f4b12ecafeaa6f72431dc970d0b9155103232 +FROM docker.io/library/python:3.12.11-bookworm@sha256:bea386df48d7ee07eed0a1f3e6f9d5c0292c228b8d8ed2ea738b7a57b29c4470 RUN \ apt-get update && \ @@ -30,9 +30,7 @@ RUN \ net-tools \ sudo \ podman \ - wget \ - python3 \ - python3-pip + wget RUN \ echo "**** install code-server ****" && \ diff --git a/container/coder/entrypoint.sh b/container/coder/entrypoint.sh new file mode 100644 index 0000000..f5b1057 --- /dev/null +++ b/container/coder/entrypoint.sh @@ -0,0 +1,37 @@ +#!/bin/bash + +collect_port=0 +port="8888" +delim='=' + +for var in "$@" +do + echo "$var" + + if [ "$collect_port" == "1" ]; then + echo "Collecting external port $var" + port=$var + collect_port=0 + fi + + splitarg=${var%%$delim*} + + if [ "$splitarg" == "--port" ]; then + if [ ${#splitarg} == ${#var} ]; then + collect_port=1 + else + port=${var#*$delim} + echo "Setting external port $port" + fi + fi +done + +destport=$((port + 1)) + +echo "Using internal port $destport" + +if [ -z "$CODE_SERVER_WS" ]; then + CODE_SERVER_WS="/workspace" +fi + +jhsingle-native-proxy --port $port --destport $destport code-server {--}auth none {--}bind-addr 0.0.0.0:$destport {--}user-data-dir /workspace $CODE_SERVER_WS \ No newline at end of file From 4bfbe841fb063f5a2d550e99220fec11de89c3b0 Mon Sep 17 00:00:00 2001 From: Fabrice Brito Date: Wed, 17 Sep 2025 15:46:30 +0200 Subject: [PATCH 57/61] updates container to python 3.12.11 --- container/coder/Dockerfile.coder | 2 +- dask-gateway/Dockerfile.client | 3 ++- dask-gateway/Dockerfile.coder | 8 +++----- dask-gateway/Dockerfile.worker | 2 +- 4 files changed, 7 insertions(+), 8 deletions(-) diff --git a/container/coder/Dockerfile.coder b/container/coder/Dockerfile.coder index 4346cbd..70d37bd 100644 --- a/container/coder/Dockerfile.coder +++ b/container/coder/Dockerfile.coder @@ -13,7 +13,7 @@ RUN \ RUN \ echo "**** install node repo ****" && \ curl -s https://deb.nodesource.com/gpgkey/nodesource.gpg.key | apt-key add - && \ - echo 'deb https://deb.nodesource.com/node_14.x jammy main' \ + echo 'deb https://deb.nodesource.com/node_14.x bookworm main' \ > /etc/apt/sources.list.d/nodesource.list && \ echo "**** install build dependencies ****" && \ apt-get update && \ diff --git a/dask-gateway/Dockerfile.client b/dask-gateway/Dockerfile.client index dddaead..91c0ef3 100644 --- a/dask-gateway/Dockerfile.client +++ b/dask-gateway/Dockerfile.client @@ -1,4 +1,5 @@ -FROM docker.io/library/python:3.10.12@sha256:bac3a0e0d16125977e351c861e2f4b12ecafeaa6f72431dc970d0b9155103232 +FROM docker.io/library/python:3.12.11-bookworm@sha256:bea386df48d7ee07eed0a1f3e6f9d5c0292c228b8d8ed2ea738b7a57b29c4470 + # Install necessary packages RUN apt-get update && apt-get install -y --no-install-recommends \ diff --git a/dask-gateway/Dockerfile.coder b/dask-gateway/Dockerfile.coder index f04c9a8..791277f 100644 --- a/dask-gateway/Dockerfile.coder +++ b/dask-gateway/Dockerfile.coder @@ -1,4 +1,4 @@ -FROM docker.io/library/python:3.10.12@sha256:bac3a0e0d16125977e351c861e2f4b12ecafeaa6f72431dc970d0b9155103232 +FROM docker.io/library/python:3.12.11-bookworm@sha256:bea386df48d7ee07eed0a1f3e6f9d5c0292c228b8d8ed2ea738b7a57b29c4470 RUN \ apt-get update && \ @@ -13,7 +13,7 @@ RUN \ RUN \ echo "**** install node repo ****" && \ curl -s https://deb.nodesource.com/gpgkey/nodesource.gpg.key | apt-key add - && \ - echo 'deb https://deb.nodesource.com/node_14.x jammy main' \ + echo 'deb https://deb.nodesource.com/node_14.x bookworm main' \ > /etc/apt/sources.list.d/nodesource.list && \ echo "**** install build dependencies ****" && \ apt-get update && \ @@ -30,9 +30,7 @@ RUN \ net-tools \ sudo \ podman \ - wget \ - python3 \ - python3-pip + wget RUN \ echo "**** install code-server ****" && \ diff --git a/dask-gateway/Dockerfile.worker b/dask-gateway/Dockerfile.worker index 72b95d8..7ae8f0a 100644 --- a/dask-gateway/Dockerfile.worker +++ b/dask-gateway/Dockerfile.worker @@ -1,4 +1,4 @@ -FROM docker.io/library/python:3.10.12@sha256:bac3a0e0d16125977e351c861e2f4b12ecafeaa6f72431dc970d0b9155103232 +FROM docker.io/library/python:3.12.11-bookworm@sha256:bea386df48d7ee07eed0a1f3e6f9d5c0292c228b8d8ed2ea738b7a57b29c4470 ADD requirements.txt /tmp/requirements.txt From 42a0be9fc84798fd183683bcda33780f4a451610 Mon Sep 17 00:00:00 2001 From: Fabrice Brito Date: Wed, 17 Sep 2025 16:54:45 +0200 Subject: [PATCH 58/61] fixes for updated zarr-cloud-format --- zarr-cloud-native-format/files/init.sh | 2 +- zarr-cloud-native-format/skaffold.yaml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/zarr-cloud-native-format/files/init.sh b/zarr-cloud-native-format/files/init.sh index d87d282..3da6f5d 100644 --- a/zarr-cloud-native-format/files/init.sh +++ b/zarr-cloud-native-format/files/init.sh @@ -17,6 +17,6 @@ echo '{"workbench.colorTheme": "Visual Studio Dark"}' > /workspace/User/settings python -m venv /workspace/.venv source /workspace/.venv/bin/activate -/workspace/.venv/bin/python -m pip install --no-cache-dir odc-stac ipykernel cwltool zarr matplotlib +/workspace/.venv/bin/python -m pip install --no-cache-dir "odc-stac" ipykernel cwltool zarr matplotlib loguru click "graphviz" "pillow" "cwl-loader>=0.5.0" "eoap-cwlwrap" "plantuml" "cwl2puml>=0.8.0" stactools[validate] rioxarray /workspace/.venv/bin/python -m ipykernel install --user --name cwl_eoap_env --display-name "Python (EOAP)" \ No newline at end of file diff --git a/zarr-cloud-native-format/skaffold.yaml b/zarr-cloud-native-format/skaffold.yaml index c02141e..fc8aa7d 100644 --- a/zarr-cloud-native-format/skaffold.yaml +++ b/zarr-cloud-native-format/skaffold.yaml @@ -11,7 +11,7 @@ deploy: namespace: zarr-cloud-native-format createNamespace: true setValues: - coder.coderImage: eoepca/pde-code-server:1.0.0 + coder.coderImage: ghcr.io/eoap/dev-platform-eoap/eoap-coder:0.1.0 coder.workspace: zarr-cloud-native-format coderstorageClassName: standard coder.workspaceStorage: 10Gi From c4862488d782d4b5c3fb71410dacae98569f418f Mon Sep 17 00:00:00 2001 From: Fabrice Brito Date: Wed, 17 Sep 2025 17:14:55 +0200 Subject: [PATCH 59/61] adds graphviz to coder image --- container/coder/Dockerfile.coder | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/container/coder/Dockerfile.coder b/container/coder/Dockerfile.coder index 70d37bd..be43270 100644 --- a/container/coder/Dockerfile.coder +++ b/container/coder/Dockerfile.coder @@ -30,7 +30,8 @@ RUN \ net-tools \ sudo \ podman \ - wget + wget \ + graphviz RUN \ echo "**** install code-server ****" && \ From a867cd8a2ff9fad76dd30691f2a3cc34f825be3a Mon Sep 17 00:00:00 2001 From: Fabrice Brito Date: Mon, 13 Oct 2025 14:10:34 +0200 Subject: [PATCH 60/61] wip fixes on dask gateway --- dask-gateway/files/init.sh | 31 ++++++++++++++++++++++++++----- dask-gateway/skaffold.yaml | 28 +++++----------------------- 2 files changed, 31 insertions(+), 28 deletions(-) diff --git a/dask-gateway/files/init.sh b/dask-gateway/files/init.sh index 3d30b9b..bfd9bed 100644 --- a/dask-gateway/files/init.sh +++ b/dask-gateway/files/init.sh @@ -20,17 +20,38 @@ echo '{"workbench.colorTheme": "Visual Studio Dark"}' > /workspace/User/settings pip install --no-warn-script-location git+https://github.com/Terradue/calrissian@dask-gateway # calrissian -python -m venv /workspace/.venv -source /workspace/.venv/bin/activate -/workspace/.venv/bin/pip install --upgrade uv -/workspace/.venv/bin/python -m uv pip install --no-cache-dir numpy==1.26.3 dask==2025.4.0 distributed==2025.4.0 dask-gateway==2025.4.0 pystac bokeh rioxarray==0.18.1 loguru==0.7.3 odc-stac[botocore]==0.3.10 stackstac==0.5.1 pystac-client==0.8.6 planetary-computer==1.0.0 xarray-spatial==0.4.0 "xarray[complete]==2025.4.0" ipykernel rio-color==2.0.1 +# eopf-sentinel-2 environment +python -m venv /workspace/.venv-eopf +source /workspace/.venv-eopf/bin/activate +/workspace/.venv-eopf/bin/pip install --upgrade uv +/workspace/.venv-eopf/bin/python -m uv pip install --no-cache-dir numpy==1.26.3 dask==2025.4.0 distributed==2025.4.0 dask-gateway==2025.4.0 pystac bokeh rioxarray==0.18.1 loguru==0.7.3 odc-stac[botocore]==0.3.10 stackstac==0.5.1 pystac-client==0.8.6 planetary-computer==1.0.0 xarray-spatial==0.4.0 "xarray[complete]==2025.4.0" ipykernel rio-color==2.0.1 res=$? if [ $res -ne 0 ]; then echo "Failed to install packages" exit $res fi -/workspace/.venv/bin/python -m ipykernel install --user --name dask-gateway --display-name "Python (Dask Application Package)" +cd /workspace/dask-app-package/eopf-sentinel-2 +/workspace/.venv-eopf/bin/python -m uv pip install --no-cache-dir -e . + +/workspace/.venv-eopf/bin/python -m ipykernel install --user --name dask-gateway --display-name "Python (EOPF Dask Application Package)" + +# cloudless-mosaic environment +python -m venv /workspace/.venv-cloudless-mosaic +source /workspace/.venv-cloudless-mosaic/bin/activate + +/workspace/.venv-cloudless-mosaic/bin/pip install --upgrade uv +/workspace/.venv-cloudless-mosaic/bin/python -m uv pip install --no-cache-dir numpy==1.26.3 dask==2025.4.0 distributed==2025.4.0 dask-gateway==2025.4.0 pystac bokeh rioxarray==0.18.1 loguru==0.7.3 odc-stac[botocore]==0.3.10 stackstac==0.5.1 pystac-client==0.8.6 planetary-computer==1.0.0 xarray-spatial==0.4.0 "xarray[complete]==2025.4.0" ipykernel rio-color==2.0.1 +res=$? +if [ $res -ne 0 ]; then + echo "Failed to install packages" + exit $res +fi + +cd /workspace/dask-app-package/cloudless-mosaic +/workspace/.venv-cloudless-mosaic/bin/python -m uv pip install --no-cache-dir -e . + +/workspace/.venv-cloudless-mosaic/bin/python -m ipykernel install --user --name dask-gateway --display-name "Python (Cloudless Mosaic Dask Application Package)" export AWS_DEFAULT_REGION="us-east-1" diff --git a/dask-gateway/skaffold.yaml b/dask-gateway/skaffold.yaml index 7773e08..d8a0394 100644 --- a/dask-gateway/skaffold.yaml +++ b/dask-gateway/skaffold.yaml @@ -2,24 +2,6 @@ apiVersion: skaffold/v4beta9 kind: Config metadata: name: eoap-dask-gateway -build: - tagPolicy: - envTemplate: - template: 2h - - artifacts: - - image: ttl.sh/eoap-dask-gateway-worker - context: . - docker: - dockerfile: Dockerfile.worker - - image: ttl.sh/eoap-dask-gateway-daskclient - context: . - docker: - dockerfile: Dockerfile.client - - image: ttl.sh/eoap-dask-gateway-coder - context: . - docker: - dockerfile: Dockerfile.coder deploy: statusCheckDeadlineSeconds: 2400 @@ -33,7 +15,7 @@ deploy: valuesFiles: - values.yaml setValueTemplates: - gateway.backend.image.name: ttl.sh/eoap-dask-gateway-worker + gateway.backend.image.name: ghcr.io/eoap/dev-platform-eoap/dask-gateway-worker:0.1.0 gateway.backend.image.tag: 2h gateway.backend.imagePullSecrets: - name: kaniko-secret @@ -44,18 +26,18 @@ deploy: namespace: eoap-dask-gateway createNamespace: true setValueTemplates: - coder.coderImage: ttl.sh/eoap-dask-gateway-coder - daskGateway.image: ttl.sh/eoap-dask-gateway-worker:2h + coder.coderImage: ghcr.io/eoap/dev-platform-eoap/dask-gateway-coder:0.1.0 + daskGateway.image: ghcr.io/eoap/dev-platform-eoap/dask-gateway-worker:0.1.0 setValues: coder.workspace: dask-app-package - coder.storageClassName: openebs-kernel-nfs-scw + coder.storageClassName: standard coder.workspaceStorage: 10Gi coderResources.limits.cpu: '2' coderResources.limits.memory: '6442450944' coderResources.requests.cpu: '1' coderResources.requests.memory: '4294967296' calrissian.enabled: true - calrissian.storageClassName: openebs-kernel-nfs-scw + calrissian.storageClassName: standard skaffold.enabled: true daskGateway.enabled: true daskGateway.daskGatewayUrl: "http://traefik-dask-gateway.eoap-dask-gateway.svc.cluster.local:80" From 8b9b23205a8fcc46e62a429521ac87d12375ac54 Mon Sep 17 00:00:00 2001 From: Fabrice Brito Date: Mon, 13 Oct 2025 15:56:14 +0200 Subject: [PATCH 61/61] bump dask-gateway version --- dask-gateway/codemeta.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dask-gateway/codemeta.json b/dask-gateway/codemeta.json index dfba51c..6455fc9 100644 --- a/dask-gateway/codemeta.json +++ b/dask-gateway/codemeta.json @@ -1,3 +1,3 @@ { - "version": "0.1.0" + "version": "0.2.0" } \ No newline at end of file