SCALRCORE-32047 Add Amazon EFS file system support

Scalr · Sep 20, 2024 · 1aef9ec · 1aef9ec
1 parent ce3c2f0
commit 1aef9ec
Show file tree

Hide file tree

Showing 9 changed files with 132 additions and 4 deletions.
diff --git a/charts/agent-docker/README.md b/charts/agent-docker/README.md
@@ -59,4 +59,4 @@ Multiple Deployments can be created within a single Kubernetes cluster.
 | tolerations | list | `[]` | Tolerations for the Scalr Agent pods, allowing them to run on tainted nodes |
 
 ----------------------------------------------
-Autogenerated from chart metadata using [helm-docs v1.11.0](https://github.com/norwoodj/helm-docs/releases/v1.11.0)
+Autogenerated from chart metadata using [helm-docs v1.14.2](https://github.com/norwoodj/helm-docs/releases/v1.14.2)
diff --git a/charts/agent-k8s/README.md b/charts/agent-k8s/README.md
@@ -86,6 +86,25 @@ $ helm upgrade ... \
     --set agent.data_home="/var/lib/{unique-name}"
 ```
 
+## Amazon EFS
+
+Amazon EFS can be used as a shared ReadWriteMany volume instead of a node disk. To configure it,
+install the `Amazon EFS CSI Driver` via an add-on. See the documentation: https://docs.aws.amazon.com/eks/latest/userguide/efs-csi.html#efs-install-driver.
+Ensure the add-on is active before proceeding.
+
+Next, configure the Amazon EFS file system ID using the `efsVolumeHandle` option:
+
+```console
+$ helm upgrade ... \
+    --set efsVolumeHandle="fs-582a03f3"
+    # Alternatively, if using an Access Point:
+    # see: https://docs.aws.amazon.com/efs/latest/ug/accessing-fs-nfs-permissions.html#accessing-fs-nfs-permissions-access-points
+    --set efsVolumeHandle="fs-582a03f3::fsap-01e050b7d9a3109d5"
+```
+
+When using EFS, all workers will operate within the same disk namespace, while the controller
+will continue to use an ephemeral directory as its data home.
+
 ## Maintainers
 
 | Name | Email | Url |
@@ -119,17 +138,20 @@ $ helm upgrade ... \
 | agent.worker_on_stop_action | string | `"drain"` | Defines the SIGTERM/SIGHUP/SIGINT signal handler's shutdown behavior. Options: "drain" or "grace-shutdown" or "force-shutdown". |
 | controllerNodeSelector | object | `{}` | Kubernetes Node Selector for assigning controller agent to specific node in the cluster. Example: `--set controllerNodeSelector."cloud\\.google\\.com\\/gke-nodepool"="scalr-agent-controller-pool"` |
 | controllerTolerations | list | `[]` | Kubernetes Node Selector for assigning worker agents and scheduling agent tasks to specific nodes in the cluster. The selector must match a node's labels for the pod to be scheduled on that node. Expects input structure as per specification <https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#toleration-v1-core>. Example: `--set controllerTolerations[0].operator=Equal,controllerTolerations[0].effect=NoSchedule,controllerTolerations[0].key=dedicated,controllerTolerations[0].value=scalr-agent-controller-pool` |
+| efsMountOptions | list | `[]` | Amazon EFS mount options to define how the EFS storage volume should be mounted. |
+| efsVolumeHandle | string | `""` | Amazon EFS file system ID to use EFS storage as data home directory. |
 | fullnameOverride | string | `""` |  |
 | image.pullPolicy | string | `"Always"` | The pullPolicy for a container and the tag of the image. |
 | image.repository | string | `"scalr/agent"` | Docker repository for the Scalr Agent image. |
 | image.tag | string | `""` | Overrides the image tag whose default is the chart appVersion. |
 | imagePullSecrets | list | `[]` |  |
 | nameOverride | string | `""` |  |
-| podAnnotations | object | `{}` |  |
+| podAnnotations | object | `{}` | The Agent Pods annotations. |
 | resources.limits.cpu | string | `"1000m"` |  |
 | resources.limits.memory | string | `"1024Mi"` |  |
 | resources.requests.cpu | string | `"250m"` |  |
 | resources.requests.memory | string | `"256Mi"` |  |
+| securityContext | object | `{"runAsGroup":0,"runAsUser":0}` | The Agent Pods security context. |
 | serviceAccount.annotations | object | `{}` | Annotations to add to the service account |
 | serviceAccount.create | bool | `true` | Specifies whether a service account should be created |
 | serviceAccount.name | string | `""` | If not set and create is true, a name is generated using the fullname template |
@@ -138,4 +160,4 @@ $ helm upgrade ... \
 | workerTolerations | list | `[]` | Kubernetes Node Tolerations for the agent worker and the agent task pods. Expects input structure as per specification <https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#toleration-v1-core>. Example: `--set workerTolerations[0].operator=Equal,workerTolerations[0].effect=NoSchedule,workerTolerations[0].key=dedicated,workerTolerations[0].value=scalr-agent-worker-pool` |
 
 ----------------------------------------------
-Autogenerated from chart metadata using [helm-docs v1.11.0](https://github.com/norwoodj/helm-docs/releases/v1.11.0)
+Autogenerated from chart metadata using [helm-docs v1.14.2](https://github.com/norwoodj/helm-docs/releases/v1.14.2)
diff --git a/charts/agent-k8s/README.md.gotmpl b/charts/agent-k8s/README.md.gotmpl
@@ -78,6 +78,25 @@ $ helm upgrade ... \
     --set agent.data_home="/var/lib/{unique-name}"
 ```
 
+## Amazon EFS
+
+Amazon EFS can be used as a shared ReadWriteMany volume instead of a node disk. To configure it,
+install the `Amazon EFS CSI Driver` via an add-on. See the documentation: https://docs.aws.amazon.com/eks/latest/userguide/efs-csi.html#efs-install-driver.
+Ensure the add-on is active before proceeding.
+
+Next, configure the Amazon EFS file system ID using the `efsVolumeHandle` option:
+
+```console
+$ helm upgrade ... \
+    --set efsVolumeHandle="fs-582a03f3"
+    # Alternatively, if using an Access Point:
+    # see: https://docs.aws.amazon.com/efs/latest/ug/accessing-fs-nfs-permissions.html#accessing-fs-nfs-permissions-access-points
+    --set efsVolumeHandle="fs-582a03f3::fsap-01e050b7d9a3109d5"
+```
+
+When using EFS, all workers will operate within the same disk namespace, while the controller
+will continue to use an ephemeral directory as its data home.
+
 {{ template "chart.maintainersSection" . }}
 
 {{ template "chart.requirementsSection" . }}

diff --git a/charts/agent-k8s/templates/controller.yaml b/charts/agent-k8s/templates/controller.yaml
@@ -26,6 +26,10 @@ spec:
         {{- toYaml . | nindent 8 }}
       {{- end }}
       serviceAccountName: {{ include "agent-k8s.serviceAccountName" . }}
+      securityContext:
+      {{- with .Values.securityContext }}
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
       containers:
         - name: agent-k8s
           image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
@@ -85,8 +89,15 @@ spec:
               value: "{{ .Values.workerNodeSelector | toJson | b64enc }}"
             - name: SCALR_KUBERNETES_TASK_TOLERATIONS
               value: "{{ .Values.workerTolerations | toJson | b64enc }}"
+            {{- if .Values.efsVolumeHandle }}
+            - name: SCALR_KUBERNETES_EFS_CLAIM_NAME
+              value: "agent-k8s-efs-claim"
+            {{- end }}
           resources:
             {{- toYaml .Values.resources | nindent 12 }}
+          volumeMounts:
+            - name: data-home
+              mountPath: "{{ .Values.agent.data_home }}"
       {{- with .Values.controllerNodeSelector }}
       nodeSelector:
         {{- toYaml . | nindent 8 }}
@@ -95,4 +106,7 @@ spec:
       tolerations:
         {{- toYaml . | nindent 8 }}
       {{- end }}
+      volumes:
+      - name: data-home
+        emptyDir: {}
       terminationGracePeriodSeconds: {{ .Values.terminationGracePeriodSeconds }}
diff --git a/charts/agent-k8s/templates/pv.yaml b/charts/agent-k8s/templates/pv.yaml
@@ -0,0 +1,18 @@
+{{- if .Values.efsVolumeHandle -}}
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+  name: agent-k8s-efs-pv
+  namespace: {{ .Release.Namespace | quote }}
+spec:
+  capacity:
+    storage: 5Gi
+  volumeMode: Filesystem
+  accessModes:
+    - ReadWriteMany
+  persistentVolumeReclaimPolicy: Retain
+  storageClassName: agent-k8s-efs-sc
+  csi:
+    driver: efs.csi.aws.com
+    volumeHandle: {{ .Values.efsVolumeHandle }}
+{{- end }}
diff --git a/charts/agent-k8s/templates/pvc.yaml b/charts/agent-k8s/templates/pvc.yaml
@@ -0,0 +1,14 @@
+{{- if .Values.efsVolumeHandle -}}
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: agent-k8s-efs-claim
+  namespace: {{ .Release.Namespace | quote }}
+spec:
+  accessModes:
+    - ReadWriteMany
+  storageClassName: agent-k8s-efs-sc
+  resources:
+    requests:
+      storage: 5Gi
+{{- end }}
diff --git a/charts/agent-k8s/templates/storageclass.yaml b/charts/agent-k8s/templates/storageclass.yaml
@@ -0,0 +1,16 @@
+{{- if .Values.efsVolumeHandle -}}
+kind: StorageClass
+apiVersion: storage.k8s.io/v1
+metadata:
+  name: agent-k8s-efs-sc
+  namespace: {{ .Release.Namespace | quote }}
+provisioner: efs.csi.aws.com
+reclaimPolicy: Retain
+parameters:
+  provisioningMode: efs-ap
+  directoryPerms: "775"
+mountOptions:
+{{- with .Values.efsMountOptions }}
+  {{- toYaml . | nindent 8 }}
+{{- end }}
+{{- end }}
diff --git a/charts/agent-k8s/templates/worker.yaml b/charts/agent-k8s/templates/worker.yaml
@@ -24,6 +24,10 @@ spec:
         {{- toYaml . | nindent 8 }}
       {{- end }}
       serviceAccountName: {{ include "agent-k8s.serviceAccountName" . }}
+      securityContext:
+      {{- with .Values.securityContext }}
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
       containers:
         - name: agent-k8s
           image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
@@ -75,11 +79,15 @@ spec:
               value: "{{ .Values.agent.gc_plugins_workspace_size_limit }}"
             - name: SCALR_KUBERNETES_MODE
               value: "worker"
+            {{- if .Values.efsVolumeHandle }}
+            - name: SCALR_KUBERNETES_EFS_CLAIM_NAME
+              value: "agent-k8s-efs-claim"
+            {{- end }}
           resources:
             {{- toYaml .Values.resources | nindent 12 }}
           volumeMounts:
             - name: data-home
-              mountPath: {{ .Values.agent.data_home }}
+              mountPath: "{{ .Values.agent.data_home }}"
       {{- with .Values.workerNodeSelector }}
       nodeSelector:
         {{- toYaml . | nindent 8 }}
@@ -90,7 +98,12 @@ spec:
       {{- end }}
       volumes:
         - name: data-home
+          {{- if .Values.efsVolumeHandle }}
+          persistentVolumeClaim:
+            claimName: agent-k8s-efs-claim
+          {{- else }}
           hostPath:
             path: {{ .Values.agent.data_home }}
             type: DirectoryOrCreate
+          {{- end }}
       terminationGracePeriodSeconds: {{ .Values.terminationGracePeriodSeconds }}
diff --git a/charts/agent-k8s/values.yaml b/charts/agent-k8s/values.yaml
@@ -92,6 +92,18 @@ serviceAccount:
   # -- If not set and create is true, a name is generated using the fullname template
   name: ""
 
+# -- Amazon EFS file system ID to use EFS storage as data home directory.
+efsVolumeHandle: ""
+
+# -- Amazon EFS mount options to define how the EFS storage volume should be mounted.
+efsMountOptions: []
+
+# -- The Agent Pods security context.
+securityContext:
+  runAsUser: 0
+  runAsGroup: 0
+
+# -- The Agent Pods annotations.
 podAnnotations: {}
 
 # -- Provides the amount of grace time prior to the agent-k8s container being forcibly terminated when marked for deletion or restarted.