[YUNIKORN-3000] Add support for setting GOGC / GOMEMLIMIT in Helm cha…

…rt (#185) Added helm variables to allow customizing GOGC / GOMEMLIMIT for YuniKorn components. GOGC is set to 100 (the default) for all services. GOMEMLIMIT is configured by default to 200MiB for the web and admission controller containers, and 1536MiB for the scheduler. This provides ample headroom to avoid encountering OOMKilled events in Kubernetes. Closes: #185
apache · Dec 13, 2024 · 60a4eb3 · 60a4eb3
1 parent 9ef8317
commit 60a4eb3
Show file tree

Hide file tree

Showing 4 changed files with 31 additions and 0 deletions.
diff --git a/helm-charts/yunikorn/README.md b/helm-charts/yunikorn/README.md
@@ -86,6 +86,8 @@ The following table lists the configurable parameters of the YuniKorn chart and
 | `admissionController.resources.requests.memory` | Admission controller memory resource requests               | `500Mi`                         |
 | `admissionController.resources.limits.cpu`      | Admission controller CPU resource limit                     | `500m`                          |
 | `admissionController.resources.limits.memory`   | Admission controller memory resource limit                  | `500Mi`                         |
+| `admissionController.goMemoryLimit`             | Admission controller memory limit (GOMEMLIMIT)              | `200MiB`                        |
+| `admissionController.goGC`                      | Admission controller GC threshold (GOGC)                    | `100`                           |
 | `admissionController.nodeSelector`              | Admission controller deployment nodeSelector(s)             | `{}`                            |
 | `admissionController.tolerations`               | Admission controller deployment tolerations                 | `[]`                            |
 | `admissionController.affinity`                  | Admission controller deployment affinity                    | `{}`                            |
@@ -99,13 +101,17 @@ The following table lists the configurable parameters of the YuniKorn chart and
 | `resources.requests.memory`                     | Memory resource requests                                    | `1Gi`                           |
 | `resources.limits.cpu`                          | CPU resource limit                                          | `4`                             |
 | `resources.limits.memory`                       | Memory resource limit                                       | `2Gi`                           |
+| `goMemoryLimit`                                 | Memory limit (GOMEMLIMIT)                                   | `1536MiB`                       |
+| `goGC`                                          | GC threshold (GOGC)                                         | `100`                           |
 | `web.image.repository`                          | Web app image repository                                    | `apache/yunikorn`               |
 | `web.image.tag`                                 | Web app image tag                                           | `web-latest`                    |
 | `web.image.pullPolicy`                          | Web app image pull policy                                   | `Always`                        |
 | `web.resources.requests.cpu`                    | Web app CPU resource requests                               | `100m`                          |
 | `web.resources.requests.memory`                 | Web app memory resource requests                            | `100Mi`                         |
 | `web.resources.limits.cpu`                      | Web app CPU resource limit                                  | `200m`                          |
 | `web.resources.limits.memory`                   | Web app memory resource limit                               | `500Mi`                         |
+| `web.goMemoryLimit`                             | Web app memory limit (GOMEMLIMIT)                           | `200MiB`                        |
+| `web.goGC`                                      | Web app GC threshold (GOGC)                                 | `100`                           |
 | `embedAdmissionController`                      | Flag for enabling/disabling the admission controller        | `true`                          |
 | `enableSchedulerPlugin`                         | Flag for enabling/disabling scheduler plugin mode           | `false`                         |
 | `enableWebService`                              | Flag for enabling/disabling web service                     | `true`                          |

diff --git a/helm-charts/yunikorn/templates/admission-controller-deployment.yaml b/helm-charts/yunikorn/templates/admission-controller-deployment.yaml
@@ -105,6 +105,10 @@ spec:
               valueFrom:
                 fieldRef:
                   fieldPath: metadata.namespace
+            - name: GOMEMLIMIT
+              value: "{{ .Values.admissionController.goMemoryLimit }}"
+            - name: GOGC
+              value: "{{ .Values.admissionController.goGC }}"
           ports:
             - containerPort: 9089
               name: webhook-api

diff --git a/helm-charts/yunikorn/templates/deployment.yaml b/helm-charts/yunikorn/templates/deployment.yaml
@@ -97,6 +97,10 @@ spec:
               valueFrom:
                 fieldRef:
                   fieldPath: metadata.namespace
+            - name: GOMEMLIMIT
+              value: "{{ .Values.goMemoryLimit }}"
+            - name: GOGC
+              value: "{{ .Values.goGC }}"
           resources:
             {{- toYaml .Values.resources | nindent 12 }}
           {{- with .Values.schedulerSecurityContext }}
@@ -111,6 +115,11 @@ spec:
             - name: http2
               containerPort: {{ .Values.service.portWeb }}
               protocol: TCP
+          env:
+            - name: GOMEMLIMIT
+              value: "{{ .Values.web.goMemoryLimit }}"
+            - name: GOGC
+              value: "{{ .Values.web.goGC }}"
           resources:
             {{- toYaml .Values.web.resources | nindent 12 }}
           {{- with .Values.webSecurityContext }}

diff --git a/helm-charts/yunikorn/values.yaml b/helm-charts/yunikorn/values.yaml
@@ -57,6 +57,8 @@ admissionController:
     limits:
       cpu: 500m
       memory: 500Mi
+  goMemoryLimit: 200MiB
+  goGC: 100
   nodeSelector: {}
   tolerations: []
   affinity: {}
@@ -81,6 +83,8 @@ web:
     limits:
       cpu: 200m
       memory: 500Mi
+  goMemoryLimit: 200MiB
+  goGC: 100
 
 service:
   type: ClusterIP
@@ -105,6 +109,14 @@ resources:
     cpu: 4
     memory: 2Gi
 
+# Go memory limit (GOMEMLIMIT) for the scheduler. This should be updated in tandem with
+# resources.limits.memory to avoid OOMKilled scenarios.
+goMemoryLimit: 1536MiB
+
+# Go garbage-collection threshold (GOGC) as a percentage of freshly allocated data to
+# live data remaining after the previous GC.
+goGC: 100
+
 # Bootstrap configuration for YuniKorn - will be rendered into yunikorn-defaults ConfigMap.
 # Any valid options for YuniKorn may be specified here.
 # Example: