OIDC with custom CA not usable

[lusor]

Describe the bug

I am trying to use OIDC with Weave Gitops, however due to my identity server having a custom/self-signed SSL certificate, Gitops cannot connect to the IdP and terminates.
I could not find any way to let Gitops use a custom CA, either by passing an additional argument or via the helm chart.

Environment

• Weave-Gitops: 0.38.0
• Flux: irrelevant
• Kubernetes: irrelevant

To Reproduce
Steps to reproduce the behavior:

Configure issuerURL in secret oidc-auth to an HTTPS service which is configured with a custom/self-signed certificate and start/restart the Gitops pod.

Expected behavior

There is a way to provide a custom CA file for Gitops to trust

Actual Behavior

Gitops pod terminates because it does not trust the certificate of the IdP.

[bigkevmcd]

We are going to provide a simple way to do this, but you can customise a HelmRelease with a post-render

  postRenderers:
    - kustomize:
        patchesStrategicMerge:
          - apiVersion: apps/v1
            kind: Deployment
            metadata:
              name: weave-gitops-enterprise-mccp-cluster-service
              namespace: flux-system
            spec:
              template:
                spec:
                  containers:
                    - name: clusters-service
                      volumeMounts:
                        - mountPath: /usr/local/share/ca-certificates
                          name: custom-ca
                      env:
                        - name: SSL_CERT_FILE
                          value: /usr/local/share/ca-certificates/ca-bundle.crt
                  volumes:
                    - configMap:
                        defaultMode: 420
                        name: custom-ca
                      name: custom-ca

[bigkevmcd]

Implementing a simple way to do this is high up on our priority list.

[lusor]

I have added an option to specify an existing Secret which holds one or more CA certificates which get added to the base certificates in an init container and then mounted in the main one. I used the debug image of gitops' base image to get access to a shell. Maybe this can help in finding a (generic) solution.

diff --git a/charts/gitops-server/templates/deployment.yaml b/charts/gitops-server/templates/deployment.yaml
index 02fe434f..8b203682 100644
--- a/charts/gitops-server/templates/deployment.yaml
+++ b/charts/gitops-server/templates/deployment.yaml
@@ -36,6 +36,27 @@ spec:
       securityContext:
         {{- toYaml . | nindent 8 }}
       {{- end }}
+      {{- if .Values.caCerts.enable }}
+      initContainers:
+        - name: cacerts
+          {{- with .Values.securityContext }}
+          securityContext:
+            {{- toYaml . | nindent 12 }}
+          {{- end }}
+          image: "{{ .Values.caCerts.image }}"
+          imagePullPolicy: {{ .Values.image.pullPolicy }}
+          command: ["sh"]
+          args:
+            - "-c"
+            - "cat /etc/ssl/certs/ca-certificates.crt /tmp/cacerts-volume/* > /tmp/ssl-certs/ca-certificates.crt"
+          resources:
+            {{- toYaml .Values.resources | nindent 12 }}
+          volumeMounts:
+            - name: ssl-certs
+              mountPath: "/tmp/ssl-certs"
+            - name: cacerts-volume
+              mountPath: "/tmp/cacerts-volume"
+      {{- end }}
       containers:
         - name: {{ .Chart.Name }}
           {{- with .Values.securityContext }}
@@ -95,9 +116,14 @@ spec:
           {{- end }}
           resources:
             {{- toYaml .Values.resources | nindent 12 }}
-          {{- if or .Values.serverTLS.enable .Values.extraVolumeMounts }}
+          {{- if or .Values.caCerts.enable .Values.serverTLS.enable .Values.extraVolumeMounts }}
           volumeMounts:
           {{- end }}
+          {{- if .Values.caCerts.enable }}
+            - name: ssl-certs
+              readOnly: true
+              mountPath: "/etc/ssl/certs"
+          {{- end }}
           {{- if .Values.serverTLS.enable }}
             - name: tls-volume
               readOnly: true
@@ -106,9 +132,18 @@ spec:
           {{- with .Values.extraVolumeMounts }}
             {{- toYaml . | nindent 12 }}
           {{- end }}
-      {{- if or .Values.serverTLS.enable .Values.extraVolumes }}
+      {{- if or .Values.caCerts.enable .Values.serverTLS.enable .Values.extraVolumes }}
       volumes:
       {{- end }}
+      {{- if .Values.caCerts.enable }}
+        - name: ssl-certs
+          emptyDir:
+            medium: Memory
+            sizeLimit: 4Mi
+        - name: cacerts-volume
+          secret:
+            secretName: {{ .Values.caCerts.secretName }}
+      {{ end }}
       {{- if .Values.serverTLS.enable }}
         - name: tls-volume
           secret:
diff --git a/charts/gitops-server/values.yaml b/charts/gitops-server/values.yaml
index 8e27d5c9..49303746 100644
--- a/charts/gitops-server/values.yaml
+++ b/charts/gitops-server/values.yaml
@@ -200,3 +200,7 @@ metrics:
       prometheus.io/scrape: "true"
       prometheus.io/path: "/metrics"
       prometheus.io/port: "{{ .Values.metrics.service.port }}"
+caCerts:
+  enable: false
+  image: gcr.io/distroless/base:debug
+  secretName: ""

[c-bonzon]

in our case we tried using the existing variables on the chart (extraVolumeMounts, extraVolumes, envVars), and it worked.

extraVolumeMounts:
  - name: custom-ca
    mountPath: /usr/local/share/ca-certificates/custom-root-ca.crt
    subPath: custom-root-ca.crt
    readOnly: false
extraVolumes:
  - name: custom-ca
    configMap:
      name: custom-root-ca
	  
envVars:
  - name: SSL_CERT_FILE
    value: /usr/local/share/ca-certificates/custom-root-ca.crt