[helm] workload-api-server chart uses inconsistent GCS secret logic from server/worker/workload-launcher

[stevenmurphy12]

Helm Chart Version

0.445.3

What step the error happened?

Upgrading the Platform or Helm Chart

Relevant information

Problem

I'm attempting to set up my OSS Airbyte deployment to use GCS logging. Relevant chart values are below (from a FluxCD patch):

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: airbyte
spec:
  values:
    global:
      # gcs-credentials-ext is a secret that contains the GCS credentials, created externally to the helm deployment
      storage:
        type: "GCS"
        storageSecretName: gcs-credentials-ext
        bucket:
          log: my_log_bucket
          state: my_log_bucket
          workloadOutput: my_log_bucket

gcs-credentials-ext was created externally to the Helm chart deployment, as I wanted to avoid having my base64 encoded credentials held in source control at global.storage.gcs.credentialsJson

Analysis

When I run the helm install, the pod for airbyte-workload-api-server gets stuck in the ContainerCreating state, with the following error:

Warning FailedMount 3s (x4 over 6s) kubelet MountVolume.SetUp failed for volume "gcs-log-creds-volume" : secret "airbyte-gcs-log-creds" not found

Upon analysis of the workload API server template, this is due to the secretName defaulting to releasename-gcs-log-creds because it is looking for a value .Values.global.credVolumeOverride). See - https://github.com/airbytehq/airbyte-platform/blob/f8146bdc352726509bc5bbd8870052e666178944/charts/airbyte-workload-api-server/templates/deployment.yaml#L202

This is at odds with the other components that refer to the same secret (they all use Values.global.storage.storageSecretName):

• Workload launcher - https://github.com/airbytehq/airbyte-platform/blob/f8146bdc352726509bc5bbd8870052e666178944/charts/airbyte-workload-launcher/templates/deployment.yaml#L596
• Worker - https://github.com/airbytehq/airbyte-platform/blob/f8146bdc352726509bc5bbd8870052e666178944/charts/airbyte-worker/templates/deployment.yaml#L587
• Server - https://github.com/airbytehq/airbyte-platform/blob/f8146bdc352726509bc5bbd8870052e666178944/charts/airbyte-server/templates/deployment.yaml#L516

Proposal

I propose that this line https://github.com/airbytehq/airbyte-platform/blob/f8146bdc352726509bc5bbd8870052e666178944/charts/airbyte-workload-api-server/templates/deployment.yaml#L202, be changed from:

secretName: {{ ternary (printf "%s-gcs-log-creds" ( .Release.Name )) (.Values.global.credVolumeOverride) (eq .Values.global.deploymentMode "oss") }}

to:

secretName: {{ ternary (printf "%s-gcs-log-creds" ( .Release.Name )) .Values.global.storage.storageSecretName (not ((.Values.global.storage).storageSecretName)) }}

Relevant log output

No response

[Scalahansolo]

Running into this as well while trying to spin up the airbyte chart with GCS. It seems like this is at direct odds with Airbyte's deployment docs at https://docs.airbyte.com/deploying-airbyte/integrations/secrets. This chart just doesn't work?

[stevenmurphy12]

@Scalahansolo What version are you running? The chart was fixed in another thread here - airbytehq/airbyte-platform#355

I've just upgraded to 1.2.0 recently

[Scalahansolo]

I was using 1.3.0. The chart doesn't even have secretsManager as a value you can set for secrets as well. So the issue above is for storage, and seems still broken on 1.3 and secrets don't even look configurable in the way the docs say they can be.

[stevenmurphy12]

I tend not to refer to the documentation as it seems to go out of date very quickly unfortunately, instead I download the helm chart here and debug locally.

https://airbytehq.github.io/helm-charts/index.yaml

My values for GCS creds are here:

yaml
    global:
      storage:
        #TODO - Remove base64 encoded credentials and use secret instead
        type: "GCS"
        bucket:
          log: log_bucket_here
          state: log_bucket_here
          workloadOutput: log_bucket_here
        gcs:
          credentialsJson: "base64encodedjsonkey"
          credentialsPath: /secrets/gcs-log-creds/gcp.json

      # Need to set these here as env vars as this version of Helm chart (0.55.4) doesn't support setting them properly
      env_vars:
        CONTAINER_ORCHESTRATOR_SECRET_MOUNT_PATH: /secrets/gcs-log-creds
        CONTAINER_ORCHESTRATOR_SECRET_NAME: airbyte-gcs-log-creds

I'm not using a secret manager. Instead I provide a base64 encoded JSON key which then gets created as a secret by the Helm chart.

This isn't ideal of course, having secrets in source control (albeit obfuscated). Though it's worked for me so far (I recently did an upgrade from a version 9 months old to 1.2.0). I expect there is a better way to do it now, though not had a chance to explore it yet. The Helm charts have been quite buggy, the Airbyte team have been making quite a lot of changes to them. Though I did start using it prior to the V1 release so it is not surprising

[kev-datams]

Hello, looks like it properly works using storageSecretName instead of secretName under the global.storage section (Helm chart version 1.1.1). No need for complementary env vars.