Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

helm values file is unable to recognize the secret file which is encrypted using sops #457

Open
priyas16 opened this issue Jun 19, 2024 · 10 comments
Labels
bug Something isn't working

Comments

@priyas16
Copy link

Current Behavior

We are trying to deploy our changes using helm in Argo CD. Our Application file has reference to both the values.yaml and the secret which is encrypted using sops like below. But its not getting deployed, saying the fields referenced through the encrypted file are not set.

helm:
valueFiles:
- values.yaml
- secrets+age-import:///helm-secrets-private-keys/key.txt?accessToken.enc.yaml

Have verified manually that the encryption is working properly by decrypting the values and was able to deploy the helm manually with the same secret fields. Also in order to debug tried passing one of the field value directly inside the values.yaml and that error is not seen now. We would require your help to resolve this issue. Thanks!

Expected Behavior

We are trying to deploy our changes using helm in Argo CD. Our Application file has reference to both the values.yaml and the secret which is encrypted using sops like below.

helm:
valueFiles:
- values.yaml
- secrets+age-import:///helm-secrets-private-keys/key.txt?accessToken.enc.yaml

We are expecting the values.yaml file recognises the accessToken.enc.yaml and deploy the changes.

Steps To Reproduce

No response

Environment

  • Helm Version: 3.14.0
  • Helm Secrets Version: 4.4.2
  • ArgoCD Version : V2.10.3
  • OS: RHEL 8.9
  • Shell: 4.4

Anything else?

No response

@priyas16 priyas16 added the bug Something isn't working label Jun 19, 2024
@jkroepke
Copy link
Owner

tbh. it's not easy to debug from my side, but I feel one thing a bit strange:

But its not getting deployed, saying the fields referenced through the encrypted file are not set.

An age encrypted file looks like this:

global_secret: ENC[AES256_GCM,data:8J9WPHdNUszCZQ==,iv:4/cZkfTgRrIZ29wybSHbe+6v0Xi5rsvqCcezesYfFEs=,tag:xrGJHNZVKxCuqTjXsU+Yow==,type:str]
key: ENC[AES256_GCM,data:gXs/PWsVPICrYYDZOBIDuc1PUqlVnGm8pVQ5UFPhbQ/BdntoIyX4sCtbdBOEyGqvbmkLEh++1NyWsNJdkY0KTA0u1BOt7Th+AhUeWjmjOpFWsbcji/GseDHDL4il2cEM81OQn2zOvC9Nm+AVvWeWJDAN7VQX9ynvI5ccYCa8RJDnl7iOryTKTdV9U6biFH/r6j3cRAvuEvbTAsCpMB8dNqeWVOSkhaKWkpOMHE5yfmeG7RBXD5WlY7XZNf8brKL2KoQdmr3yX+HFy0K0neROmyguK66hiNKSmUdXcXpUBp6KLFoULGXwNdI8WoGpfHeivOSVKwV/vwQf8igN/OOQuxNwcTwKFzlhcrLEc36KIqEtlXagN88uWTJayHgNzoqAaarkl54vu9ufw4x5IpGWpfJSDoPLfTmp9Y/E+Q2tN9b7rKJpdbkK20M2bURBulMKPJCBdK7ETh7p1ReP/GbCe5O6wNbHd01S6SKlBpqlkasLqz/2UCq9b7aPWHISTpBnMw+r+Q8sqqlsj0j2KgC8h0qdG6E9uwVlz+P0XWpu/t8Ss9D7mn6QNAF9sRL7LxjF2r6L7v1odUOpFB0hqEN4p2Nv42j6uQdpTor5nayYTbUISSb6xOO7zLOI/j1dssju3gfZVOjeXxApclFnE5O2ifl5YxcEqxRFDNEukuXd0GKIxZYkBTADJfmE7A2K0YELcTJdaHTgS2zqapPSiPlemxgBEKwIJZXw9Eflh0F1adcxn/bwU14v/38atDZ+tXuL3UIOL0AIjZcr1FI+RmorYr8lgGBlylNWYMjXWkHqdpqAeUNlw13ZDgm5eE4oiS7qbUZwJSP4JhTspvdI6yLvYQA/4bgdbhqK732KFIYyFpK0jUmU2kfy2wIMSfcDwmMb1CL+12WmnAE2nRVsRPzALrnNH4jZSrkF5d+juUO//K7bMNBSrlq1gHqT+KcnVC3EE5gEcuNeyuxVlJURLTmrnOnB+8WE+04ZRh+ez5/EiVhisUf9TZkd+d/hh0BhMt3wSsLxWriFZ3R6R766PHFYPE8TOHpKBxudLxXMEHuDOtPZtZYChHCHfhw/BpHJGcsv4m98iq8fyDITb9GA8NW/GwppZrontnOcBW3I+wa2kLNseRccuKx0hzx0vXescN2+0AQWVAg53zXcFpgHENcXG5Ey0zab7CHcyDGYWPabVSGbW4tqcl7gUCnAx7Yapg==,iv:Iwz2S9gpSerlXEIu4hmby0+anIWA63JI8EBxrfA+DVM=,tag:PlYpI2qZ7t+TX8QPELNbVw==,type:str]
service:
port: ENC[AES256_GCM,data:RPI=,iv:Hwx1wfPCuwXVn6kI2QX2oA5aN0uPdmIQh01hMxZI0bI=,tag:82MzqSJ4hPR9JEa8o5e5EA==,type:int]

Even the files are not decrypted by help, the keys should be still present, but the values should be encrypted only. Could you confirm that this:

helm:
  valueFiles:
  - values.yaml
  - accessToken.enc.yaml

is not return any errors?

@priyas16
Copy link
Author

tbh. it's not easy to debug from my side, but I feel one thing a bit strange:

But its not getting deployed, saying the fields referenced through the encrypted file are not set.

An age encrypted file looks like this:

global_secret: ENC[AES256_GCM,data:8J9WPHdNUszCZQ==,iv:4/cZkfTgRrIZ29wybSHbe+6v0Xi5rsvqCcezesYfFEs=,tag:xrGJHNZVKxCuqTjXsU+Yow==,type:str]
key: ENC[AES256_GCM,data:gXs/PWsVPICrYYDZOBIDuc1PUqlVnGm8pVQ5UFPhbQ/BdntoIyX4sCtbdBOEyGqvbmkLEh++1NyWsNJdkY0KTA0u1BOt7Th+AhUeWjmjOpFWsbcji/GseDHDL4il2cEM81OQn2zOvC9Nm+AVvWeWJDAN7VQX9ynvI5ccYCa8RJDnl7iOryTKTdV9U6biFH/r6j3cRAvuEvbTAsCpMB8dNqeWVOSkhaKWkpOMHE5yfmeG7RBXD5WlY7XZNf8brKL2KoQdmr3yX+HFy0K0neROmyguK66hiNKSmUdXcXpUBp6KLFoULGXwNdI8WoGpfHeivOSVKwV/vwQf8igN/OOQuxNwcTwKFzlhcrLEc36KIqEtlXagN88uWTJayHgNzoqAaarkl54vu9ufw4x5IpGWpfJSDoPLfTmp9Y/E+Q2tN9b7rKJpdbkK20M2bURBulMKPJCBdK7ETh7p1ReP/GbCe5O6wNbHd01S6SKlBpqlkasLqz/2UCq9b7aPWHISTpBnMw+r+Q8sqqlsj0j2KgC8h0qdG6E9uwVlz+P0XWpu/t8Ss9D7mn6QNAF9sRL7LxjF2r6L7v1odUOpFB0hqEN4p2Nv42j6uQdpTor5nayYTbUISSb6xOO7zLOI/j1dssju3gfZVOjeXxApclFnE5O2ifl5YxcEqxRFDNEukuXd0GKIxZYkBTADJfmE7A2K0YELcTJdaHTgS2zqapPSiPlemxgBEKwIJZXw9Eflh0F1adcxn/bwU14v/38atDZ+tXuL3UIOL0AIjZcr1FI+RmorYr8lgGBlylNWYMjXWkHqdpqAeUNlw13ZDgm5eE4oiS7qbUZwJSP4JhTspvdI6yLvYQA/4bgdbhqK732KFIYyFpK0jUmU2kfy2wIMSfcDwmMb1CL+12WmnAE2nRVsRPzALrnNH4jZSrkF5d+juUO//K7bMNBSrlq1gHqT+KcnVC3EE5gEcuNeyuxVlJURLTmrnOnB+8WE+04ZRh+ez5/EiVhisUf9TZkd+d/hh0BhMt3wSsLxWriFZ3R6R766PHFYPE8TOHpKBxudLxXMEHuDOtPZtZYChHCHfhw/BpHJGcsv4m98iq8fyDITb9GA8NW/GwppZrontnOcBW3I+wa2kLNseRccuKx0hzx0vXescN2+0AQWVAg53zXcFpgHENcXG5Ey0zab7CHcyDGYWPabVSGbW4tqcl7gUCnAx7Yapg==,iv:Iwz2S9gpSerlXEIu4hmby0+anIWA63JI8EBxrfA+DVM=,tag:PlYpI2qZ7t+TX8QPELNbVw==,type:str]
service:
port: ENC[AES256_GCM,data:RPI=,iv:Hwx1wfPCuwXVn6kI2QX2oA5aN0uPdmIQh01hMxZI0bI=,tag:82MzqSJ4hPR9JEa8o5e5EA==,type:int]

Even the files are not decrypted by help, the keys should be still present, but the values should be encrypted only. Could you confirm that this:

helm:
  valueFiles:
  - values.yaml
  - accessToken.enc.yaml

is not return any errors?

Hi jkroepke,

Yes, we are seeing the error. Have attached it for your reference.
image

@jkroepke
Copy link
Owner

If

helm:
  valueFiles:
  - values.yaml
  - accessToken.enc.yaml

returns the same error, then the keys accessToken.enc.yaml are wrong. The file should contain splunkObservability.realm or splunkPlatform.endpoint.

@priyas16
Copy link
Author

Hi jkroepke,

Thanks for your response. Please find below the different error message for 2 scenarios of passing the encrypted file.

Scenario1:

helm:
valueFiles:

  • values.yaml
  • secrets+age-import:///helm-secrets-private-keys/key.txt?accessToken.enc.yaml

Error:
image

Scenario2:

helm:
valueFiles:

  • values.yaml
  • accessToken.enc.yaml

Error:
image

But the keys are correct we did verify by decrypting them.

Could you please check and let us know your thoughts. Thanks!

@jkroepke
Copy link
Owner

Could it be possible that you miss an additional value property which is required in addition?

Since, we can see a difference between encrypted an non-encrypted. I expected that encryption works as expected.

@ajaykumar4
Copy link
Contributor

ajaykumar4 commented Jul 4, 2024

@jkroepke even I'm facing same issue
I add all the configuration mentioned in document for argocd and sops with age.
In argo application.yaml

helm:
  valueFiles:
    - values.yaml
    - values.sops.yaml

Getting error values.sops.yaml file not found.

Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = Manifest generation error (cached): `helm template . --name-template argocd --namespace argocd --kube-version 1.30 --values secrets://values.sops.yaml <api versions removed> --include-crds` failed exit status 1: [helm-secrets] File does not exist: values.sops.yaml Error: failed parsing --set-file data: plugin "scripts/run.sh downloader" exited with error [helm-secrets] helm template command errored on value 'secrets://values.sops.yaml' [helm-secrets] File does not exist: secrets://values.sops.yaml Error: plugin "secrets" exited with error

If I remove values.sops.yaml, then it works fine.
https://github.com/ajaykumar4/home-lab-argocd/tree/main/kubernetes/apps/argocd/argocd

@Harishkrishna17
Copy link

Harishkrishna17 commented Jul 5, 2024

@jkroepke Even we have the same problem , Pasting my configuration below

Please help

argocd helm chart file :

repoServer:
  env:
  ## See https://github.com/jkroepke/helm-secrets/blob/main/docs/ARGOCD.md#option-2-init-container
  - name: HELM_PLUGINS
    value: /custom-tools/helm-plugins/
  - name: HELM_SECRETS_SOPS_PATH
    value: /custom-tools/sops
  - name: HELM_SECRETS_KUBECTL_PATH
    value: /custom-tools/kubectl
  volumeMounts:
  ## See https://github.com/jkroepke/helm-secrets/blob/main/docs/ARGOCD.md#option-2-init-container
  - mountPath: /custom-tools
    name: custom-tools
  ## See https://github.com/jkroepke/helm-secrets/blob/main/docs/ARGOCD.md#method-1-mount-the-private-key-from-a-kubernetes-secret-as-volume
  - mountPath: /helm-secrets-private-keys/
    name: helm-secrets-private-keys

  # -- Additional volumes to the repo server pod
  volumes:
  ## See https://github.com/jkroepke/helm-secrets/blob/main/docs/ARGOCD.md#option-2-init-container
  - name: custom-tools
    emptyDir: {}
  ## See https://github.com/jkroepke/helm-secrets/blob/main/docs/ARGOCD.md#method-1-mount-the-private-key-from-a-kubernetes-secret-as-volume
  - name: helm-secrets-private-keys
    secret:
      secretName: helm-secrets-private-keys
  initContainers:
  ## See https://github.com/jkroepke/helm-secrets/blob/main/docs/ARGOCD.md#option-2-init-container
  - name: download-tools
    image: alpine:latest
    command: [sh, -ec]
    env:
    - name: HELM_SECRETS_VERSION
      value: "4.6.0"
    - name: KUBECTL_VERSION
      value: "1.30.1"
    - name: VALS_VERSION
      value: "0.37.1"
    - name: SOPS_VERSION
      value: "3.8.1"
    args:
    - |
      mkdir -p /custom-tools/helm-plugins
      wget -qO- https://github.com/jkroepke/helm-secrets/releases/download/v${HELM_SECRETS_VERSION}/helm-secrets.tar.gz | tar -C /custom-tools/helm-plugins -xzf-;
      wget -qO /custom-tools/curl https://github.com/moparisthebest/static-curl/releases/latest/download/curl-amd64
      wget -qO /custom-tools/sops https://github.com/getsops/sops/releases/download/v${SOPS_VERSION}/sops-v${SOPS_VERSION}.linux.amd64
      wget -qO /custom-tools/kubectl https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/amd64/kubectl
      wget -qO- https://github.com/helmfile/vals/releases/download/v${VALS_VERSION}/vals_${VALS_VERSION}_linux_amd64.tar.gz | tar -xzf- -C /custom-tools/ vals;
      cp /custom-tools/helm-plugins/helm-secrets/scripts/wrapper/helm.sh /custom-tools/helm
      chmod +x /custom-tools/*
    volumeMounts:
    - mountPath: /custom-tools
      name: custom-tools

argo yaml file

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: harish
  namespace: argocd
spec:
  project: default
  source:
    repoURL: my-repo
    targetRevision: master
    path: helm-charts/platform/kube-prometheus-stack
    helm:
      valueFiles:
      - values-dev.yaml
      - secrets+age-import:///helm-secrets-private-keys/key.txt?values.sops.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

argocd confimap :

apiVersion: v1
data:
  admin.enabled: "true"
  application.instanceLabelKey: argocd.argoproj.io/instance
  exec.enabled: "false"
  helm.valuesFileSchemes: secrets+gpg-import, secrets+gpg-import-kubernetes, secrets+age-import,
    secrets+age-import-kubernetes, secrets,secrets+literal, https
  server.rbac.log.enforce.enable: "false"
  statusbadge.enabled: "false"
  timeout.hard.reconciliation: 0s
  timeout.reconciliation: 180s
  url: https://argocd.example.com
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: argocd
    meta.helm.sh/release-namespace: argocd
  creationTimestamp: "2024-07-05T14:30:12Z"
  labels:
    app.kubernetes.io/component: server
    app.kubernetes.io/instance: argocd
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
    app.kubernetes.io/version: v2.11.3
    helm.sh/chart: argo-cd-7.1.3
  name: argocd-cm
  namespace: argocd
  resourceVersion: "3889857"
  uid: ce560fb5-5800-4ca3-bde0-78293095eea7

image
Screenshot 2024-07-05 at 20 30 23

error:

Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = helm template . --name-template harish --namespace argocd --kube-version 1.29 --values <path to cached source>/helm-charts/platform/kube-prometheus-stack/values-dev.yaml --values secrets+age-import:///helm-secrets-private-keys/key.txt?values.sops.yaml <api versions removed> --include-crds failed exit status 1: No keys found in file [helm-secrets] Error while decrypting file: values.sops.yaml Error: plugin "scripts/run.sh downloader" exited with error

@jkroepke
Copy link
Owner

jkroepke commented Jul 6, 2024

@jkroepke even I'm facing same issue

I add all the configuration mentioned in document for argocd and sops with age.

In argo application.yaml


helm:

  valueFiles:

    - values.yaml

    - values.sops.yaml

Getting error values.sops.yaml file not found.


Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = Manifest generation error (cached): `helm template . --name-template argocd --namespace argocd --kube-version 1.30 --values secrets://values.sops.yaml <api versions removed> --include-crds` failed exit status 1: [helm-secrets] File does not exist: values.sops.yaml Error: failed parsing --set-file data: plugin "scripts/run.sh downloader" exited with error [helm-secrets] helm template command errored on value 'secrets://values.sops.yaml' [helm-secrets] File does not exist: secrets://values.sops.yaml Error: plugin "secrets" exited with error

If I remove values.sops.yaml, then it works fine.

https://github.com/ajaykumar4/home-lab-argocd/tree/main/kubernetes/apps/argocd/argocd

Last time I used ArgoCD, it does not extract the helm tar.gz package that contains the secrets.yaml. helm-secrets just looks on the file system and may not found the file on the filesystem.

@jkroepke
Copy link
Owner

jkroepke commented Jul 6, 2024

@jkroepke Even we have the same problem , Pasting my configuration below

Please help

argocd helm chart file :

repoServer:

  env:

  ## See https://github.com/jkroepke/helm-secrets/blob/main/docs/ARGOCD.md#option-2-init-container

  - name: HELM_PLUGINS

    value: /custom-tools/helm-plugins/

  - name: HELM_SECRETS_SOPS_PATH

    value: /custom-tools/sops

  - name: HELM_SECRETS_KUBECTL_PATH

    value: /custom-tools/kubectl

  volumeMounts:

  ## See https://github.com/jkroepke/helm-secrets/blob/main/docs/ARGOCD.md#option-2-init-container

  - mountPath: /custom-tools

    name: custom-tools

  ## See https://github.com/jkroepke/helm-secrets/blob/main/docs/ARGOCD.md#method-1-mount-the-private-key-from-a-kubernetes-secret-as-volume

  - mountPath: /helm-secrets-private-keys/

    name: helm-secrets-private-keys



  # -- Additional volumes to the repo server pod

  volumes:

  ## See https://github.com/jkroepke/helm-secrets/blob/main/docs/ARGOCD.md#option-2-init-container

  - name: custom-tools

    emptyDir: {}

  ## See https://github.com/jkroepke/helm-secrets/blob/main/docs/ARGOCD.md#method-1-mount-the-private-key-from-a-kubernetes-secret-as-volume

  - name: helm-secrets-private-keys

    secret:

      secretName: helm-secrets-private-keys

  initContainers:

  ## See https://github.com/jkroepke/helm-secrets/blob/main/docs/ARGOCD.md#option-2-init-container

  - name: download-tools

    image: alpine:latest

    command: [sh, -ec]

    env:

    - name: HELM_SECRETS_VERSION

      value: "4.6.0"

    - name: KUBECTL_VERSION

      value: "1.30.1"

    - name: VALS_VERSION

      value: "0.37.1"

    - name: SOPS_VERSION

      value: "3.8.1"

    args:

    - |

      mkdir -p /custom-tools/helm-plugins

      wget -qO- https://github.com/jkroepke/helm-secrets/releases/download/v${HELM_SECRETS_VERSION}/helm-secrets.tar.gz | tar -C /custom-tools/helm-plugins -xzf-;

      wget -qO /custom-tools/curl https://github.com/moparisthebest/static-curl/releases/latest/download/curl-amd64

      wget -qO /custom-tools/sops https://github.com/getsops/sops/releases/download/v${SOPS_VERSION}/sops-v${SOPS_VERSION}.linux.amd64

      wget -qO /custom-tools/kubectl https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/amd64/kubectl

      wget -qO- https://github.com/helmfile/vals/releases/download/v${VALS_VERSION}/vals_${VALS_VERSION}_linux_amd64.tar.gz | tar -xzf- -C /custom-tools/ vals;

      cp /custom-tools/helm-plugins/helm-secrets/scripts/wrapper/helm.sh /custom-tools/helm

      chmod +x /custom-tools/*

    volumeMounts:

    - mountPath: /custom-tools

      name: custom-tools

argo yaml file


apiVersion: argoproj.io/v1alpha1

kind: Application

metadata:

  name: harish

  namespace: argocd

spec:

  project: default

  source:

    repoURL: my-repo

    targetRevision: master

    path: helm-charts/platform/kube-prometheus-stack

    helm:

      valueFiles:

      - values-dev.yaml

      - secrets+age-import:///helm-secrets-private-keys/key.txt?values.sops.yaml

  destination:

    server: https://kubernetes.default.svc

    namespace: argocd

  syncPolicy:

    automated:

      prune: true

      selfHeal: true

argocd confimap :

apiVersion: v1

data:

  admin.enabled: "true"

  application.instanceLabelKey: argocd.argoproj.io/instance

  exec.enabled: "false"

  helm.valuesFileSchemes: secrets+gpg-import, secrets+gpg-import-kubernetes, secrets+age-import,

    secrets+age-import-kubernetes, secrets,secrets+literal, https

  server.rbac.log.enforce.enable: "false"

  statusbadge.enabled: "false"

  timeout.hard.reconciliation: 0s

  timeout.reconciliation: 180s

  url: https://argocd.example.com

kind: ConfigMap

metadata:

  annotations:

    meta.helm.sh/release-name: argocd

    meta.helm.sh/release-namespace: argocd

  creationTimestamp: "2024-07-05T14:30:12Z"

  labels:

    app.kubernetes.io/component: server

    app.kubernetes.io/instance: argocd

    app.kubernetes.io/managed-by: Helm

    app.kubernetes.io/name: argocd-cm

    app.kubernetes.io/part-of: argocd

    app.kubernetes.io/version: v2.11.3

    helm.sh/chart: argo-cd-7.1.3

  name: argocd-cm

  namespace: argocd

  resourceVersion: "3889857"

  uid: ce560fb5-5800-4ca3-bde0-78293095eea7

image

Screenshot 2024-07-05 at 20 30 23

error:

Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = helm template . --name-template harish --namespace argocd --kube-version 1.29 --values <path to cached source>/helm-charts/platform/kube-prometheus-stack/values-dev.yaml --values secrets+age-import:///helm-secrets-private-keys/key.txt?values.sops.yaml <api versions removed> --include-crds failed exit status 1: No keys found in file [helm-secrets] Error while decrypting file: values.sops.yaml Error: plugin "scripts/run.sh downloader" exited with error

Sound like age does not find a valid decryption key in /helm-secrets-private-keys/key.txt but the error is not the same.

@priyas16
Copy link
Author

priyas16 commented Jul 8, 2024

Could it be possible that you miss an additional value property which is required in addition?

Since, we can see a difference between encrypted an non-encrypted. I expected that encryption works as expected.

Hi @jkroepke,

Only these two parameters (realm and accessToken) are required and they are working fine when we deploy the values.yaml manually. The issue occurs while using ArgoCD with the encrypted file reference.

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants