Skip to content

Vault, KES and MinIO Deployed in k8s Cluster

Jump to bottom
Cesar Celis Hernandez edited this page Apr 4, 2022 · 12 revisions

Kind cluster with 4 nodes containing vault pod, kes pod and MinIO pods to explain how spec.kes is used in tenant.yaml using Kustomize. This is NOT a generic example that does not work, it is a WORKING example for you to understand how it works step by step. Hope you like it!

Steps with Kustomize

  1. Delete any previous cluster, create a new cluster, install operator, deploy vault and wait for pod to be ready
kind delete clusters kind
kind create cluster --config ~/operator/testing/kind-config.yaml
kubectl apply -k ~/operator/resources
kubectl apply -f ~/operator/examples/vault/deployment.yaml
kubectl wait --namespace default \
	--for=condition=ready pod \
	--selector=app=vault \
	--timeout=120s
  1. Setup vault and get the credentials:
VAULT_ROOT_TOKEN=$(kubectl logs -l app=vault | grep "Root Token: " | sed -e "s/Root Token: //g")
kubectl exec $(kubectl get pods -l app=vault  | grep -v NAME | awk '{print $1}') -- sh -c 'VAULT_TOKEN='$VAULT_ROOT_TOKEN' VAULT_ADDR="http://127.0.0.1:8200" vault auth enable approle'
kubectl exec $(kubectl get pods -l app=vault  | grep -v NAME | awk '{print $1}') -- sh -c 'VAULT_TOKEN='$VAULT_ROOT_TOKEN' VAULT_ADDR="http://127.0.0.1:8200" vault secrets enable kv'
kubectl cp ~/operator/examples/vault/kes-policy.hcl $(kubectl get pods -l app=vault  | grep -v NAME | awk '{print $1}'):/kes-policy.hcl
kubectl exec $(kubectl get pods -l app=vault  | grep -v NAME | awk '{print $1}') -- sh -c 'VAULT_TOKEN='$VAULT_ROOT_TOKEN' VAULT_ADDR="http://127.0.0.1:8200" vault policy write kes-policy /kes-policy.hcl'
kubectl exec $(kubectl get pods -l app=vault  | grep -v NAME | awk '{print $1}') -- sh -c 'VAULT_TOKEN='$VAULT_ROOT_TOKEN' VAULT_ADDR="http://127.0.0.1:8200" vault write auth/approle/role/kes-role token_num_uses=0 secret_id_num_uses=0 period=5m policies=kes-policy'
ROLE_ID=$(kubectl exec $(kubectl get pods -l app=vault  | grep -v NAME | awk '{print $1}') -- sh -c 'VAULT_TOKEN='$VAULT_ROOT_TOKEN' VAULT_ADDR="http://127.0.0.1:8200" vault read auth/approle/role/kes-role/role-id' | grep "role_id    " | sed -e "s/role_id    //g")
SECRET_ID=$(kubectl exec $(kubectl get pods -l app=vault  | grep -v NAME | awk '{print $1}') -- sh -c 'VAULT_TOKEN='$VAULT_ROOT_TOKEN' VAULT_ADDR="http://127.0.0.1:8200" vault write -f auth/approle/role/kes-role/secret-id' | grep "secret_id             " | sed -e "s/secret_id             //g")
echo $ROLE_ID
echo $SECRET_ID
  1. Prepare ~/operator/examples/kustomization/tenant-kes-encryption/tenant.yaml file:

Where spec.kesSecret.name is kes-configuration:

apiVersion: minio.min.io/v2
kind: Tenant
metadata:
  name: storage
  namespace: minio-tenant
spec:
  ## Define configuration for KES (stateless and distributed key-management system)
  ## Refer https://github.com/minio/kes
  kes:
    image: "" # minio/kes:v0.17.6
    replicas: 2
    kesSecret:
      name: kes-configuration
    imagePullPolicy: "IfNotPresent"
    externalCertSecret: null
    clientCertSecret: null
    ## Key name to be created on the KMS, default is "my-minio-key"
    keyName: ""
    resources: { }
    nodeSelector: { }
    affinity:
      nodeAffinity: { }
      podAffinity: { }
      podAntiAffinity: { }
    tolerations: [ ]
    annotations: { }
    labels: { }
    serviceAccountName: ""
    securityContext:
      runAsUser: 1000
      runAsGroup: 1000
      runAsNonRoot: true
      fsGroup: 1000
  1. Prepare ~/operator/examples/kustomization/tenant-kes-encryption/kes-configuration-secret.yaml:

Where approle.id is ROLE_ID and approle.secret is SECRET_ID

apiVersion: v1
kind: Secret
metadata:
  name: kes-configuration
type: Opaque
stringData:
  server-config.yaml: |-
    address: 0.0.0.0:7373
    root: disabled
    tls:
      key: /tmp/kes/server.key
      cert: /tmp/kes/server.crt
    policy:
      default-policy:
        paths:
        - /v1/key/create/my-minio-key
        - /v1/key/generate/my-minio-key
        - /v1/key/decrypt/my-minio-key
        identities:
        - ${MINIO_KES_IDENTITY}
    cache:
      expiry:
        any: 5m0s
        unused: 20s
    log:
      error: "on"
      audit: "off"
    keys:
      vault:
        endpoint: http://vault.default.svc.cluster.local:8200
        prefix: my-minio
        approle:
          id: 3672dca5-c059-6a34-7a4d-02217718a28f
          secret: 03672538-7145-b599-264f-24fda65ecc42
          retry: 15s
        tls: {}
        status:
          ping: 10s
  1. Deploy the tenant with KES using Vault 
kubectl apply -k ~/operator/examples/kustomization/tenant-kes-encryption
  1. KES replica is connected to Vault and serving in port 7373 
$ k logs storage-kms-encrypted-kes-0 -n tenant-kms-encrypted

Authenticating to Hashicorp Vault 'http://vault.default.svc.cluster.local:8200' ... 
Endpoint: https://127.0.0.1:7373        https://10.244.1.3:7373       

Admin:    _     [ disabled ]
Auth:     off   [ any client can connect but policies still apply ]

Keys:     Hashicorp Vault: http://vault.default.svc.cluster.local:8200

CLI:      export KES_SERVER=https://127.0.0.1:7373
          export KES_CLIENT_KEY=<client-private-key>   // e.g. $HOME/root.key
          export KES_CLIENT_CERT=<client-certificate>  // e.g. $HOME/root.cert
          kes --help
  1. Port forward MinIO and run mc commands:
kubectl port-forward storage-kms-encrypted-ss-0-0 -n tenant-kms-encrypted 9000
mc config host add kestest https://localhost:9000 minio minio123 --insecure
mc admin kms key status kestest --insecure
  1. You should see Encryption and Decryption working:
Key: my-minio-key
   - Encryption ✔
   - Decryption ✔
  1. You can use tools like k9s or Lens to view the pods:
Screen Shot 2022-04-01 at 10 20 49 PM

Note: This document was using Kustomize next related post will be using helm

Steps with Helm

  1. Kill any previous cluster to start clean
kind delete clusters kind
  1. Create new cluster
kind create cluster --config ~/operator/testing/kind-config.yaml
  1. Create Operator, create Vault and then wait for Vault to be ready for further configuration
kubectl apply -k ~/operator/resources
kubectl apply -f ~/operator/examples/vault/deployment.yaml
kubectl wait --namespace default \
	--for=condition=ready pod \
	--selector=app=vault \
	--timeout=120s
  1. Configure Vault and get credentials:
VAULT_ROOT_TOKEN=$(kubectl logs -l app=vault | grep "Root Token: " | sed -e "s/Root Token: //g")
kubectl exec $(kubectl get pods -l app=vault  | grep -v NAME | awk '{print $1}') -- sh -c 'VAULT_TOKEN='$VAULT_ROOT_TOKEN' VAULT_ADDR="http://127.0.0.1:8200" vault auth enable approle'
kubectl exec $(kubectl get pods -l app=vault  | grep -v NAME | awk '{print $1}') -- sh -c 'VAULT_TOKEN='$VAULT_ROOT_TOKEN' VAULT_ADDR="http://127.0.0.1:8200" vault secrets enable kv'
kubectl cp ~/operator/examples/vault/kes-policy.hcl $(kubectl get pods -l app=vault  | grep -v NAME | awk '{print $1}'):/kes-policy.hcl
kubectl exec $(kubectl get pods -l app=vault  | grep -v NAME | awk '{print $1}') -- sh -c 'VAULT_TOKEN='$VAULT_ROOT_TOKEN' VAULT_ADDR="http://127.0.0.1:8200" vault policy write kes-policy /kes-policy.hcl'
kubectl exec $(kubectl get pods -l app=vault  | grep -v NAME | awk '{print $1}') -- sh -c 'VAULT_TOKEN='$VAULT_ROOT_TOKEN' VAULT_ADDR="http://127.0.0.1:8200" vault write auth/approle/role/kes-role token_num_uses=0 secret_id_num_uses=0 period=5m policies=kes-policy'
ROLE_ID=$(kubectl exec $(kubectl get pods -l app=vault  | grep -v NAME | awk '{print $1}') -- sh -c 'VAULT_TOKEN='$VAULT_ROOT_TOKEN' VAULT_ADDR="http://127.0.0.1:8200" vault read auth/approle/role/kes-role/role-id' | grep "role_id    " | sed -e "s/role_id    //g")
SECRET_ID=$(kubectl exec $(kubectl get pods -l app=vault  | grep -v NAME | awk '{print $1}') -- sh -c 'VAULT_TOKEN='$VAULT_ROOT_TOKEN' VAULT_ADDR="http://127.0.0.1:8200" vault write -f auth/approle/role/kes-role/secret-id' | grep "secret_id             " | sed -e "s/secret_id             //g")
echo $ROLE_ID
echo $SECRET_ID
  1. Provide credentials in ~/operator/examples/kustomization/tenant-kes-encryption/kes-configuration-secret.yaml

stringData.server-config.yaml.keys.vault.approle.id = ROLE_ID and stringData.server-config.yaml.keys.vault.approle.secret = SECRET_ID

apiVersion: v1
kind: Secret
metadata:
  name: kes-configuration
type: Opaque
stringData:
  server-config.yaml: |-
    address: 0.0.0.0:7373
    root: disabled
    tls:
      key: /tmp/kes/server.key
      cert: /tmp/kes/server.crt
    policy:
      default-policy:
        paths:
        - /v1/key/create/my-minio-key
        - /v1/key/generate/my-minio-key
        - /v1/key/decrypt/my-minio-key
        identities:
        - ${MINIO_KES_IDENTITY}
    cache:
      expiry:
        any: 5m0s
        unused: 20s
    log:
      error: "on"
      audit: "off"
    keys:
      vault:
        endpoint: http://vault.default.svc.cluster.local:8200
        prefix: my-minio
        approle:
          id: 36df6ade-1ac3-ad81-5fff-2b862d09247a
          secret: 6213986f-ea2c-2d74-ce82-bc93bfab26d2
          retry: 15s
        tls: {}
        status:
          ping: 10s
  1. Create the secret
kubectl apply -f ~/operator/examples/kustomization/tenant-kes-encryption/kes-configuration-secret.yaml
  1. Add kes in ~/operator/helm/tenant/templates/tenant.yaml 
spec:
  {{ if dig "kes" "enabled" false . }}
  kes:
    image: {{ .kes.image | quote }}
    replicas: {{ .kes.replicas | int }}
    kesSecret:
      name: {{ .kes.kesSecret.name | quote }}
    imagePullPolicy: {{ .kes.imagePullPolicy | quote }}
    externalCertSecret: {{ .kes.externalCertSecret | quote }}
    clientCertSecret: {{ .kes.clientCertSecret | quote }}
    ## Key name to be created on the KMS, default is "my-minio-key"
    keyName: {{ .kes.keyName | quote }}
    {{- with (dig "resources" (dict) .) }}
    resources:
      {{ toYaml . | nindent 4 }}
    {{- end }}
    {{- with (dig "nodeSelector" (dict) .) }}
    nodeSelector:
      {{ toYaml . | nindent 4 }}
    {{- end }}
    affinity:
      nodeAffinity: { }
      podAffinity: { }
      podAntiAffinity: { }
    tolerations: [ ]
    {{- with (dig "annotations" (dict) .) }}
    annotations:
      {{ toYaml . | nindent 4 }}
    {{- end }}
    {{- with (dig "labels" (dict) .) }}
    labels:
      {{ toYaml . | nindent 4 }}
    {{- end }}
    serviceAccountName: {{ .kes.serviceAccountName | quote }}
    securityContext:
      runAsUser: {{ .kes.securityContext.runAsUser | int }}
      runAsGroup: {{ .kes.securityContext.runAsGroup | int }}
      runAsNonRoot: {{ .kes.securityContext.runAsNonRoot }}
      fsGroup: {{ .kes.securityContext.fsGroup | int }}
  {{ end }}
  1. Set values in ~/operator/helm/tenant/values.yaml 
    ## Define configuration for KES (stateless and distributed key-management system)
    ## Refer https://github.com/minio/kes
    kes:
      enabled: true
      image: "" # minio/kes:v0.17.6
      replicas: 2
      kesSecret:
        name: kes-configuration
      imagePullPolicy: "IfNotPresent"
      externalCertSecret: null
      clientCertSecret: null
      ## Key name to be created on the KMS, default is "my-minio-key"
      keyName: ""
      resources: { }
      nodeSelector: { }
      affinity:
        nodeAffinity: { }
        podAffinity: { }
        podAntiAffinity: { }
      tolerations: [ ]
      annotations: { }
      labels: { }
      serviceAccountName: ""
      securityContext:
        runAsUser: 1000
        runAsGroup: 1000
        runAsNonRoot: true
        fsGroup: 1000
  1. Install the tenant with helm
cd ~/operator/helm
helm install --debug --namespace tenant-ns \
  --create-namespace tenant ./tenant \
  -f ~/operator/helm/tenant/values.yaml
Clone this wiki locally