tenant install failed - references non-existent secret key: public.crt

[chancez]

Expected Behavior

We were using minio-operator 5.0.6 in CI using the operator and tenant helm charts. We configure minio with TLS using cert-manager based on the following the docs. This works correctly.

Current Behavior

After upgrading to 5.0.10 CI started failing while waiting for the minio tenant statefulset to become ready. In the k8s events for the minio tenant statefulset, we see the following warnings:

  Normal   Scheduled    9m34s                 default-scheduler  Successfully assigned minio/quickstart-ss-0-0 to kind-control-plane
  Warning  FailedMount  80s (x12 over 9m34s)  kubelet            MountVolume.SetUp failed for volume "quickstart-tls" : references non-existent secret key: public.crt
  Warning  FailedMount  47s (x4 over 7m31s)   kubelet            Unable to attach or mount volumes: unmounted volumes=[quickstart-tls], unattached volumes=[], failed to process volumes=[]: timed out waiting for the condition

Possible Solution

Make public.crt optional in the volume mount, or remove it entirely.

Steps to Reproduce (for bugs)

1. Install cert-manager, and follow the docs for setting up minio-operator w/ cert-manager
2. Install minio-operator
3. Install minio tenant using helm using the values.yaml for minio-tenant that are provided at the bottom of the issue
4. See minio tenant never becomes ready

Context

We use minio for testing our product with s3 compatible object storage, and in our Quickstart docs. This issue prevents us from upgrading minio.

Regression

Is this issue a regression? Yes
If Yes, optionally please include minio-operator version caused the regression: It regressed somewhere between 5.0.6 and 5.0.10.

Your Environment

• Version used (minio-operator): 5.0.10
• Environment name and version (e.g. kubernetes v1.17.2): kind v0.20.0, Kubernetes 1.27.3.
• Server type and version:
• Operating System and version (uname -a): ubuntu 22.04 via GitHub hosted runners
• Link to your deployment file:

secrets:
  enabled: true
  name: minio-secret
  accessKey: minio
  secretKey: quickstart
tenant:
  name: quickstart
  configuration:
    name: minio-secret
  pools:
    - servers: 1
      volumesPerServer: 4
      size: 2Gi
      storageClassName: standard
  buckets:
    - name: "timescape"
    - name: "clickhouse"
  certificate:
    externalCertSecret:
      - name: tls-minio
        type: kubernetes.io/tls
  kes:
    configuration: false
  log:
    disabled: true
  prometheus:
    disabled: true

[cniackz]

Sorry guys, yes this is broken. I can reproduce it locally with our kustomize example published by Lenin:

MountVolume.SetUp failed for volume "myminio-tls" : references non-existent secret key: public.crt

Tomorrow I will continue with it and fix it

[cniackz]

[cniackz]

For now testing notes at my repo: https://github.com/cniackz/public/wiki/cert%E2%80%90manager, will fix and move the notes to minio wiki once this is working again.

[cniackz]

Thank you @chancez for reporting this issue 👍

[cniackz]

@chancez the solution I found is simple, you just need to delete operator-ca-tls secret from minio-operator ns and from tenant's ns. Then regenerate it with pubilc.crt instead, here the PR:

#1842

[cniackz]

@chancez the page I am updating is: https://github.com/minio/operator/blob/master/docs/cert-manager.md#create-operator-ca-tls-secret where the secret has now to contain public.crt to work for cert-manager

[chancez]

@cniackz I don't really understand how the operator secret is relevant here. The issue is the TLS secret the tenant pod is mounting.

I understand that the operator should have the CA so it can communicate securely with the tenant, that makes sense, but seems orthogonal to the issue of the tenant's volume mount issue.

[chancez]

Also why public.crt instead of ca.crt? The latter is somewhat more 'standard' (though there is no standard naming convention, but I'd argue ca.crt is more common than public.crt.

[chancez]

Okay, after looking at the tenant pod, I see why you're suggesting that. The tenant pod is mounting the operator-ca-tls.

Why though? Does the tenant talk to the operator? Why does it need a CA? The docs say

Copy the cert-manager CA from the tenant certificate, this will allow Operator to trust the cert-manager CA and allow Operator to trust the Tenant certificate

This would mean the operator needs the CA, not the tenant, so again: why is the tenant mounting the operator-ca-tls secret? Additionally, my operator pod doesn't even mount the operator-ca-tls secret, so I'm even more confused.

Here's the tenant secret volume mount for reference:

    - name: quickstart-tls
      projected:
        defaultMode: 420
        sources:
        - secret:
            items:
            - key: tls.crt
              path: public.crt
            - key: tls.key
              path: private.key
            - key: tls.crt
              path: CAs/public.crt
            name: tls-minio
        - secret:
            items:
            - key: public.crt
              path: CAs/operator-ca.crt
            name: operator-ca-tls

[chancez]

I can confirm the suggested change in the docs fixes it, but I still have the questions about why the why the tenant is mounting the CA instead of the operator.

[chancez]

Okay, so while this fixes the tenant starting, after making these adjustments, the operator no longer can validate the certificate of the minio tenant. So your suggested fix, fixes the tenant being able to start, but breaks the operator's ability to verify the tenant's certificate.

2023-10-31T17:57:27.517977709Z I1031 17:57:27.517741       1 event.go:298] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"quickstart", UID:"f5d00c0c-c01a-4262-b780-4b860484ff8d", APIVersion:"minio.min.io/v2", ResourceVersion:"2490", FieldPath:""}): type: 'Warning' reason: 'BucketsCreatedFailed' Buckets creation failed: Put "https://minio.minio.svc.cluster.local/timescape/": tls: failed to verify certificate: x509: certificate signed by unknown authority
2023-10-31T17:58:27.580430356Z E1031 17:58:27.580301       1 main-controller.go:697] error syncing 'minio/quickstart': Put "https://minio.minio.svc.cluster.local/timescape/": tls: failed to verify certificate: x509: certificate signed by unknown authority
2023-10-31T17:58:27.580952566Z I1031 17:58:27.580540       1 event.go:298] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"quickstart", UID:"f5d00c0c-c01a-4262-b780-4b860484ff8d", APIVersion:"minio.min.io/v2", ResourceVersion:"2628", FieldPath:""}): type: 'Warning' reason: 'BucketsCreatedFailed' Buckets creation failed: Put "https://minio.minio.svc.cluster.local/timescape/": tls: failed to verify certificate: x509: certificate signed by unknown authority

[chancez]

So I tried creating the secret with both ca.crt and public.crt:

kubectl -n minio-operator create secret generic operator-ca-tls --from-file=ca.crt=ca-cert.pem --from-file=public.crt=ca-cert.pem

But that wasn't sufficient, the operator is still unable to verify the tenant's certificate.

[cniackz]

In version Version 5.0.11 of Operator, this issue should no longer persist.

[bendemott]

This problem still exists in 5.0.15

[larssb]

Yeah still happens in v6.0.3 as well.

[rolinh]

That's because the minio image in v6.0.3 is not recent enough. It should work if you override it with tag RELEASE.2024-08-29T01-40-52Z or newer (tenant.image.tag). Or wait for v6.0.4 to be released.

[larssb]

@rolinh thank you. Thought that the needed fix was there long time ago now that it was declared fixed mid May this year.

Thank you very much for the info.