Kubernetes MinIO Operator installation issue ""operator TLS secret not found: secrets "operator-tls" not found""

[ravipokala]

NOTE

All GitHub issues are addressed on a best-effort basis at MinIO's sole discretion. There are no Service Level Agreements (SLA) or Objectives (SLO). Remember our Code of Conduct when engaging with MinIO Engineers and the larger community.

For urgent issues (e.g. production down, etc.), subscribe to SUBNET for direct to engineering support.
I am trying to install MinIO Operator in K8S with kubectl minio plug-in. Operator and related resources are installed but not able create the tenants because of issue with TLS, Pod logs show the error message operator TLS secret not found: secrets "operator-tls" not found

Operator CSR is approved.
k get csr
NAME AGE SIGNERNAME REQUESTOR CONDITION
operator-minio-operator-csr 2m4s kubernetes.io/legacy-unknown system:serviceaccount:minio-operator:minio-operator Approved

Expected Behavior

Current Behavior

k logs pod/minio-operator-765bf7584f-p2kk8
I1022 06:34:27.872349 1 main.go:70] Starting MinIO Operator
E1022 06:34:28.390319 1 main.go:167] Error updating CRD with caBundle: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "tenants.minio.min.io": the object has been modified; please apply your changes to the latest version and try again
I1022 06:34:28.390376 1 main.go:169] caBundle on CRD updated
I1022 06:34:28.391414 1 main-controller.go:244] Setting up event handlers
I1022 06:34:28.391695 1 leaderelection.go:248] attempting to acquire leader lease minio-operator/minio-operator-lock...
I1022 06:34:28.397812 1 main-controller.go:496] minio-operator-765bf7584f-zz7g9: is the leader, removing any leader labels that I 'minio-operator-765bf7584f-p2kk8' might have
ravikp@engtech-dev-01 /homes/ravikp/k8/charts>k logs pod/minio-operator-765bf7584f-zz7g9
I1022 06:34:27.871084 1 main.go:70] Starting MinIO Operator
I1022 06:34:28.319476 1 main.go:169] caBundle on CRD updated
I1022 06:34:28.320264 1 main-controller.go:244] Setting up event handlers
I1022 06:34:28.320530 1 leaderelection.go:248] attempting to acquire leader lease minio-operator/minio-operator-lock...
I1022 06:34:28.337811 1 leaderelection.go:258] successfully acquired lease minio-operator/minio-operator-lock
I1022 06:34:28.337901 1 main-controller.go:478] minio-operator-765bf7584f-zz7g9: I am the leader, applying leader labels on myself
I1022 06:34:28.337981 1 main-controller.go:390] Waiting for API to start
I1022 06:34:28.338018 1 main-controller.go:382] Starting HTTP Upgrade Tenant Image server
I1022 06:34:28.354749 1 main-controller.go:351] Using Kubernetes CSR Version: v1beta1
I1022 06:34:28.376184 1 operator.go:94] operator TLS secret not found: secrets "operator-tls" not found
W1022 06:34:28.378842 1 warnings.go:70] certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
W1022 06:34:28.396406 1 warnings.go:70] certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
I1022 06:34:28.397936 1 operator.go:94] operator TLS secret not found: secrets "operator-tls" not found
W1022 06:34:28.399456 1 warnings.go:70] certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
W1022 06:34:28.404946 1 warnings.go:70] certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
W1022 06:34:28.416114 1 warnings.go:70] certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
I1022 06:34:28.416291 1 csr.go:183] Start polling for certificate of csr/operator-minio-operator-csr, every 5s, timeout after 20m0s
W1022 06:34:33.420469 1 warnings.go:70] certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
W1022 06:34:38.419672 1 warnings.go:70] certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
W1022 06:34:43.419079 1 warnings.go:70] certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest

Possible Solution

Steps to Reproduce (for bugs)

Context

Regression

Your Environment

• Version used (minio --version):
• Server setup and configuration:
• Operating System and version (uname -a):

[harshavardhana]

@ravipokala what is the operator version?

[ravipokala]

v4.5.2

[ravipokala]

@harshavardhana : Minio version is v4.5.2

[vadmeste]

@ravipokala it looks like the operator TLS certificate is only approved but not issued. Can you describe your CSR, kubectl describe csr operator-minio-operator-csr ?

[ravipokala]

Please find the requested information below.

kubectl describe csr operator-minio-operator-csr
Name: operator-minio-operator-csr
Labels:
Annotations:
CreationTimestamp: Mon, 24 Oct 2022 14:45:57 -0700
Requesting User: system:serviceaccount:minio-operator:minio-operator
Signer: kubernetes.io/legacy-unknown
Status: Approved
Subject:
Common Name: system:node:operator.minio-operator.svc
Serial Number:
Organization: system:nodes
Subject Alternative Names:
DNS Names: operator
operator.minio-operator.svc
operator.minio-operator.svc.cluster.local
Events:

Our K8S cluster kube-controller-manager can't include signing key info as described @ https://min.io/docs/minio/kubernetes/upstream/operations/installation.html?ref=docs-redirect#kubernetes-tls-certificate-api
Instead, we have a cert-manager.

[ravipokala]

Hi,
I did follow the instructions @ https://github.com/minio/operator/blob/master/docs/tls.md#using-cert-manager to use cert-manager to generate the certificate. MinIO operators fail to create tenants below is the error message. quick response is much appreciated.

k logs pod/minio-operator-565bd6644-5ghzk
I1026 21:35:44.468941 1 main.go:70] Starting MinIO Operator
I1026 21:35:44.946086 1 main.go:169] caBundle on CRD updated
I1026 21:35:44.946710 1 main-controller.go:244] Setting up event handlers
I1026 21:35:44.946867 1 leaderelection.go:248] attempting to acquire leader lease minio-operator/minio-operator-lock...
I1026 21:35:44.957288 1 leaderelection.go:258] successfully acquired lease minio-operator/minio-operator-lock
I1026 21:35:44.957416 1 main-controller.go:478] minio-operator-565bd6644-5ghzk: I am the leader, applying leader labels on myself
I1026 21:35:44.957438 1 main-controller.go:390] Waiting for API to start
I1026 21:35:44.957469 1 main-controller.go:382] Starting HTTP Upgrade Tenant Image server
I1026 21:35:45.061226 1 main-controller.go:351] Using Kubernetes CSR Version: v1beta1
I1026 21:35:45.068168 1 main-controller.go:355] Starting HTTPS API server
I1026 21:35:45.068232 1 main-controller.go:393] Waiting for Upgrade Server to start
I1026 21:35:45.068253 1 main-controller.go:397] Starting Tenant controller
I1026 21:35:45.068264 1 main-controller.go:400] Waiting for informer caches to sync
I1026 21:35:45.068290 1 main-controller.go:405] Starting workers
I1026 21:35:45.073224 1 monitoring.go:100] 'mysqldb/test' no pool is initialized
2022/10/26 21:35:45 http: TLS handshake error from 10.20.11.0:4163: remote error: tls: bad certificate
2022/10/26 21:35:45 http: TLS handshake error from 10.20.240.0:16256: remote error: tls: bad certificate
2022/10/26 21:35:46 http: TLS handshake error from 10.20.11.0:4639: remote error: tls: bad certificate
2022/10/26 21:35:46 http: TLS handshake error from 10.20.240.0:9157: remote error: tls: bad certificate
2022/10/26 21:35:47 http: TLS handshake error from 10.20.11.0:54817: remote error: tls: bad certificate
2022/10/26 21:35:47 http: TLS handshake error from 10.20.240.0:43275: remote error: tls: bad certificate
2022/10/26 21:35:48 http: TLS handshake error from 10.20.11.0:38855: remote error: tls: bad certificate
2022/10/26 21:35:48 http: TLS handshake error from 10.20.240.0:5937: remote error: tls: bad certificate
2022/10/26 21:35:49 http: TLS handshake error from 10.20.11.0:56931: remote error: tls: bad certificate
2022/10/26 21:35:49 http: TLS handshake error from 10.20.240.0:35385: remote error: tls: bad certificate
W1026 21:35:49.982628 1 warnings.go:70] certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
W1026 21:35:49.990565 1 warnings.go:70] certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
E1026 21:35:49.990630 1 main-controller.go:596] error syncing 'mysqldb/test': missing 'public.crt' in / secret

Cert-Manager generates certificate with "tls.crt" I don't see a way change that to "public.crt"

[ravipokala]

Any pointer to fix the issue is much appreciated.

[PatiUdayKiran-ab-scm]

Same issue.
I1102 07:23:19.195899 1 monitoring.go:100] 'minio/minio' no pool is initialized E1102 07:23:24.103631 1 main-controller.go:596] error syncing 'minio/minio': missing 'public.crt' in minio/operator-tls secret, re-creating it

[chancez]

I'm also seeing errors about the public.crt when upgrading from 5.0.6 to 5.0.10 when using cert-manager for certificates.

[cesnietor]

@chancez could you please open a separate issue for your case since it's not the same.
Closing this one since there is no longer an operator-tls secret in operator 5.0.10.

[chancez]

Fair enough. Filed #1839