Cant configure backup without credentialsSecret

[ItielOlenick]

Report

When not providing credentialsSecret in the s3 section of the deploy/cr.yaml backups can't be taken.

More about the problem

Documentation states:

Using AWS EC2 instances for backups makes it possible to automate access to AWS S3 buckets based on IAM roles for Service Accounts with no need to specify the S3 credentials explicitly.
Following steps are needed to turn this feature on:

• Create the IAM instance profile * and the permission policy within where you specify the access level that grants the access to S3 buckets.
• Attach the IAM profile to an EC2 instance.
• Configure an S3 storage bucket and verify the connection from the EC2 instance to it.
• Do not provide s3.credentialsSecret for the storage in deploy/cr.yaml.

When trying to drop s3.credentialsSecret the operator logs show the following errors:

2024-04-04T08:46:20.926Z	INFO	Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes reqeueuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler	{"controller": "pxcbackup-controller", "namespace": "data", "name": "backup1", "reconcileID": "e1e6a9aa-8aff-4997-abdd-bdcdcbc75730"}
2024-04-04T08:46:20.926Z	ERROR	Reconciler error	{"controller": "pxcbackup-controller", "namespace": "data", "name": "backup1", "reconcileID": "e1e6a9aa-8aff-4997-abdd-bdcdcbc75730", "error": "create backup job: Job.batch \"xb-backup1\" is invalid: [spec.template.spec.containers[0].env[4].valueFrom.secretKeyRef.name: Invalid value: \"\": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'), spec.template.spec.containers[0].env[5].valueFrom.secretKeyRef.name: Invalid value: \"\": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')]", "errorVerbose": "Job.batch \"xb-backup1\" is invalid: [spec.template.spec.containers[0].env[4].valueFrom.secretKeyRef.name: Invalid value: \"\": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'), spec.template.spec.containers[0].env[5].valueFrom.secretKeyRef.name: Invalid value: \"\": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')]\ncreate backup job\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).Reconcile\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227

cr.yaml:

...
backup:
    image: perconalab/percona-xtradb-cluster-operator:main-pxc8.0-backup
    schedule:
      - keep: 3
        name: daily
        schedule: 0 0 * * *
        storageName: s3
    serviceAccountName: test-mysql-percona-s3
    storages:
      s3:
        s3:
          bucket: test-mysql-percona-s3
          region: eu-west-1
        type: s3
...

Steps to reproduce

1. Configure s3 backup according to https://docs.percona.com/percona-operator-for-mysql/pxc/backups-storage.html
2. Create an on-demand backup
3. watch the operator logs

Versions

1. Kubernetes v1.27.9-eks
2. Operator 1.14.0
3. Database perconalab/percona-xtradb-cluster-operator:main-pxc8.0

Anything else?

No response

[hors]

@ItielOlenick as you can see, this error is connected with your storage name. Try to use e.g. aws-s3 like a storage name:

  backup:
    image: perconalab/percona-xtradb-cluster-operator:main-pxc8.0-backup
    storages:
      aws-s3:
        type: s3
        s3:
          region: eu-west-1
          bucket: test-mysql-percona-s3

[ItielOlenick]

@hors Not really.
Look at the error:

Job.batch \"xb-backup1\" is invalid: [spec.template.spec.containers[0].env[4].valueFrom.secretKeyRef.name: Invalid value: \"\"

It clearly tries to extract the secretref from a secret named ""
Issue is with https://github.com/percona/percona-xtradb-cluster-operator/blob/main/pkg/pxc/backup/job.go#L257 not ignoring the fact that no credentialsSecret was provided

[hors]

@ItielOlenick yes, you are right. I have reproduced it. We will fix it in the next release under https://perconadev.atlassian.net/browse/K8SPXC-1382 task.

[ItielOlenick]

@hors Thank you!

Any chance you can have a look at https://perconadev.atlassian.net/browse/PXB-3270 while you are at it?
The docs state that:

Using AWS EC2 instances for backups makes it possible to automate access to AWS S3 buckets based on IAM roles for Service Accounts with no need to specify the S3 credentials explicitly.

That is not quite true. The supported feature is the instance profile credentials retrieval through IMDS and not though IRSA, which makes it somewhat useless in EKS.

[james-callahan]

I too would love to see support for IRSA, i.e. authentication via AWS_WEB_IDENTITY_TOKEN_FILE