Unable to use Azure Workload Identity for storage access for backups

[wonko]

Report

When using Azure Workload Identity (https://azure.github.io/azure-workload-identity/docs/introduction.html), no credentials or secrets are needed, as the pods are provided with a token in a well-known place, and the auth chain of the Azure libs will pick it up.

I wished to use this for the backup process, by providing a specific service account with the needed settings, and leaving the further configuration for the access unset. However, leaving those values empty/undefined, the operator gives the following error:

The error is

pxc-operator-759d84fbc9-mbnjb percona-xtradb-cluster-operator 2024-08-19T10:23:41.951Z	ERROR	Reconciler error	{"controller": "pxcbackup-controller", "namespace": "percona", "name": "ad-hoc-backup", "reconcileID": "5896341c-ce47-4080-8850-2299a61e828c", "error": "create backup job: Job.batch \"xb-ad-hoc-backup\" is invalid: [spec.template.spec.containers[0].env[4].valueFrom.secretKeyRef.name: Invalid value: \"\": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'), spec.template.spec.containers[0].env[5].valueFrom.secretKeyRef.name: Invalid value: \"\": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')]", "errorVerbose": "Job.batch \"xb-ad-hoc-backup\" is invalid: [spec.template.spec.containers[0].env[4].valueFrom.secretKeyRef.name: Invalid value: \"\": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'), spec.template.spec.containers[0].env[5].valueFrom.secretKeyRef.name: Invalid value: \"\": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')]\ncreate backup job\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).Reconcile\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"}

More about the problem

This is what my snippet for the backups looks like:

backup:
  serviceAccountName: backups
  storages:
    azure-blob:
      type: azure
      azure:
        container: [redacted]
  schedule:
    - name: "daily-backup"
      schedule: "0 0 * * *"
      keep: 30
      storageName: azure-blob

Steps to reproduce

1. Deploy a database with above backup section
2. Create an ad-hoc backup to trigger faster (or wait til midnight ;-) )
3. Observe operator and see the error

Versions

1. Kubernetes AKS 1.29.7
2. Operator percona/percona-xtradb-cluster-operator:1.14.0
3. Database percona/percona-xtradb-cluster:8.0.36-28.1

Anything else?

It's pretty much the same as #1687 but that's for AWS. Would be logical to solve those two together...