Prow: Migrate to ExternalSecrets

[lentzi90]

Our current setup requires admins to create files with credentials and other secrets locally in the kustomizations before applying changes. This process is error prone and makes automation hard since an automation tool would also need to have access to all secrets even when only touching non-sensitive parts.

We should migrate to ExternalSecrets instead. This is the same that is used for k/k prow. It has integration with OpenStack so we should be able to store the secrets there. In practice what we need to do is to remove the secrets from the kustomizations and introduce ExternalSecrets instead. The ExternalSecrets are just references to secrets stored in the external storage (openstack for us). So they can be committed in git. Admins would then need to make sure the secrets are available in openstack before attempting a deployment.

[lentzi90]

/triage accepted

[tuminoid]

This is a stepping stone for automating Prow cluster updates via GitOps 👍

[metal3-io-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues will close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[lentzi90]

/remove-lifecycle stale