Umbrella issue: mTLS with cert-manager

[alexandrevilain]

Temporal operator v0.19.0 will be released in the coming hours/days.

It looks like mTLS with cert-manager is having some issues that I would like to fix for v0.21.0 release (at least before 1.0.0 and before v1 API, in case of breaking API changes required).

Here is a list of open or closed issues that can help us find what's going wrong:

• Can't connect via client to frontend service with cert-manager mTLS certificate #722
• Rotated certificates cause all services to crashloop/failure to refresh #708
• Failed to update cluster to enable mTLS with cert-manager #472
• mTLS options show diff in argocd #501
• [cert-manager] invalid peer certificate: UnknownIssuer #509

IMO the work should be done in the following step:

• Fixing the client certificates issues
• Fixing the certificate rotation issues
• Improving the UI/UX of the feature

As a reminder here is the current certificate architecure diagram: https://temporal-operator.pages.dev/features/mtls/cert-manager/

Things to consider:

• https://github.com/cert-manager/csi-driver

cc @yujunz @ElanHasson @plaisted

[yujunz]

cc @guritinvda

[ElanHasson]

@alexandrevilain thank you for addressing this holistically! much appreciated.