Automate ssl certificate generation using certbot on HAProxy for wildcard certs

[igobranco]

Previous work:
We already have a PROC for SSL certificate generation using Let's Encrypt.

Motivation:
Currently we are using on nau.edu.pt certificates issued by Sectigo, from the RCTSaai community.
But it was decided from the major players, like Google that manages Chrome and the major Certificate entities, to reduce the max time of certificates ssl validation to 90 days.

We need to automate the SSl certificate generation using CertBot, because the Sectigo Certificate authority supports ACME protocol.

---

We already have been done a POC prof of concept to use the lets encrypt certificates.
This was implemented directly on HAProxy.

On the move to Kubernetes we should the cert-manager, but instead of using the Lets Encrypt servers, we should use the Sectigo servers.

https://share.fccn.pt/sites/rctscertificados/ACME/acme_internal_fccn/#page-toc-10
https://forms.office.com/pages/responsepage.aspx?id=0aBHeQvToEyYmDfUhhxlnO_mc0SdF0JIt4x7BHMOCmJUNFFWMVFHTVpFMFlWR0RaWTVSVkdGM0RNTS4u
https://share.fccn.pt/sites/rctscertificados/ManualUtilizador/#page-toc-3
During the internal FCCN event "Cartas na Mesa" the Emanuel Massano told me that Felipe Santana from ASR already have done that.

I've talked with Filipe Santana from ASR department, he has told that he is emitting the certificates directy using Traefik and not using cert-manager.

[igobranco]

Alternativelly we could use Caddy web server with a shared storage, for example using Redis.

[igobranco]

Duplicate https://github.com/fccn/nau-technical/issues/50

[igobranco]

The Kubernetes ingress for DEV environment was configured to use cert manager.

We still need to issue an wildcard certificate on HAProxy when any service is down.
https://www.haproxy.com/blog/haproxy-and-let-s-encrypt

[igobranco]

We just need to do this on HAProxy for wildcard certificates, when the backends are down.

[igobranco]

Implemented.
Need to configure a Jenkins job or in future run it inside K8s on a dedicated pod.

To clean old certificate files, we need to run manually the follow script:

cd /nau/ops/nau-balancer
find /nau/ops/nau-balancer/certs/ -type f -mtime +15 -exec rm {} \;

[igobranco]

In future we need to execute the acme playbook regularly.

[igobranco]

Configured Jenkins jobs.
When run for DEV environment, it has issued certificates. On July we were on middle of migration to Kubernetes, so didn't run this playbook for DEV.

[igobranco]

Done!

[igobranco]

Jobs running on Jenkins!