Deployment: Make Nirin Runners Ephemeral and Autoscale using Kubernetes

[CodeGat]

Background
Having 4 permanent runners is cool and all, but we need to think about:

• What happens if we need more runners at some point in the future - are we going to register these all individually? What about when PATs expire, we would need to go through each runner and reconfigure them, as noted in Set up Github Actions runners on NCI for deployment workflows build-ci#5 (comment)
• What happens if we need more runners during a job? For example, a highly-parallelized job.
• What happens if we don't need all the runners at any one time? There isn't much point in keeping them all listening.

The Solution
Utilize runners that are ephemeral (the runner is created and destroyed as needed) and autoscaling (the amount of runners at any one time conform to the workload). We can achieve this using Kubernetes and a GitHub-created runner controller called the Actions Runner Controller.
See here: https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/autoscaling-with-self-hosted-runners

Considerations

• Can we install Kubernetes on Nirin? If we can install Docker, surely we can install Kubernetes.
• What happens if we have a highly parallelized job that spawns many runners, that could make all runners extremely slow?

Tagging @jo-basevi as she has had some Kubernetes experience

References ACCESS-NRI/build-ci#5

[CodeGat]

Update for today:

• Installed Kubernetes on Nirin (minikube), use kubectl via alias kubectl="minikube kubectl --" in ~/.bashrc.
• Added actions runner controller helm repository (but didn't install it yet)
• Generated my own PAT with scopes listed here for PATs (https://github.com/actions/actions-runner-controller/blob/master/docs/authenticating-to-the-github-api.md), put in Bitwarden under 'Nirin Kubernetes Self-hosted ARC Runner PAT'