Decouple machine_controller & cloud-init script & VM image

[erkanerol]

Is your feature request related to a problem? Please describe.

Current status

vcdmachine_controller merges cloud-init script (provisioned by kubeadmconfig controller) and an embedded script in the controller. The embedded script has some additional steps for metering, proxy configuration, and status reporting. These steps are run as part of node initialization processes and nodes report their phases via Cloud Director guestinfo mechanism. The controller has some logic to watch and to wait the phases before marking VCDMachines as ready.

Problems

1. Users cannot use any CAPI compatible VM image since the cloud-init script requires to have vmtoolsd in the VM image.

2. It is not possible to configure the customization script since it is embedded in the controller and there is no interface to change it.

3. Phase logic in the embedded cloud-init script and in the controllers are highly coupled. You have to keep them aligned. It is dirty and hard to maintain.

4. Phase checks increase the bootstrap duration and require so many re-queuing during reconciliation. That makes machine provisioning slower.

Describe the solution you'd like

Removal of phases

According to CAPI specs, infra providers are not responsible for checking the status of cloud-init script. It is OK to observe VM is up&running to mark infra machine as Ready. machine_controller of CAPI has already some checks to mark Machine CRs as Running. See

Removal of embedded script

If additional steps are required during node bootstrap for some users, they can provide additional steps via preKubeadmCommands and postKubeadmCommands . See. Therefore, the embedded script can be removed.

Describe alternatives you've considered

Customization of phases and embedded script

We can introduce new configuration mechanisms to customize the phases and the embedded script so that users can use their own images. Note that we have to do both of them at the same time. Otherwise, one will be limited by the other one. For example, if it is possible to customize only embedded script, you have to put current phases to that script.

It is possible to implement these features in a backward compatible way to not change the behavior of CAPVCD for existing clusters.

1. New fields in CRs

We can introduce an array in VCDMachine spec to declare expected phases. The controller can respect this array if it is not empty.

We can introduce a new secret reference in VCDMachine spec to customize the embedded script. The controller can use the current one if this is empty.

Personally, I wouldn't prefer this since it changes CRDs and it will introduce new burdens in the long run since it will be a part of the API.

2. New flags in the operator

Instead of introducing new fields in the CRS, we can introduce new flags on the operator side. A flag for phase list and a flag for script path. Cluster admins (not CR owners) can configure the operator to customize phases and the script (by creating a configmap and mounting the operator pod). In this setup, VM images will have to be compatible with the script and phases but I think it is OK to allow cluster admins to introduce such a limitation.