Change handling of cgroups v1/v2 implementation in CAPA

[T-Kukawka]

User Story

Current implementation of cgroups v1 support in CAPA is performed per whole cluster basis based on a flag on cluster chart level: https://github.com/giantswarm/cluster/blob/main/helm/cluster/values.yaml#L84. In Vintage we have been able to configure the explicit use of v1 or v2 version per Node Pool.

With recent migration we have ran into issue that not all workloads were migrated to use v2 cgroups, causing issues for workloads that are not yet compatible with the solution. In the same time, for CAPA clusters configured to use v1 version, we are missing the labels on the nodes themselves that workloads use to be scheduled on supported version.

Initial solution was based on an assumption that all customers will be able to migrate away on time from the deprecated v1 version, which has proven to be wrong. Moreover, not to stop any future migrations based, we would like to add more configurability for Node Pools such that the workloads can be distributed across the nodes which either support v1 or v2 version.

Acceptance Criteria

• Add CAPA support for toggling usage of either cgroups v1 or v2 based on customer needs
• Add appropriate labels to nodes based on the cgroups version they are supporting
• Extend migration-cli to support passing the NodePool configuration from Vintage to CAPA
• Add a note for the deprecated internal value for enabling cgroups v1