CAPI Cluster Autoscaling with Machine Deployments

[alex-dabija]

Story

-As a cluster admin, I want a CAP(O|G|V|VCD) cluster to autoscale depending on the required resources in order to ensure the stability of applications running on the cluster.

Background

CAP(O|G|V|VCD) cluster don't have support for machine pools, which means we can't leverage the cloud provider's support for autoscaling. Machine deployments are supported and cluster-autoscaler does have support for them.

Giant Swarm has been running the cluster-autoscaler on the workload cluster in order to enable the cluster to still scale in case the communication between the management cluster and the workload cluster is severed.

Unfortunately, the cluster-autoscaler needs to run on the management cluster in order to be able to update the workload cluster's machine deployement resources.

Resources

• Cluster Autoscaler documentation;

Stories

Beta Give feedback

1. Cluster Autoscaler Investigation #2618
   area/kaas team/turtles +2
   
2. Upgrade cluster-api apps #2691
   area/kaas team/turtles +2
   
3. Implement cluster autoscaling in machine deployments #2693
   Ready area/kaas team/turtles +3
4. Create cluster autoscaler e2e test suite for machine deployments #2695
   Ready area/kaas team/turtles +3
5. cluster-autoscaler-app should validate values such as newPodScaleUpDelay #2640
   area/kaas team/turtles +2
   
6. https://github.com/giantswarm/giantswarm/issues/27571

[cornelius-keller]

@alex-dabija do we also need a story about making the schedular configurable to optimize for better usage of existing nodes before creating new ones? how is this currently handled in the cloud installations?

[alex-dabija]

We don't have any special configuration for the Kubernetes scheduler to optimize for better usage. We do have the vertical pod autoscaler running by default on vintage clusters which might have an impact on how many pods are running on a node.

This story is only meant to get the cluster autoscaler running on the management cluster to handle autoscaling of machine deployments.

I wouldn't worry for now about packing pods more efficiently on the nodes. I would just keep things simple for now.

[gawertm]

@alex-dabija I changed ownership from Hydra to Rocket, as most likely Rocket will implement first. We are already looking at the documentation and trying it out

[gawertm]

blocked until https://github.com/giantswarm/giantswarm/issues/23820

[Rotfuks]

Done in Clippy via: #1793

[brinker211]

@alex-dabija I see the new tickets for CAPA and CAPZ. Is there still a ticket for CAPG? This was the provider this ticket originally started with.

[brinker211]

@gawertm is the referenced internal ticket https://github.com/giantswarm/giantswarm/issues/23820 for CAPG? I see Hydra and Clippy referenced for CAPA and CAPZ but looking for the status of CAPG as well. Thanks!

[gawertm]

hi @brinker211 yes this was for CAPG but also for the Rocket providers. We initially planned to help Hydra with that. Eventhough its not a Rocket priority anymore

[gawertm]

@Rotfuks would the autoscaler topic go Turtles? lets move it form rocket backlog then :)
cc @alex-dabija

[alex-dabija]

@Rotfuks would the autoscaler topic go Turtles? lets move it form rocket backlog then :) cc @alex-dabija

Yes, Turtles makes sense. It would be great to have a common autoscaling solution for all providers (at least for CAPZ, CAPV, CAPVCD). CAPA is still using machine pools which require a different implementation.

[Rotfuks]

Here's the information transfer from the CAPZ specific Autoscaling Epic:

We already had a first discussion on it: https://gigantic.slack.com/archives/C04887ZSU20/p1670926088947149
Conclustion:
All infos around cluster-autoscaler in machinepools issues can be found here:
https://github.com/giantswarm/giantswarm/issues/19313

• We should document all further technical infos into that ticket
  CAPZ already has a feature merged for managed machine pools:
  When creating AKS clusters using autoscaler enabled, do not make an update api call to agentpool service based on difference in node count kubernetes-sigs/cluster-api-provider-azure#2444
• But it is currently being refactored to match the overall CAPI approach: ✨ MachinePool annotation for externally managed autoscaler kubernetes-sigs/cluster-api#7107
  CAPZ already has an issue open for support of autoscaling on unmanaged machine pools:
  feat: respect externally managed annotation on unmanaged MachinePools kubernetes-sigs/cluster-api-provider-azure#2588
• Let's also remember to double check scaling to 0 support when the code support lands
• Helpful Slack Thread in the CAPZ upstream community: https://kubernetes.slack.com/archives/CEX9HENG7/p1674083817745989
• As Vintage is looking into Karpenter (https://github.com/giantswarm/flexshopper/issues/359) this might also be an interesting tool for this capability.

[alex-dabija]

@Rotfuks I would keep this issue scoped to autoscaling for machine deployments because of our current focus on CAPA. I moved the #2692 to #1798. It should make it easier to track the status of CAPA for our first release.