VPA daemonset recommendations per-pod based on node metadata #5928
Description
Which component are you using?:
vertical pod autoscaler
Is your feature request designed to solve a problem? If so describe the problem this feature should solve.:
Some daemonsets are comprised of pods which have variable resource needs depending on the node they run on, and by their nature they cannot horizontally scale out of this problem.
Consider the case where a kube cluster is running a cluster autoscaler that provisions all manner of different node types based on cheapest-available capacity (e.g., karpenter using AWS spot).
In the case of dramatically variable node sizes, a pod that's a member of the datadog agent daemonset will require more resources to handle an instance with more pods on it when compared to a member of that same daemonset running on a tiny instance with only a few pods.
Describe the solution you'd like.:
I would like the VPA to (optionally) provide recommendations along an extra dimension for DaemonSets, such as ENI max pods for the host, and size DS pods individually based on this dimension. The recommender might suggest a memory configuration of any given pod based on historical memory_consumed/node_max_pods instead of a single memory value across the daemonset.
Describe any alternative solutions you've considered.:
The alternative is to overprovision the daemonset by a large margin on small instances, or to limit cluster node variability.
Additional context.:
Running on AWS EKS 1.24 with Karpenter.