Document VPA behavior for sidecar containers and how to change it

Signed-off-by: Omer Aplatony <[email protected]>

vertical-pod-autoscaler/docs/sidecar-containers.md

@@ -0,0 +1,173 @@
+# VPA Sidecar Container Management
+
+The Vertical Pod Autoscaler (VPA) has specific behavior when dealing with sidecar containers that are injected into pods via admission webhooks. This document explains the default behavior and how to customize it for your needs.
+
+## Understanding VPA and Container Policies
+
+### Default Container Policies
+
+To understand why sidecar handling is important, let's first look at how VPA manages containers by default. This default behavior is at the root of why special consideration is needed for sidecars.
+
+When you create a VPA resource for a pod, it automatically attempts to manage ALL containers in that pod - not just the ones you explicitly configure. This happens because:
+
+1. VPA applies a default `containerPolicy` with `mode: Auto` to any container not explicitly configured
+2. This automatic inclusion means VPA will try to manage resources for every container it sees
+3. Even sidecars injected into the pod will fall under VPA's management unless special steps are taken
+
+This default "manage everything" approach can cause problems with sidecars because:
+- Sidecar containers often have their own resource requirements set by their injection webhooks
+- VPA's automatic management may conflict with these requirements
+- Without proper handling, this can lead to resource conflicts and pod instability
+
+Example of a VPA resource that explicitly configures one container but affects all:
+
+```yaml
+apiVersion: autoscaling.k8s.io/v1
+kind: VerticalPodAutoscaler
+metadata:
+  name: my-app-vpa
+spec:
+  targetRef:
+    apiVersion: "apps/v1"
+    kind: Deployment
+    name: my-app
+  updatePolicy:
+    updateMode: "Auto"
+  resourcePolicy:
+    containerPolicies:
+    - containerName: main-container
+      minAllowed:
+        cpu: 100m
+        memory: 50Mi
+# Note: Other containers will still get the default Auto mode
+```
+
+## Default Behavior: Ignoring Sidecar Containers
+
+### The vpaObservedContainers Annotation
+
+VPA uses a special annotation to track which containers were present in the pod before any webhook injections:
+
+```yaml
+metadata:
+  annotations:
+    vpaObservedContainers: "main-container,logging-sidecar"
+```
+
+This annotation is crucial because:
+1. It's added by the VPA admission controller webhook
+2. Only containers listed in this annotation will be managed by VPA
+3. The annotation must be added before sidecar injection for the default behavior to work
+
+### Webhook Ordering Importance
+
+The order of webhook execution is determined alphabetically by webhook names. For example:
+
+```yaml
+webhooks:
+- name: a.sidecar-injector.kubernetes.io  # Executes first
+- name: b.vpa.kubernetes.io               # Executes second
+- name: c.another-sidecar.kubernetes.io   # Executes third
+```
+
+### The Eviction Loop Problem
+
+Without proper handling of sidecar containers, the following problematic sequence could occur:
+
+1. VPA admission controller sets resources for all containers
+2. Sidecar webhook injects a new container with its own resource requirements
+3. The pod starts with mismatched resources
+4. VPA detects the mismatch and evicts the pod
+5. The process repeats, creating an endless loop
+
+## Customizing VPA Behavior for Sidecar Containers
+
+### Option 1: Webhook Ordering
+
+To have VPA manage sidecar resources, ensure your webhook names follow this pattern:
+
+```yaml
+webhooks:
+- name: sidecar-injector.kubernetes.io    # Executes first
+- name: zz.vpa.kubernetes.io              # Executes last
+```
+
+This ensures:
+1. Sidecars are injected first
+2. VPA sees the complete pod with all sidecars
+3. The `vpaObservedContainers` annotation includes all containers
+
+### Option 2: Webhook Reinvocation
+
+Configure the VPA webhook to reinvoke after sidecar injection:
+
+```yaml
+apiVersion: admissionregistration.k8s.io/v1
+kind: MutatingWebhookConfiguration
+metadata:
+  name: vpa-webhook-config
+webhooks:
+- name: vpa.kubernetes.io
+  reinvocationPolicy: IfNeeded
+  rules:
+    - apiGroups: [""]
+      apiVersions: ["v1"]
+      operations: ["CREATE", "UPDATE"]
+      resources: ["pods"]
+```
+
+This configuration:
+1. Allows the VPA webhook to be called multiple times
+2. Ensures resource recommendations are applied after all sidecars are injected
+3. Prevents the eviction loop problem
+
+## Implementation Examples
+
+### Example 1: Istio Sidecar Integration
+
+When using Istio, which injects proxy sidecars:
+
+```yaml
+# Istio sidecar injector webhook
+apiVersion: admissionregistration.k8s.io/v1
+kind: MutatingWebhookConfiguration
+metadata:
+  name: istio-sidecar-injector
+webhooks:
+- name: a.sidecar-injector.istio.io  # Alphabetically first
+
+# VPA webhook configuration
+apiVersion: admissionregistration.k8s.io/v1
+kind: MutatingWebhookConfiguration
+metadata:
+  name: vpa-webhook-config
+webhooks:
+- name: z.vpa.kubernetes.io          # Alphabetically last
+  reinvocationPolicy: IfNeeded
+```
+
+### Example 2: Custom VPA Configuration with Sidecars
+
+```yaml
+apiVersion: autoscaling.k8s.io/v1
+kind: VerticalPodAutoscaler
+metadata:
+  name: web-app-vpa
+spec:
+  targetRef:
+    apiVersion: "apps/v1"
+    kind: Deployment
+    name: web-app
+  updatePolicy:
+    updateMode: "Auto"
+  resourcePolicy:
+    containerPolicies:
+    - containerName: main-container
+      minAllowed:
+        cpu: 100m
+        memory: 128Mi
+    - containerName: logging-sidecar  # Explicitly configure sidecar
+      minAllowed:
+        cpu: 50m
+        memory: 64Mi
+```