From 20de7b7555641e7625a3a7c1687612c9c7c1bfa1 Mon Sep 17 00:00:00 2001 From: Arvind Thirumurugan Date: Fri, 24 Jan 2025 01:28:40 -0800 Subject: [PATCH] docs: Eviction & Placement Disruption budget (#1007) --- .../EvictionAndDisruptionBudget/README.md | 71 +++++++ docs/concepts/README.md | 3 + docs/howtos/README.md | 8 +- .../eviction-placement-disruption-budget.md | 181 ++++++++++++++++++ examples/eviction/clusterpdb.yaml | 4 +- examples/eviction/eviction.yaml | 4 +- 6 files changed, 266 insertions(+), 5 deletions(-) create mode 100644 docs/concepts/EvictionAndDisruptionBudget/README.md create mode 100644 docs/howtos/eviction-placement-disruption-budget.md diff --git a/docs/concepts/EvictionAndDisruptionBudget/README.md b/docs/concepts/EvictionAndDisruptionBudget/README.md new file mode 100644 index 000000000..b0d95b887 --- /dev/null +++ b/docs/concepts/EvictionAndDisruptionBudget/README.md @@ -0,0 +1,71 @@ +# Eviction & Placement Disruption Budget + +This document explains the concept of `Eviction` and `Placement Disruption Budget` in the context of the fleet. + +## Overview + +`Eviction` provides a way to force remove resources from a target cluster once the resources have already been propagated from the hub cluster by a `Placement` object. +`Eviction` is considered as an voluntary disruption triggered by the user. `Eviction` alone doesn't guarantee that resources won't be propagated to target cluster again by the scheduler. +The users need to use [taints](../../howtos/taint-toleration.md) in conjunction with `Eviction` to prevent the scheduler from picking the target cluster again. + +The `Placement Disruption Budget` object protects against voluntary disruptions. + +The only voluntary disruption that can occur in the fleet is the eviction of resources from a target cluster which can be achieved by creating the `ClusterResourcePlacementEviction` object. + +Some cases of involuntary disruptions in the context of fleet, +- The removal of resources from a member cluster by the scheduler due to scheduling policy changes. +- Users manually deleting workload resources running on a member cluster. +- Users manually deleting the `ClusterResourceBinding` object which is an internal resource the represents the placement of resources on a member cluster. +- Workloads failing to run properly on a member cluster due to misconfiguration or cluster related issues. + +For all the cases of involuntary disruptions described above, the `Placement Disruption Budget` object does not protect against them. + +## ClusterResourcePlacementEviction + +An eviction object is used to remove resources from a member cluster once the resources have already been propagated from the hub cluster. + +The eviction object is only reconciled once after which it reaches a terminal state. Below is the list of terminal states for `ClusterResourcePlacementEviction`, +- `ClusterResourcePlacementEviction` is valid and it's executed successfully. +- `ClusterResourcePlacementEviction` is invalid. +- `ClusterResourcePlacementEviction` is valid but it's not executed. + +To successfully evict resources from a cluster, the user needs to specify: + +- The name of the `ClusterResourcePlacement` object which propagated resources to the target cluster. +- The name of the target cluster from which we need to evict resources. + +When specifying the `ClusterResourcePlacement` object in the eviction's spec, the user needs to consider the following cases: + +- For `PickFixed` CRP, eviction is not allowed; it is recommended that one directly edit the list of target clusters on the CRP object. +- For `PickAll` & `PickN` CRPs, eviction is allowed because the users cannot deterministically pick or unpick a cluster based on the placement strategy; it's up to the scheduler. + +> **Note:** After an eviction is executed, there is no guarantee that the cluster won't be picked again by the scheduler to propagate resources for a `ClusterResourcePlacement` resource. +> The user needs to specify a [taint](../../howtos/taint-toleration.md) on the cluster to prevent the scheduler from picking the cluster again. This is especially true for `PickAll ClusterResourcePlacement` because +> the scheduler will try to propagate resources to all the clusters in the fleet. + +## ClusterResourcePlacementDisruptionBudget + +The `ClusterResourcePlacementDisruptionBudget` is used to protect resources propagated by a `ClusterResourcePlacement` to a target cluster from voluntary disruption, i.e., `ClusterResourcePlacementEviction`. + +> **Note:** When specifying a `ClusterResourcePlacementDisruptionBudget`, the name should be the same as the `ClusterResourcePlacement` that it's trying to protect. + +Users are allowed to specify one of two fields in the `ClusterResourcePlacementDisruptionBudget` spec since they are mutually exclusive: + +- MaxUnavailable - specifies the maximum number of clusters in which a placement can be unavailable due to any form of disruptions. +- MinAvailable - specifies the minimum number of clusters in which placements are available despite any form of disruptions. + +for both `MaxUnavailable` and `MinAvailable`, the user can specify the number of clusters as an integer or as a percentage of the total number of clusters in the fleet. + +> **Note:** For both MaxUnavailable and MinAvailable, involuntary disruptions are not subject to the disruption budget but will still count against it. + +When specifying a disruption budget for a particular `ClusterResourcePlacement`, the user needs to consider the following cases: + +| CRP type | `MinAvailable` DB with an integer | `MinAvailable` DB with a percentage | `MaxUnavailable` DB with an integer | `MaxUnavailable` DB with a percentage | +|--------------|-----------------------------------|-------------------------------------|-------------------------------------|---------------------------------------| +| `PickFixed` | ❌ | ❌ | ❌ | ❌ | +| `PickAll` | ✅ | ❌ | ❌ | ❌ | +| `PickN` | ✅ | ✅ | ✅ | ✅ | + +> **Note:** We don't allow eviction for `PickFixed` CRP and hence specifying a `ClusterResourcePlacementDisruptionBudget` for `PickFixed` CRP does nothing. +> And for `PickAll` CRP, the user can only specify `MinAvailable` because total number of clusters selected by a `PickAll` CRP is non-deterministic. +> If the user creates an invalid `ClusterResourcePlacementDisruptionBudget` object, when an eviction is created, the eviction won't be successfully executed. \ No newline at end of file diff --git a/docs/concepts/README.md b/docs/concepts/README.md index 53bc91064..a176b5b71 100644 --- a/docs/concepts/README.md +++ b/docs/concepts/README.md @@ -27,3 +27,6 @@ Allow slightly different manifests depends on the cluster it lands. ## [PropertyProvider](PropertyProviderAndClusterProperties/README.md) More ways to select the clusters based on its property. + +## [Eviction & Placement Disruption Budget](EvictionAndDisruptionBudget/README.md) +Understand how placements can be evicted and protected from voluntary disruptions in fleet. diff --git a/docs/howtos/README.md b/docs/howtos/README.md index 7c7a79f76..de5c86988 100644 --- a/docs/howtos/README.md +++ b/docs/howtos/README.md @@ -56,4 +56,10 @@ domains: This how-to guide explains the specifics of the Fleet `ResourceOverride` API, including its resource selectors, policy, and more. `ResourceOverride` is a Fleet API that allows you to - modify or override specific attributes across namespaced resources. \ No newline at end of file + modify or override specific attributes across namespaced resources. + +* [Using the Fleet `ClusterResourcePlacementEviction` and `ClusterResourcePlacementDisruptionBudget` APIs](eviction-placement-disruption-budget.md) + + This how-to guide explains the specifics of the Fleet `ClusterResourcePlacementEviction` and + `ClusterResourcePlacementDisruptionBudget` APIs, including how to evict resources from a + cluster and protect resources from voluntary disruption. \ No newline at end of file diff --git a/docs/howtos/eviction-placement-disruption-budget.md b/docs/howtos/eviction-placement-disruption-budget.md new file mode 100644 index 000000000..a4f31af1b --- /dev/null +++ b/docs/howtos/eviction-placement-disruption-budget.md @@ -0,0 +1,181 @@ +# Using ClusterResourcePlacementEviction and ClusterResourcePlacementDisruptionBudget + +This how-to guide discusses how to create `ClusterResourcePlacementEviction` objects and `ClusterResourcePlacementDisruptionBudget` objects to evict resources from member clusters and protect resources on member clusters from voluntary disruption, respectively. + +## Evicting Resources from Member Clusters using ClusterResourcePlacementEviction + +The `ClusterResourcePlacementEviction` object is used to remove resources from a member cluster once the resources have already been propagated from the hub cluster. + +To successfully evict resources from a cluster, the user needs to specify: +- The name of the `ClusterResourcePlacement` object which propagated resources to the target cluster. +- The name of the target cluster from which we need to evict resources. + +In this example, we will create a `ClusterResourcePlacement` object with PickAll placement policy to propagate resources to an existing `MemberCluster`, add a taint to the member cluster +resource and then create a `ClusterResourcePlacementEviction` object to evict resources from the `MemberCluster`. + +We will first create a namespace that we will propagate to the member cluster. + +``` +kubectl create ns test-ns +``` + +Then we will apply a `ClusterResourcePlacement` with the following spec: + +```yaml +spec: + resourceSelectors: + - group: "" + kind: Namespace + version: v1 + name: test-ns + policy: + placementType: PickAll +``` + +The `CRP` status after applying should look something like this: + +```yaml +kubectl get crp test-crp +NAME GEN SCHEDULED SCHEDULED-GEN AVAILABLE AVAILABLE-GEN AGE +test-crp 2 True 2 True 2 5m49s +``` + +let's now add a taint to the member cluster to ensure this cluster is not picked again by the scheduler once we evict resources from it. + +Modify the cluster object to add a taint: + +```yaml +spec: + heartbeatPeriodSeconds: 60 + identity: + kind: ServiceAccount + name: fleet-member-agent-cluster-1 + namespace: fleet-system + taints: + - effect: NoSchedule + key: test-key + value: test-value +``` + +Now we will create a `ClusterResourcePlacementEviction` object to evict resources from the member cluster: + +```yaml +apiVersion: placement.kubernetes-fleet.io/v1beta1 +kind: ClusterResourcePlacementEviction +metadata: + name: test-eviction +spec: + placementName: test-crp + clusterName: kind-cluster-1 +``` + +the eviction object should look like this, if the eviction was successful: + +```yaml +kubectl get crpe test-eviction +NAME VALID EXECUTED +test-eviction True True +``` + +since the eviction is successful, the resources should be removed from the cluster, let's take a look at the `CRP` object status to verify: + +```yaml +kubectl get crp test-crp +NAME GEN SCHEDULED SCHEDULED-GEN AVAILABLE AVAILABLE-GEN AGE +test-crp 2 True 2 15m +``` + +from the object we can clearly tell that the resources were evicted since the `AVAILABLE` column is empty. If the user needs more information `ClusterResourcePlacement` object's status can be checked. + +## Protecting resources from voluntary disruptions using ClusterResourcePlacementDisruptionBudget + +In this example, we will create a `ClusterResourcePlacement` object with PickN placement policy to propagate resources to an existing MemberCluster, +then create a `ClusterResourcePlacementDisruptionBudget` object to protect resources on the MemberCluster from voluntary disruption and +then try to evict resources from the MemberCluster using `ClusterResourcePlacementEviction`. + +We will first create a namespace that we will propagate to the member cluster. + +``` +kubectl create ns test-ns +``` + +Then we will apply a `ClusterResourcePlacement` with the following spec: + +```yaml +spec: + resourceSelectors: + - group: "" + kind: Namespace + version: v1 + name: test-ns + policy: + placementType: PickN + numberOfClusters: 1 +``` + +The `CRP` object after applying should look something like this: + +```yaml +kubectl get crp test-crp +NAME GEN SCHEDULED SCHEDULED-GEN AVAILABLE AVAILABLE-GEN AGE +test-crp 2 True 2 True 2 8s +``` + +Now we will create a `ClusterResourcePlacementDisruptionBudget` object to protect resources on the member cluster from voluntary disruption: + +```yaml +apiVersion: placement.kubernetes-fleet.io/v1beta1 +kind: ClusterResourcePlacementDisruptionBudget +metadata: + name: test-crp +spec: + minAvailable: 1 +``` + +> **Note:** An eviction object is only reconciled once, after which it reaches a terminal state, if the user desires to create/apply the same eviction object again they need to delete the existing eviction object and re-create the object for the eviction to occur again. + +Now we will create a `ClusterResourcePlacementEviction` object to evict resources from the member cluster: + +```yaml +apiVersion: placement.kubernetes-fleet.io/v1beta1 +kind: ClusterResourcePlacementEviction +metadata: + name: test-eviction +spec: + placementName: test-crp + clusterName: kind-cluster-1 +``` + +> **Note:** The eviction controller will try to get the corresponding `ClusterResourcePlacementDisruptionBudget` object when a `ClusterResourcePlacementEviction` object is reconciled to check if the specified MaxAvailable or MinAvailable allows the eviction to be executed. + +let's take a look at the eviction object to see if the eviction was executed, + +```yaml +kubectl get crpe test-eviction +NAME VALID EXECUTED +test-eviction True False +``` + +from the eviction object we can see the eviction was not executed. + +let's take a look at the `ClusterResourcePlacementEviction` object status to verify why the eviction was not executed: + +```yaml +status: + conditions: + - lastTransitionTime: "2025-01-21T15:52:29Z" + message: Eviction is valid + observedGeneration: 1 + reason: ClusterResourcePlacementEvictionValid + status: "True" + type: Valid + - lastTransitionTime: "2025-01-21T15:52:29Z" + message: 'Eviction is blocked by specified ClusterResourcePlacementDisruptionBudget, + availablePlacements: 1, totalPlacements: 1' + observedGeneration: 1 + reason: ClusterResourcePlacementEvictionNotExecuted + status: "False" + type: Executed +``` + +the eviction status clearly mentions that the eviction was blocked by the specified `ClusterResourcePlacementDisruptionBudget`. \ No newline at end of file diff --git a/examples/eviction/clusterpdb.yaml b/examples/eviction/clusterpdb.yaml index df2a8ae9d..e5292e618 100644 --- a/examples/eviction/clusterpdb.yaml +++ b/examples/eviction/clusterpdb.yaml @@ -1,6 +1,6 @@ -apiVersion: placement.kubernetes-fleet.io/v1alpha1 +apiVersion: placement.kubernetes-fleet.io/v1beta1 kind: ClusterResourcePlacementDisruptionBudget metadata: name: test-crp spec: - maxUnavailable: 1 \ No newline at end of file + minAvailable: 1 diff --git a/examples/eviction/eviction.yaml b/examples/eviction/eviction.yaml index 5fc0a153d..37c9c3c75 100644 --- a/examples/eviction/eviction.yaml +++ b/examples/eviction/eviction.yaml @@ -1,7 +1,7 @@ -apiVersion: placement.kubernetes-fleet.io/v1alpha1 +apiVersion: placement.kubernetes-fleet.io/v1beta1 kind: ClusterResourcePlacementEviction metadata: name: test-eviction spec: placementName: test-crp - clusterName: cluster-1 \ No newline at end of file + clusterName: kind-cluster-1