DRA: ReservedFor Workloads

[johnbelamaric]

Enhancement Description

Currently, when the scheduler allocates a ResourceClaim for a given Pod, it adds that Pod to the ResourceClaimStatus.ReservedFor list. A claim shared among multiple pods will have multiple entries in this list. This allows the ResourceClaimController to know when to de-allocate the claim; it does so once this list is empty.

The length of this list is limited to 256 pods. However, some workloads are much larger and may share a resource claim across many more pods, even in the thousands. Simply increasing the pod list to thousands is not a good long term solution.

Instead, this proposal is to allow us to reserve it for a workload. For example, rather than listing individual pods, you could list the Job or ReplicaSet or StatefulSet that is sharing the ResourceClaim. This avoids race conditions as pods come and go, without requiring listing every pod.

cc @pohly @klueska @thebinaryone1

• One-line enhancement description (can be used as a release note): Enable resource claims to be reserved for workloads, not just individual pods
• Kubernetes Enhancement Proposal: KEP-5194: DRA: ReservedFor Workloads in 1.34 #5379
• Discussion Link: https://kubernetes.slack.com/archives/C0409NGC1TK/p1741647231531399
• Primary contact (assignee): @johnbelamaric
• Responsible SIGs: sig-scheduling
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): 1.34
  • Beta release target (x.y): 1.35
  • Stable release target (x.y): 1.36
• Alpha
  • KEP (k/enhancements) update PR(s):
    • KEP-5194: DRA: ReservedFor Workloads in 1.34 #5379
  • Code (k/k) update PR(s):
  • Docs (k/website) update PR(s):

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

[johnbelamaric]

/sig scheduling
/sig node
/wg device-management

[johnbelamaric]

/retitle DRA: ReservedFor Workloads

[johnbelamaric]

Two current options @pohly and I have been discussing with respect to this.

Option One: Add API for this

• claim.spec.reservedFor *ResourceClaimConsumerReference
  "If set, the scheduler will reserve the claim for this consumer instead of the pod which triggered allocation. It is the responsibility of whoever set up the claim like this to remove this consumer from from the status.reservedFor when the claim can be deallocated. They may also remove the allocation at the same time, otherwise the resource claim controller will do that."
• claim.status.allocation.reservedForAnyPod bool
  "If true, then the scheduler will schedule pods referencing the claim without adding them to reservedFor. The kubelet will not check whether a pod is listed as consumer of the claim when starting the pod."

In this option, the scheduler sets the boolean to true if the spec field is populated, and it copies the spec field to the status.

Option Two: No API, more implicit

In this option, the workload controller that is creating the resource claim updates the status after creating the claim, but before creating any pods referencing the claim:

1. Workload controller creates the resource claim
2. Workload controller updates the status on the resource claim with itself; since no pods reference it yet, scheduler won't yet have touched the status. This would require relaxing the current validation which only allows allocated claims to be reserved.
3. Workload controller creates pods pointing to the RC
4. Scheduler sees the pod referencing the RC, sees non-pods in the status reservedFor, and so proceeds with allocation without adding the pod
5. kubelet sees non-pod in the reserved for, so it doesn't do the sanity check
6. ResourceClaim controller doesn't de-allocate unless the status reservedFor is empty, it is the responsibility of the workload controller to clear the reservedFor (as in the first option).

With either option, we can tighten the ResoruceClaimController behavior up a bit to avoid orphans by letting it follow the references in the reservedFor and remove any that are to objects that no longer exist. But that would require read permissions for that controller on those objects.