DRA: Handle extended resource requests via DRA Driver

[klueska]

Enhancement Description

• One-line enhancement description (can be used as a release note):
  Allow DRA drivers to honor requests made via the extended resource API (e.g. nvidia.com/gpu: 2) rather than requiring a standard device plugin be used.

• Kubernetes Enhancement Proposal:

  • Add KEP for DRA: Extended Resource #5136
  • Incremental PRs:
    • TBD

• Discussion Link:

  • https://youtu.be/fKhX_lHK8Z0?si=gq5kIFHP3ve2TXyE&t=1822

• Primary contact (assignee):
  @klueska, @pohly, @johnbelamaric

• Responsible SIGs:
  /sig node
  /wg device-management

• Enhancement target (which target equals to which milestone):

  • Alpha release target: 1.34
  • Beta release target: 1.35
  • Stable release target: 1.36

• Alpha

  • KEP (k/enhancements) update PR(s):
    • Add KEP for DRA: Extended Resource #5136
  • Code (k/k) update PR(s):
    • TBD
  • Docs (k/website) update PR(s):
    • TBD

[johnbelamaric]

+1 yes please!

[johnbelamaric]

We need to sort out the requirements. A few initial questions:

1. For newly created pods, I think it's clear we want this to be transparent. Existing manifests that use the extended resource API should continue to work as before, without modification.
2. Can we handle this invisibly in the driver layer, or do we need to have DRA invoked at the control plane level and select the specific devices? If we don't, we will likely have a race condition - unless the scheduler can do some magical accounting (which seems possible).
3. How do we handle upgrades? If we have a node running device plugin, and we switch to the DRA driver (or we upgrade to a driver that supports both), do you have to delete the pods? Do they automatically adopt the devices? If so, how do we write those back to the allocation logic (since no DRA claim exists).
4. What happens if there are pods in a deployment, and some land on nodes with device plugin and some with DRA drivers?
5. We talked about letting specific device classes be advertised as specific extended resources. This could mean the existing resource names get mapped to specific device classes by the admin. It could also mean we have a convention like deviceclass.k8s.io/foo: 4 for extended resource names. How do these choices interplay with the questions above?

[lengrongfu]

Can each dra-driver implement a webhook to create a ResourceClaimTemplate after creating a pod and modify the application method of resources in the pod?

[klueska]

@lengrongfu that is what this KEP would be designed to avoid. There would be integrated scheduler support for all drivers, rather than requiring each DRA driver to provide a webhook.

[alculquicondor]

Open questions (from SIG Scheduling meeting):

• How to handle resource quotas
• Scheduling throughput (API requests and overall processing).

[ffromani]

/cc

[yliaog]

/cc

[johnbelamaric]

/sig scheduling

[johnbelamaric]

/assign @yliaog

Yu, I am assigning to you, let me know if that's OK

[haircommander]

/label lead-opted-in
/milestone v1.33

note: PRR freeze is tomorrow! you need to have a KEP update for this opened before then. Thanks!

[johnbelamaric]

/stage alpha

[dipesh-rawat]

Hello @klueska @pohly @johnbelamaric @yliaog 👋, v1.33 Enhancements team here.

Just checking in as we approach enhancements freeze on 02:00 UTC Friday 14th February 2025 / 19:00 PDT Thursday 13th February 2025.

This enhancement is targeting stage alpha for v1.33 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: v1.32.
• KEP readme has up-to-date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here). If your production readiness review is not completed yet, please make sure to fill the production readiness questionnaire in your KEP by the PRR Freeze deadline on Thursday 6th February 2025 so that the PRR team has enough time to review your KEP.

For this KEP, we would need to update the following:

• Create the KEP readme using the latest template and merge it in the k/enhancements repo.
• Ensure that the KEP has undergone a production readiness review and has been merged into k/enhancements.

The status of this enhancement is marked as At risk for enhancements freeze. Please keep the issue description up-to-date with appropriate stages as well

If you anticipate missing enhancements freeze, you can file an exception request in advance. Thank you!

[dipesh-rawat]

Hi @klueska @pohly @johnbelamaric @yliaog 👋, 1.33 Enhancements team here,

Just a quick friendly reminder as we approach the enhancements freeze later this week, at 02:00 UTC Friday 14th February 2025 / 19:00 PDT Thursday 13th February 2025.

The current status of this enhancement is marked as At risk for enhancement freeze. There are a few requirements mentioned in the comment #5004 (comment) that still need to be completed.

If you anticipate missing enhancements freeze, you can file an exception request in advance. Thank you!

[johnbelamaric]

@dipesh-rawat we will be doing this in 1.34 instead - I updated the description above, can you do whatever else the release team needs to properly account for that?

[haircommander]

/remove-label lead-opted-in
/remove-milestone v1.33

[dipesh-rawat]

I see that this issue has been opted-out of v1.33 and is now planned for a future release. I will go ahead and mark it as Deferred on the v1.33 board for tracking purposes - do let the enhancement team know otherwise.

/milestone clear