[KEP-5282]Add KEP for Implicit Tolerations

[cici37]

• One-line PR description:

• Issue link: Implicit tolerations #5282

• Other comments:

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: cici37
Once this PR has been reviewed and has the lgtm label, please assign johnbelamaric for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• keps/prod-readiness/OWNERS
• keps/sig-scheduling/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[cici37]

/assign @dom4ha @johnbelamaric @sanposhiho

[mortent]

/wg device-management

[sanposhiho]

We will take a look, but very likely we cannot make it by the KEP freeze because our bandwidth is limited and there are other on-going KEPs.

[dom4ha]

Isn't PreEnqueue too early? In case of PrioritizedAlternatives we may have tainted GPU listed as only one of the options. If we put toleration too early, we (very theoretically) could schedule a pod on tainted node but not use that resource for which the node was tainted.

Alternatively, as mentioned before, tolerations could be also fully implicit (based on policy configuration similar to DeviceTaintRule) and "applied" invisibly by the TaintToleration plugin during the Filtering phase. However, the problem is that during the Filtering phase in the TaintToleration plugin we may not know which devices got allocated by the DRA plugin, so it may be hard to take decision based on it.

This is why I think that tainting nodes is not the best way to prevent scheduling pods on nodes with scarce resources. Ideally picking or discarding nodes with scarce unused resources should be rather based on scoring mechanisms, or filtering, which could block some placements based on custom policies. So if we forget about taints and toleration mechanism, such logic could be implemented inside DRA plugin.

[dom4ha]

We will take a look, but very likely we cannot make it by the KEP freeze because our bandwidth is limited and there are other on-going KEPs.

I agree with @sanposhiho , but also I think there will be ton of code changes in this cycle, which will be very challenging to handle. I'm not convinced that the use case really justifies the effort, as it's not clear to me whether taints and tolerations should be the solution we recommend.

[SergeyKanzhelev]

what if there are multiple resolutions possible? Will we remove all tolerations so scheduler can decide later where to schedule?

[cici37]

The KEP is discussed in the latest sig scheduling meeting: #5282 (comment) and we'll hold the merge til get an agreement on the use case support.