KEP-5328: Node Capabilities

[pravk03]

• One-line PR description: Add the initial KEP for KEP 5328: Node Capabilities

• Issue link: Node Capabilities #5328

• Other comments:

[k8s-ci-robot]

Welcome @pravk03!

It looks like this is your first PR to kubernetes/enhancements 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/enhancements has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @pravk03. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[wojtek-t]

@dom4ha @sanposhiho @macsko - FYI

[pravk03]

/cc @tallclair @yujuhong

[sanposhiho]

/sig scheduling

[wojtek-t]

Those are all good tests, but these are feature tests - not enablement/disablement.

Enablement/disablement is a test that (as stated in the comment in the template above) that switches the feature gate in the middle of the test.

[pravk03]

Updated. Please take a look.

[wojtek-t]

friendly ping

[SergeyKanzhelev]

We discussed this KEP yesterday on a call. Some notes from that call:

Many examples in this KEP of past enhancements that might have used or may use capabilities are not accurate.

1. For example, LMSConfig will unlikely be using capabilities as a way to understand which LMSConfig is enabled. Reasons being that AppArmor will also want to know that the specific profile is installed. So some sort of LMSConfig object would be a better design. Also many DaemonSets today use both profiles and kubelet just picks whatever is applicable. This will be much harder if LMSConfig will be introduced as capability - each DaemonSet will need to be declared in 3 shapes - one with SeLinux, one with AppArmor, one without either.

2. Swap is not a good fit for capabilities as a way to discover that the Node has swap configured. Even though it may be an easy way to "avoid API review and introduce a capability", capability is very limiting in it's functionality. For swap discoverability, swap-specific node status (allocatables?) is a better option.

3. Runtime handlers as a list of handlers is also not a good fit. Default handler runc is not specified in pod spec. So it will not be used by scheduler and by definition must not be added to capabilities. Non-default handlers may need more details on what it is. And names list may not fit into the value length limits. Special object representing the runtime is a better choice here.

Examples where capabilities are useful are:

1. Feature gates with the specific field in Pod Spec. Like Sidecar, PodLevelResources, etc. Basically, discoverability whether the new filed will work on a given kubelet.
2. Capability like feature.kubernetes.io/guaranteedQOSPodCPUResize representing the fact that the Feature Gate while in alpha or beta had a certain limitaiton before and now this limitation was lifted. Often it is lifted with the new FG, but not always.
3. Container runtime missing APIs like for user namespaces support. This also related to UserNamespace capability. And the k8s expectation is that soon ALL nodes will support UserNamespaces. So capabilitty has a lifetime bound to the FG.

We discussed that examples above may be often solved for individual vendors (which control the list of enabled FG per node version) by introducing the semver-base node selector. But capabilities for sure provide way better API for this.

I would suggest in this KEP:

1. Remove any mention on non-FG related capabilities from the readme, unless there is a good example that can be articulated and explained why capability is a good fit there.
2. Add a note that the capability is a part of API and requires API review. In k/k codebase we will need to protect the list of capabilities with the api-approvers OWNER file.
3. Unless we find good examples, let's state that capabilities have a lifetime and we do not expect any long-lived capability.
4. If we are limiting capabilities to feature gate related features, maybe we should rename capabilities to featureGates to avoid reusing it for long term capabilities long term.

We also discussed that capabilities must be applied to DaemonSets with no exceptions.

I also want to see something explaining how capabilities and Cluster Autoscaler will work together.

[pravk03]

Thanks a lot @SergeyKanzhelev for the discussion and the feedback.

I am okay with most of the above suggestions and I will address them in the KEP. I has some thoughts regarding naming.

If we are limiting capabilities to feature gate related features, maybe we should rename capabilities to featureGates to avoid reusing it for long term capabilities long term.

• While our initial examples used to demonstrate the functionality are tied to feature gates, renaming it to featureGates would be too restrictive for future use cases.
• A capability should represent a logical use case, which could enabled by a single feature gate, but it could also be a combination of multiple feature gates plus specific configurations. featureGates wouldn't accurately represent such capabilities.

I am definitely open to naming suggestions, but I believe the name should be broad enough to accommodate future use-cases without requiring a new API field down the road.

[ajaysundark]

2. Swap is not a good fit for capabilities as a way to discover that the Node has swap configured. Even though it may be an easy way to "avoid API review and introduce a capability", capability is very limiting in it's functionality. For swap discoverability, swap-specific node status (allocatables?) is a better option.

Referring my earlier reply on this discussion comment -

For swap, node-capability is much needed for 'placement-control' to protect a latency-sensitive pod is never scheduled on a swap-enabled node.
Scheduler control for swap needs two questions:

1. whether workload needs swap (new api for swap preference from pod-spec)
2. whether node is swap configured

A swap-capability will provide the signal for (2), allowing for simple and clear scheduling rules.

Alternatives like 'NFD' exists for detecting swap on a node. But it is out-of-tree and not aware of the Kubelet's specific swap configuration.

[SergeyKanzhelev]

I am definitely open to naming suggestions, but I believe the name should be broad enough to accommodate future use-cases without requiring a new API field down the road.

Can we have any examples listed that will justify this. Right now the KEP suggests to use it for FG-related capabilities, while not giving a good examples where it would be non-FG related.

[macsko]

The scheduling part looks good for alpha
/approve as SIG Scheduling

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: macsko, pravk03
Once this PR has been reviewed and has the lgtm label, please ask for approval from wojtek-t and additionally assign dchen1107 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• keps/prod-readiness/OWNERS
• keps/sig-node/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[pravk03]

Can we have any examples listed that will justify this. Right now the KEP suggests to use it for FG-related capabilities, while not giving a good examples where it would be non-FG related.

The guaranteedQOSPodCPUResize example used in the KEP isn't purely a feature gate; it's a logical capability derived from a combination of feature gates and the Kubelet's cpuManagerPolicy configuration.

While this is still in early stages, this recent discussion about making the pod requirement for exclusive resources more explicit also indicates a need for non-FG capabilities. The API field itself should be forward-facing enough to support such potential use-cases ?.

[SergeyKanzhelev]

Can we have any examples listed that will justify this. Right now the KEP suggests to use it for FG-related capabilities, while not giving a good examples where it would be non-FG related.

The guaranteedQOSPodCPUResize example used in the KEP isn't purely a feature gate; it's a logical capability derived from a combination of feature gates and the Kubelet's cpuManagerPolicy configuration.

While this is still in early stages, this recent discussion about making the pod requirement for exclusive resources more explicit also indicates a need for non-FG capabilities. The API field itself should be forward-facing enough to support such potential use-cases ?.

Those are all examples of FG-related capabilities. Not the generic long-term capabilities.

[tallclair]

It seems like most of the concerns with this are around the specific capabilities being added, but this KEP doesn't actually propose adding any capabilities. The examples given are hypothetical examples based on features currently in development, but no new features will be able to depend on capabilities until it goes to beta. This creates a bit of a chicken-and-egg situation, where it's hard to point to exactly how capabilities will be used until we have users lined up, but we can't line up users yet.

[SergeyKanzhelev]

It seems like most of the concerns with this are around the specific capabilities being added, but this KEP doesn't actually propose adding any capabilities. The examples given are hypothetical examples based on features currently in development, but no new features will be able to depend on capabilities until it goes to beta. This creates a bit of a chicken-and-egg situation, where it's hard to point to exactly how capabilities will be used until we have users lined up, but we can't line up users yet.

we kind of need to know what will be expected use cases. Maybe past examples or hypothetical examples thought thru end-to-end. Right now this KEP is limited to just set of name/value pairs and a scenario of FG discoverability. But already we are thinking there MAY be need to support capabilities for node selection, ability to declare tolerations for capabilities, ability to have node-restricted capabilities. Knowing the scope would help to understand if API proposed is needed (among alternatives if the set of use cases is limited) and if needed, what shape should it have.

[pravk03]

Maybe past examples or hypothetical examples thought thru end-to-end

RuntimeClass was intended as a past example used to illustrate non-FG related runtime capabilities in the earlier version of the proposal. I agree that it had some missing details and thanks for highlighted them in your comment.

3. Runtime handlers as a list of handlers is also not a good fit. Default handler runc is not specified in pod spec. So it will not be used by scheduler and by definition must not be added to capabilities. Non-default handlers may need more details on what it is. And names list may not fit into the value length limits. Special object representing the runtime is a better choice here.

I have tried to address these the Case Study section.