KEP-4969: Cluster Domain Downward API #4972

nightkr · 2024-11-21T13:24:22Z

One-line PR description: Initial KEP draft

Issue link: Cluster Domain Downward API #4969

Other comments:

k8s-ci-robot · 2024-11-21T13:24:30Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: nightkr
Once this PR has been reviewed and has the lgtm label, please assign mikezappa87 for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

keps/sig-network/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2024-11-21T13:24:31Z

Welcome @nightkr!

It looks like this is your first PR to kubernetes/enhancements 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/enhancements has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2024-11-21T13:24:32Z

Hi @nightkr. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

lfrancke

just two minor nitpicks/typos

lfrancke · 2024-11-21T13:46:24Z

keps/sig-network/4969-cluster-domain-downward-api/README.md

+Currently there is no way for cluster workloads to query for this domain name,
+leaving them either use relative domain names or take it as manual configuration.


Suggested change

Currently there is no way for cluster workloads to query for this domain name,

leaving them either use relative domain names or take it as manual configuration.

Currently, there is no way for cluster workloads to query this domain name,

leaving them to either use relative domain names or configure it manually.

6c655bf

I left "query for this" as-is, because I read the revised variant as "ask what the domain name (that I already know) means" rather than "ask what the domain name is".

lfrancke · 2024-11-21T13:46:35Z

keps/sig-network/4969-cluster-domain-downward-api/README.md

+Currently there is no way for cluster workloads to query for this domain name,
+leaving them either use relative domain names or take it as manual configuration.
+
+This KEP proposes adding a new Downward API for that workloads can use to request it.


Suggested change

This KEP proposes adding a new Downward API for that workloads can use to request it.

This KEP proposes adding a new Downward API that workloads can use to request it.

adrianmoisey · 2024-11-21T19:46:44Z

keps/sig-network/4969-cluster-domain-downward-api/README.md

+- `nodePropertyRef` (@aojea)
+- `runtimeConfigs` (@thockin)


I guess this is out of scope of this KEP, but I guess this change sets a precedent of other configs that could be passed down into the Pod.
I wonder if someone with more experience than me has an idea/vision of what that could look like, which may then determine what name to decide on?

Looking forward, I suspect that both cases will become relevant eventually. This property just occupies an awkward spot since it doesn't really have one clean owner.

adrianmoisey · 2024-11-22T09:01:53Z

/ok-to-test

nightkr · 2024-11-22T12:11:39Z

/retest

keps/sig-network/4969-cluster-domain-downward-api/README.md

sftim · 2024-11-22T12:58:24Z

keps/sig-network/4969-cluster-domain-downward-api/README.md

+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->


Suggested change

keps/sig-network/4969-cluster-domain-downward-api/README.md

sftim · 2024-11-22T13:00:38Z

keps/sig-network/4969-cluster-domain-downward-api/README.md

+            clusterPropertyRef: clusterDomain
+```
+
+`foo` can now perform the query by running `curl http://bar.$NAMESPACE.svc.$CLUSTER_DOMAIN/`.


Extra credit: define a sidecar that relies on interpolating $(CLUSTER_DOMAIN)

sftim · 2024-11-22T13:02:28Z

keps/sig-network/4969-cluster-domain-downward-api/README.md

+environments, since `node-b` might not be able to resolve `cluster.local`
+FQDNs correctly.
+
+For this KEP to make sense, this would have to be explicitly prohibited.


I don't know if that's true.

For Pod consumes, the downward API is implemented by the kubelet, so each kubelet can expose its local view of the cluster domain.
We would still strongly recommend against having multiple cluster domains defined across your cluster - anything else sounds really unwise - but technically it can be made to work.

Yeah, it's one of those.. it would be implementable without making such a declaration, but it would likely be another thing leading to pretty confusing behaviour. Maybe the language can be softened somewhat.

sftim · 2024-11-22T13:08:57Z

keps/sig-network/4969-cluster-domain-downward-api/README.md

+<!--
+What are the caveats to the proposal?
+What are some important details that didn't come across above?
+Go in to as much detail as necessary here.
+This might be a good place to talk about core concepts and how they relate.
+-->


A related detail.

A kubelet that is formally aware of its cluster domain could report back either the cluster domain value, or a hash of it, via .status.

A kubelet that is formally aware of its cluster domain could report back either the cluster domain value, or a hash of it, via a somethingz HTTP endpoint.

A kubelet that is formally aware of its cluster domain could report back either the cluster domain value, or a hash of it, via a Prometheus static-series label
eg kubelet_cluster_domain{domain_name="cluster.example"} 1

If we decide to make the API server aware of cluster domain, adding that info could help with troubleshooting and general observability.

It's already available via the kubelet's /configz (mentioned in Alternatives). I don't have a strong opinion either way on adding it to the Node status.

Co-authored-by: Tim Bannister <[email protected]>

sftim · 2024-11-22T15:04:20Z

keps/sig-network/4969-cluster-domain-downward-api/README.md

+The ConfigMap written by k3s[^prior-art-k3s] could be blessed, requiring that 
+all other distributions also provide it. However, this would require additional
+migration effort from each distribution.
+
+Additionally, this would be problematic to query for: users would have to query
+it manually using the Kubernetes API (since ConfigMaps cannot be mounted across
+Namespaces), and users would require RBAC permission to query wherever it is stored.


I thought centralized management was a non goal though? I recommend highlighting that this alternative isn't aligned with this KEP's goals.

In terms of the non-goal I was mostly referring to configuring the kubelet itself (which would have been a retread of #281). Maybe that could be clarified.

sftim · 2024-11-22T15:04:26Z

keps/sig-network/4969-cluster-domain-downward-api/README.md

+This roughly shares the arguments for/against as [the ConfigMap alternative](#alternative-configmap),
+although it would allow more precise RBAC policy targeting.


I thought centralized management was a non goal though? I recommend highlighting that this alternative isn't aligned with this KEP's goals.

sftim · 2024-11-22T15:05:00Z

keps/sig-network/4969-cluster-domain-downward-api/kep.yaml

+# The following PRR answers are required at alpha release
+# List the feature gate name and the components for which it must be enabled
+feature-gates:
+  - name: MyFeature


We could pick the feature gate name even at provisional stage.

Sure. PodClusterDomain would align with PodHostIPs (the only other downward API feature gate listed on https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/), but happy to hear other takes.

sftim · 2024-11-22T15:05:54Z

keps/sig-network/4969-cluster-domain-downward-api/README.md

+
+## Design Details
+
+A new Downward API `clusterPropertyRef: clusterDomain` would be introduced, which can be projected into an environment variable or a volume file.


What's this a reference to?

I was mostly trying to be consistent with fieldRef and resourceFieldRef, which also don't quite correspond to the pod/container objects (since they're not quite 1:1), but I'm certainly not married to it.

keps/sig-network/4969-cluster-domain-downward-api/README.md

Co-authored-by: Tim Bannister <[email protected]>

Initial draft for KEP-4969

8f6bd3f

k8s-ci-robot added the kind/kep Categorizes KEP tracking issues and PRs modifying the KEP directory label Nov 21, 2024

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. sig/network Categorizes an issue or PR as relevant to SIG Network. labels Nov 21, 2024

k8s-ci-robot requested review from MikeZappa87 and thockin November 21, 2024 13:24

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Nov 21, 2024

k8s-ci-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Nov 21, 2024

nightkr mentioned this pull request Nov 21, 2024

Cluster Domain Downward API #4969

Open

4 tasks

lfrancke suggested changes Nov 21, 2024

View reviewed changes

k8s-ci-robot added the do-not-merge/invalid-commit-message Indicates that a PR should not merge because it has an invalid commit message. label Nov 21, 2024

Grammar fixes (thanks lfrancke)

a878d62

nightkr force-pushed the kep-4969 branch from 6c655bf to a878d62 Compare November 21, 2024 15:02

k8s-ci-robot removed the do-not-merge/invalid-commit-message Indicates that a PR should not merge because it has an invalid commit message. label Nov 21, 2024

adrianmoisey reviewed Nov 21, 2024

View reviewed changes

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Nov 22, 2024

Update ToC

6962f89

sftim reviewed Nov 22, 2024

View reviewed changes

nightkr and others added 2 commits November 22, 2024 15:54

Update keps/sig-network/4969-cluster-domain-downward-api/README.md

cb664d2

Co-authored-by: Tim Bannister <[email protected]>

Update keps/sig-network/4969-cluster-domain-downward-api/README.md

bd2527f

Co-authored-by: Tim Bannister <[email protected]>

sftim reviewed Nov 22, 2024

View reviewed changes

keps/sig-network/4969-cluster-domain-downward-api/README.md Outdated Show resolved Hide resolved

nightkr and others added 2 commits November 22, 2024 17:52

Update keps/sig-network/4969-cluster-domain-downward-api/README.md

0f012a3

Co-authored-by: Tim Bannister <[email protected]>

Update keps/sig-network/4969-cluster-domain-downward-api/README.md

26356a9

Co-authored-by: Tim Bannister <[email protected]>

lfrancke mentioned this pull request Nov 26, 2024

Improve DNS lookup performance by working around ndots stackabletech/issues#656

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KEP-4969: Cluster Domain Downward API #4972

KEP-4969: Cluster Domain Downward API #4972

nightkr commented Nov 21, 2024

k8s-ci-robot commented Nov 21, 2024

k8s-ci-robot commented Nov 21, 2024

k8s-ci-robot commented Nov 21, 2024

lfrancke left a comment

lfrancke Nov 21, 2024

nightkr Nov 21, 2024

lfrancke Nov 21, 2024

nightkr Nov 21, 2024

adrianmoisey Nov 21, 2024

nightkr Nov 21, 2024

adrianmoisey commented Nov 22, 2024

nightkr commented Nov 22, 2024

sftim Nov 22, 2024

sftim Nov 22, 2024

sftim Nov 22, 2024

nightkr Nov 26, 2024

sftim Nov 22, 2024

nightkr Nov 22, 2024

sftim Nov 22, 2024

nightkr Nov 26, 2024

sftim Nov 22, 2024

sftim Nov 22, 2024

nightkr Nov 26, 2024

sftim Nov 22, 2024

nightkr Nov 26, 2024

		Currently there is no way for cluster workloads to query for this domain name,
		leaving them either use relative domain names or take it as manual configuration.

	This KEP proposes adding a new Downward API for that workloads can use to request it.
	This KEP proposes adding a new Downward API that workloads can use to request it.

		This roughly shares the arguments for/against as [the ConfigMap alternative](#alternative-configmap),
		although it would allow more precise RBAC policy targeting.


		## Design Details

		A new Downward API `clusterPropertyRef: clusterDomain` would be introduced, which can be projected into an environment variable or a volume file.

KEP-4969: Cluster Domain Downward API #4972

Are you sure you want to change the base?

KEP-4969: Cluster Domain Downward API #4972

Conversation

nightkr commented Nov 21, 2024

k8s-ci-robot commented Nov 21, 2024

k8s-ci-robot commented Nov 21, 2024

k8s-ci-robot commented Nov 21, 2024

lfrancke left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adrianmoisey commented Nov 22, 2024

nightkr commented Nov 22, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment