KEP 5067: Pod Generation KEP creation

[natasha41575]

• One-line PR description: Create new KEP 5067: Pod Generation

• Issue link: Pod Generation #5067

• Other comments:

[k8s-ci-robot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[natasha41575]

/cc @tallclair

[sftim]

Some early feedback; hope it helps

[tallclair]

/assign @liggitt

Mind being the API approver for this?

[SergeyKanzhelev]

so when it is disabled on kubelet, it will stop setting it. Will it result in infinite loop of reconciliation as it sees that the observedGeneration is set on API server, but it is not setting it. So it wants to update it to empty, but API server doesn't allow it?

[natasha41575]

I think you are right... let me make sure I have this right

1. the kubelet has the feature enabled, and observedGeneration is set on a pod
2. the feature is disabled on the kubelet
3. the kubelet stops setting observedGeneration, so tries to "clear" it
4. when the API server sees the observedGeneration unset in the incoming, it preserves the existing value, not letting it be cleared
5. Repeat 3-4 forever

I'm not sure what is the right solution is. Is the best thing for the API server to allow observedGeneration to be cleared? I can see this being a problem with older clients that don't know about the observedGeneration field (similar to the risk you raised about older webhooks). But seems like making sure the feature can be disabled on kubelet is more important?

[SergeyKanzhelev]

kubelet can keep setting the observedGeneration if feature gate is disabled (it will either generate it from the pod spec if FG is on or just take whatever observedGeneration status already have and preserve it if FG is off)

@liggitt do we have a precedence here?

[liggitt]

We've certainly used presence of API fields to control kubelet behavior instead of or in combination with feature gates.

I think a reasonable path in the kubelet here is to use feature enabled || pod status has non-zero observedGeneration to decide whether to propagate metadata.generation to status.observedGeneration.

[natasha41575]

I think a reasonable path in the kubelet here is to use feature enabled || pod status has non-zero observedGeneration to decide whether to propagate metadata.generation to status.observedGeneration.

SGTM

[SergeyKanzhelev]

Depending on the complexity of the logic in kubelet to calculate the "new" value of observedGeneration, we may be better off by preserving the existing value if FG is disabled instead of calculating a new one. Thank you for confirming 

[natasha41575]

I think the right way to set the new value of observedGeneration is to have the kubelet pass down the value of whatever metadata.generation is when it starts the sync loop all the way down to wherever it updates the pod status. From perusing through the source code, I don't anticipate this logic to be very complicated. So if the main concern is the complexity of the logic, I think we should be fine continuing to update observedGeneration after the FG is disabled rather than preserving the existing value.

[johnbelamaric]

sgtm

[natasha41575]

I folded this into the body of the KEP under "How can this feature be enabled / disabled in a live cluster?"

I'll wait for @SergeyKanzhelev's confirmation before closing this thread.

[natasha41575]

/milestone v1.33

[k8s-ci-robot]

@natasha41575: You must be a member of the kubernetes/milestone-maintainers GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your Milestone Maintainers Team and have them propose you as an additional delegate for this responsibility.

In response to this:

/milestone v1.33

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[johnbelamaric]

I'm ok with this for PRR. I think there is one update needed capturing the results of the kubelet behavior discussion.

[johnbelamaric]

sgtm