Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[PR #793/fca0dc04 backport][stable-3] Fix k8s_drain runs into timeout with pods from stateful sets. #807

Conversation

patchback[bot]
Copy link

@patchback patchback bot commented Dec 10, 2024

This is a backport of PR #793 as merged into main (fca0dc0).

SUMMARY

Fixes #792 .

The function wait_for_pod_deletion in k8s_drain never checks on which node a pod is actually running:

            try:
                response = self._api_instance.read_namespaced_pod(
                    namespace=pod[0], name=pod[1]
                )
                if not response:
                    pod = None
                time.sleep(wait_sleep)

This means that if a pod is successfully evicted and restarted with the same name on a new node, k8s_drain does not notice and thinks that the original pod is still running. This is the case for pods which are part of a stateful set.

ISSUE TYPE
  • Bugfix Pull Request
COMPONENT NAME

k8s_drain

Copy link

SUMMARY
Fixes #792 .
The function wait_for_pod_deletion in k8s_drain never checks on which node a pod is actually running:
            try:
                response = self._api_instance.read_namespaced_pod(
                    namespace=pod[0], name=pod[1]
                )
                if not response:
                    pod = None
                time.sleep(wait_sleep)
This means that if a pod is successfully evicted and restarted with the same name on a new node, k8s_drain does not notice and thinks that the original pod is still running. This is the case for pods which are part of a stateful set.

ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME
k8s_drain

Reviewed-by: Mike Graves <[email protected]>
(cherry picked from commit fca0dc0)
@gravesm gravesm force-pushed the patchback/backports/stable-3/fca0dc0485bf3748b61ac547957617e1e66573be/pr-793 branch from 0684451 to 61d298d Compare December 11, 2024 13:27
Copy link

Copy link

Build succeeded (gate pipeline).
https://ansible.softwarefactory-project.io/zuul/buildset/1c0f069802014bb7afe6a1f0ea1d037b

✔️ ansible-galaxy-importer SUCCESS in 4m 18s
✔️ build-ansible-collection SUCCESS in 5m 59s

@softwarefactory-project-zuul softwarefactory-project-zuul bot merged commit c6bd8ab into stable-3 Dec 11, 2024
47 checks passed
@gravesm gravesm deleted the patchback/backports/stable-3/fca0dc0485bf3748b61ac547957617e1e66573be/pr-793 branch December 11, 2024 14:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants