fix: Omit cleaning containerless kernels which are still creating its container #2317
+38
−9
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Your org has enabled the Graphite merge queue for merging into mainAdd the label “flow:merge-queue” to the PR and Graphite will automatically add it to the merge queue when it’s ready to merge. Or use the label “flow:hotfix” to add to the merge queue as a hot fix. You must have a Graphite account and log in to Graphite in order to use the merge queue. Sign up using this link. |
github-actions
bot
added
comp:agent
This was referenced Jun 20, 2024
This stack of pull requests is managed by Graphite. Learn more about stacking. |
1 task
fregataa
force-pushed
the
topic/06-20-fix_kernel_status_to_omit_being_cleaned
branch
from
f0985d1
to
17b9bfa
Compare
fregataa
changed the title
fregataa
force-pushed
the
topic/05-23-fix_enhanced_kernel_termination_handling
branch
2 times, most recently
from
bd7d1e4
to
25e70f3
Compare
fregataa
changed the title
This was referenced Jun 25, 2024
fregataa
force-pushed
the
topic/05-23-fix_enhanced_kernel_termination_handling
branch
5 times, most recently
from
552b746
to
59cbbc9
Compare
Base automatically changed from
topic/05-23-fix_enhanced_kernel_termination_handling
to
main
July 5, 2024 11:51
fregataa
force-pushed
the
topic/06-20-fix_kernel_status_to_omit_being_cleaned
branch
from
17b9bfa
to
dcca90e
Compare
fregataa
force-pushed
the
topic/06-20-fix_kernel_status_to_omit_being_cleaned
branch
from
c8c6916
to
a648eb3
Compare
fregataa
commented
Jul 12, 2024
| @@ -233,6 +235,9 @@ def __getstate__(self) -> Mapping[str, Any]: | |||
| return props | |||
|
|
|||
| def __setstate__(self, props) -> None: | |||
| # Used when a `Kernel` object is loaded from pickle data. | |||
| if "state" not in props: | |||
| props["state"] = KernelLifecycleStatus.RUNNING | |||
There was a problem hiding this comment.
When we shutdown and restart an agent to update its version, kernel_registry is dumped as a pickle file and the agent loads the pickle file when it restarts. Old Kernel objects that are dumped before the version update do not have state field when we restart agent.
We need to insert the state value to the old Kernel objects.
kyujin-cho
deleted the
topic/06-20-fix_kernel_status_to_omit_being_cleaned
branch
lablup-octodog
pushed a commit
that referenced
this pull request
Jul 15, 2024
… container (#2317) Co-authored-by: Kyujin Cho <[email protected]> Backported-from: main (24.09) Backported-to: 24.03 Backport-of: 2317
github-merge-queue bot
pushed a commit
that referenced
this pull request
Jul 15, 2024
… container (#2317) (#2468) Co-authored-by: Sanghun Lee <[email protected]> Co-authored-by: Kyujin Cho <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When we create a kernel, agent registers the kernel to
kernel_registrybefore creating an actual container.sync_container_lifecycles()task scanskernel_registryand deregister kernels that do not have an actual container.If the task scans at the moment before the container is created after the kernel is registered, the kernel being created gets removed, which is a malfunction.
Let's add
statefield toKernelobject so that the task can omit to clean kernels being created.Checklist: (if applicable)