[VirtualCluster] Clean up after removing node instances from a virtual cluster #441
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Chong-Li
changed the title
Chong-Li
changed the title
wumuzi520
reviewed
Jan 7, 2025
| @@ -38,6 +38,13 @@ bool VirtualClusterManager::UpdateVirtualCluster( | |||
| return false; | |||
| } | |||
|
|
|||
| // The local node is removed from a (indivisible) virtual cluster, we have to clean up | |||
There was a problem hiding this comment.
Now the virtual_cluster_data is only removed from virtual_clusters_ when virtual_cluster_data.is_removed(), but the raylet may not receive the removed publish event, so that the virtual_cluster_data(removed from gcs) will never be erased at raylet side.
wumuzi520
reviewed
Jan 8, 2025
| std::string local_node_instance_id_; | ||
| std::function<void()> local_node_cleanup_fn_; | ||
| /// The (indivisible) virtual cluster to which the local node belongs. | ||
| std::string virtual_cluster_id_; |
There was a problem hiding this comment.
local_virtual_cluster_id_
wumuzi520
reviewed
Jan 8, 2025
| @@ -32,19 +32,40 @@ bool VirtualClusterManager::UpdateVirtualCluster( | |||
| const auto &virtual_cluster_id = virtual_cluster_data.id(); | |||
There was a problem hiding this comment.
if (local_cluster_data.contains(local_node)) {
if(virtual_cluster_data.contains(local_node)) {
if (virtual_cluster_data.removed()) {
clean();
} else {
if (virtual_cluster_data.cluster_id() != local_cluster_data.cluster_id()) {
clean();
}
}
} else {
// normal shrink.
clean();
}
}
Signed-off-by: Chong Li <[email protected]>
wumuzi520
reviewed
Jan 9, 2025
| virtual_clusters_.erase(input_virtual_cluster_id); | ||
| } else { | ||
| // Whether the local node in the input data. | ||
| bool local_node_in_data = |
There was a problem hiding this comment.
Suggested change
| bool local_node_in_data = | |
| bool input_virtual_cluster_contains_local_node = |
wumuzi520
reviewed
Jan 9, 2025
| if (!virtual_cluster_id_set.contains(curr_iter->first)) { | ||
| auto virtual_cluster_data = curr_iter->second; | ||
| virtual_cluster_data.set_is_removed(true); | ||
| updated_subscribe(curr_iter->first, std::move(virtual_cluster_data)); |
There was a problem hiding this comment.
invoke subscribe directly.
This was referenced Jan 9, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
When gcs is removing a node instance from a virtual cluster, it has to drain the node firstly. Otherwise, the pending tasks will be staying at the node.
On the other hand, nodes' resource view are pulled to gcs periodically. So when gcs is figuring out which nodes should be removed from the virtual cluster, it may use an outdated resource view. Therefore, the
drain nodeoperation has to take care of the running tasks at the node.Related issue number
#409
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.