Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't consider scheduler idle while executing Scheduler.update_graph #8877

Merged
merged 7 commits into from
Sep 24, 2024
Merged

Don't consider scheduler idle while executing Scheduler.update_graph #8877

Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
3cfbe01
Don't consider scheduler idle during update_graph
hendrikmakait Sep 19, 2024
d49f35d
Add test
hendrikmakait Sep 23, 2024
b08a33e
fix test
hendrikmakait Sep 23, 2024
dd92bc5
minor
hendrikmakait Sep 23, 2024
ed1d8bc
Update distributed/tests/test_scheduler.py
hendrikmakait Sep 24, 2024
1150c41
Adjust test plugin
hendrikmakait Sep 24, 2024
40f3a35
Update distributed/tests/test_scheduler.py
hendrikmakait Sep 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 12 additions & 2 deletions distributed/scheduler.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3686,6 +3686,7 @@ class Scheduler(SchedulerState, ServerNode):
_client_connections_removed_total: int
_workers_added_total: int
_workers_removed_total: int
_active_graph_updates: int

def __init__(
self,
Expand Down Expand Up @@ -4049,6 +4050,7 @@ async def post(self):
self._client_connections_removed_total = 0
self._workers_added_total = 0
self._workers_removed_total = 0
self._active_graph_updates = 0

##################
# Administration #
Expand Down Expand Up @@ -4841,6 +4843,7 @@ async def update_graph(
stimulus_id: str | None = None,
) -> None:
start = time()
self._active_graph_updates += 1
try:
try:
graph = deserialize(graph_header, graph_frames).data
Expand Down Expand Up @@ -4913,8 +4916,11 @@ async def update_graph(
# (which may not have been added to who_wants yet)
client=client,
)
end = time()
self.digest_metric("update-graph-duration", end - start)
finally:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just noting that this finally block is being used to ensure we always decrement the _active_graph_updates counter

self._active_graph_updates -= 1
assert self._active_graph_updates >= 0
end = time()
self.digest_metric("update-graph-duration", end - start)

def _generate_taskstates(
self,
Expand Down Expand Up @@ -8607,6 +8613,10 @@ def check_idle(self) -> float | None:
self.idle_since = None
return None

if self._active_graph_updates > 0:
self.idle_since = None
return None

if (
self.queued
or self.unrunnable
Expand Down
Loading