Skip to content

Commit

Permalink
Set n_workers for dask LocalCluster
Browse files Browse the repository at this point in the history
 - Explicity set n_workers for LocalCluster creation (through Client).
 If this is not set, for machines with a high CPU count, the number of
 workers will be much larger than 7. In `client.scatter` with
 `broadcast=True`, this can cause hangs due to how dask distributed works.
 It's presumed that this bug is only apparent in more recent runs on
 machines with high CPU counts, although that is not confirmed. But,
 this fix does allow the example to run to completion on a machine
 with CPU_COUNT=128.
  • Loading branch information
talumbau committed Feb 6, 2024
1 parent c0802c6 commit 6843966
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion examples/run_og_usa.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@

def main():
# Define parameters to use for multiprocessing
client = Client()
num_workers = min(multiprocessing.cpu_count(), 7)
client = Client(n_workers=num_workers)
print("Number of workers = ", num_workers)

# Directories to save data
Expand Down

0 comments on commit 6843966

Please sign in to comment.