Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial planning phase: efficient data management for persistent workers #672

Closed
wlandau opened this issue Jan 13, 2019 · 3 comments
Closed

Comments

@wlandau
Copy link
Member

wlandau commented Jan 13, 2019

Following up on #561 (comment). My thoughts so far:

  • Minimize the data shuffling required to load the dependencies of each target on each persistent worker.
  • Maybe assign each target to the worker with the most dependency data already loaded. Here, there are tradeoffs with target runtimes when it comes to load balancing.
  • Maybe implement a mechanism that shares memory across workers. Maybe POSIX threads for local parallelism?

@mschubert, is this different from what you have in mind?

@wlandau wlandau changed the title Efficient distribution of dependency data shared across persistent workers Planning phase: efficient distribution of dependency data shared across persistent workers Jan 13, 2019
@wlandau wlandau changed the title Planning phase: efficient distribution of dependency data shared across persistent workers Initial planning phase: efficient data management for persistent workers Jan 20, 2019
@wlandau
Copy link
Member Author

wlandau commented Apr 3, 2019

Through #800 and related issues, I am reducing the size of the common data sent to the workers. We could also reduce the data sent over with each individual target and use existing worker data to come up with sensible worker affinities. @mschubert, were you thinking about something similar for #561 (comment)?

@wlandau
Copy link
Member Author

wlandau commented Jul 12, 2019

Looks like this is a major focus of clustermq going forward: #933 (comment), mschubert/clustermq#151, mschubert/clustermq#154, mschubert/clustermq#160, mschubert/clustermq#161.

@wlandau wlandau closed this as completed Jul 12, 2019
@wlandau
Copy link
Member Author

wlandau commented Jul 12, 2019

Will reopen if anything needs to happen in drake.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant