Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add options for max memory, n_jobs #219

Open
mschubert opened this issue Nov 19, 2020 · 0 comments
Open

Add options for max memory, n_jobs #219

mschubert opened this issue Nov 19, 2020 · 0 comments

Comments

@mschubert
Copy link
Owner

One of the ideas of the option-based configuration of clustermq is that code should be portable between compute environments, e.g. when moving to a different HPC scheduler.

If moving from HPC to local parallelization (even on a heavy duty server), resources will be much more limiting.

For a multicore setup, it would be nice to provide options for setting limits on total concurrent jobs (defaulting to cores/available cores) and memory (defaulting to memory available on that machine)

This way HPC scripts would not need to be adjusted, but could run (at reduced capacity) on multicore environments

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant