Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for running tasks on a Dask cluster #2041

Closed
bouweandela opened this issue May 19, 2023 · 1 comment
Closed

Add support for running tasks on a Dask cluster #2041

bouweandela opened this issue May 19, 2023 · 1 comment
Labels
enhancement New feature or request

Comments

@bouweandela
Copy link
Member

Since iris 3.6, it is possible to use Dask distributed with iris. This is a great new feature that makes it possible to implement distributed computing (e.g. running a single recipe on multiple cluster nodes) in ESMValCore and ESMValTool. See #1714 for an example implementation. I would like to add this feature.

@bouweandela bouweandela added this to the v2.10.0 milestone May 19, 2023
@bouweandela bouweandela added the enhancement New feature or request label May 19, 2023
@bouweandela bouweandela changed the title Add support for computing on a Dask cluster Add support for running tasks on a Dask cluster May 25, 2023
@bouweandela
Copy link
Member Author

After gaining some more experience with Dask I'm not sure if this is a good idea because

  • diagnostic tasks are likely to use more memory than workers have
  • for preprocessing tasks, metadata handling may also exceed the amount of memory that is available on a Dask worker, though this could probably be fixed
  • for preprocessing tasks a more fine-grained approach would parallelize much better

@bouweandela bouweandela removed this from the v2.10.0 milestone Sep 21, 2023
@bouweandela bouweandela closed this as not planned Won't fix, can't repro, duplicate, stale Feb 12, 2024
@bouweandela bouweandela mentioned this issue Feb 12, 2024
15 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant