Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Trying to get drake transient workers going via {future.clustermq} #237

Closed
pat-s opened this issue Feb 27, 2021 · 1 comment
Closed

Trying to get drake transient workers going via {future.clustermq} #237

pat-s opened this issue Feb 27, 2021 · 1 comment

Comments

@pat-s
Copy link

pat-s commented Feb 27, 2021

I was playing around today with {future.clustermq} and wanted to share my experience. Maybe @mschubert has some thoughts here to push this initiative by @HenrikBengtsson.
This relates to futureverse/future#204.

I've started with @wlandau {drake} instead of {targets} because I haven't dived yet into the latter.
But this is more about {future.clustermq} than about drake vs. targets and if this one works at some point, porting it to drake might not be that hard hopefully.

I got stuck at the step sending the future to the worker for evaluation.
The following snippet should get you to https://github.com/pat-s/future.clustermq/blob/e0a25dc4cedd12663aca1a561cff9a72e00ec3b4/R/ClusterMQFuture-class.R#L240 where I am trying to evaluate the expression.

Note: It could be that the semantics for the workers object at the linked reference are totally wrong - I do not have deep knowledge of the {clustermq} internals nor the {future} internals.
I was just playing around with what was already there.
In essence this is just a start to push this initiative a bit and maybe we can all do this together.

# remotes::install_github("pat-s/future.clustermq@test)

library(drake)

# Create the template file. You may have to modify it.
# drake_hpc_template_file("slurm_clustermq.tmpl")

# Configure clustermq.
# options(clustermq.scheduler = "slurm", template = "slurm_clustermq_future.tmpl")

options(clustermq.scheduler = "multiprocess")
suppressPackageStartupMessages(library(future.clustermq))
# FIXME: is this even correct?
future::plan("clustermq")

load_mtcars_example()
make(my_plan, parallelism = "future", jobs = 1)
@mschubert
Copy link
Owner

Plan to address this is #119

@mschubert mschubert closed this as not planned Won't fix, can't repro, duplicate, stale May 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants