Controlling nested parallelism with environmental variables

[alexpghayes]

I finally made the plunge and switched my BLAS implementation to the Intel MKL library. This creates a small problem when running targets workers in parallel: targets is spawning workers, and then MKL will spawn workers within each targets workers. For my targets workflow, I want the parallelism to come exclusively from targets. But the rest of the time (i.e. interactive dev) I want MKL to run in parallel mode.

I can set the MKL to serial evaluation by setting the environment variable MKL_NUM_THREADS=1. How can I pass this to targets? I don't want to set this in my .Renviron file because I don't want this behaviour globally.

[wlandau]

Maybe Sys.setenv(MKL_NUM_THREADS = 1) somewhere in _targets.R? _targets.R is kind of like its own .Rprofile for tar_make(), tar_visnetwork(), etc. (see correction below)

[wlandau]

Sorry, that actually will not work for tar_make_clustermq() or tar_make_future() in the general case because local environment variables do not get sent to the workers. If you do not want to modify the project-level .Renviron file, you can pass MKL_NUM_THREADS = 1 to env in the callr_arguments argument of tar_make*() and various other functions. Most arguments from https://callr.r-lib.org/reference/r.html can be passed this way.

[alexpghayes]

Didn't realize I could have a project level .Renviron, but that's a perfect solution in retrospect. Thanks for looking into this!

Will changes to .Renviron invalidate cached results?

[wlandau]

Didn't realize I could have a project level .Renviron, but that's a perfect solution in retrospect.

Oh, well I didn't realize this would appeal to you. If so, I recommend it.

Will changes to .Renviron invalidate cached results?

Not in your case. If the target value itself depends on an environment variable, it should be invalidated but won't be in the general case. (To invalidate a target depending on the value of an environment variable, it is best to assign it to a global object in _targets.R like you would a function so targets can watch it for changes. Or you could set it in a target that always runs using tar_target(Sys.getenv(...), cue = tar_cue(mode = "always")), but I think that would be overkill.)