[help] Advice on highly heterogeneous workflows

[connor-duffin]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

Our workflow is highly heterogeneous, ranging from single targets that use lots of memory on their own, to dynamically branched targets that are embarrassingly parallel. I have set up heterogeneous workers for the different parts of the pipeline on their own, but it seems they all run at the same time - in reality I was hoping that this would separate out the running of the pipeline, so that we don't run out of memory on our machine!

As a quickfix I am running something like:

targets::tar_make("highly_parallel_target")  # with lots of branches
targets::tar_make("expensive_target")
targets::tar_make()  # remainder of the targets

I was wondering if there was any advice on dealing with such a pipeline, and, if any, best practices that should be followed.