tar_render_docker?

[joelnitta]

I have an idea for a feature, wondering if others would find it useful.

tar_render() is a key function for me to generate the final report of my analysis. However, rendering to PDF often requires many external dependencies (latex, various latex packages, pandoc, etc). I typically run my targets workflows within a docker container, so I make sure those are all available. However, recently I've started switching to a different mode: I version-control my R packages with renv, and call external programs that run in docker containers as needed*.

Would it be possible to implement a slightly different version of tar_render() that has the same API but in the background runs everything in a container (e.g., rocker/verse)?

*This is particularly useful for bioinformatics workflows, which rely on a large number of custom programs that each execute some step in the workflow. This way I can use the docker containers in a modular fashion across projects instead of creating one massive container that includes all of the needed programs for one project.

[wlandau]

Neat idea. Do you have a public example of a workflow that uses Docker this way? Docker is blocked in my work's HPC environment, so I do not have hands-on practice with the technique you describe.

I know future can wrap a PSOCK cluster around a single container, which lets you run R commands in the container asychronously. I wonder if future could handle multiple containers. If it could, then targets::tar_make_future() could do it too.

[joelnitta]

Do you have a public example of a workflow that uses Docker this way?

Unforunately, no... but I have a 90% complete blog post that has been languishing for a couple of months 😅 This is the perfect motivation for me to finish that up! I will certainly let you know when I do so.

Actually, it occurs to me that this could be potentially useful for any R code, not just rendering Rmd. That is one case where the external dependencies are particularly obvious, but there are many packages that require external dependencies (e.g., gdal for sf, etc). I could see how it would be nice to have modular containers that can run code requiring those dependencies to avoid creating monolithic containers for each project (but either approach may be appropriate according to the situation).

I know future can wrap a PSOCK cluster around a single container, which lets you run R commands in the container asychronously. I wonder if future could handle multiple containers. If it could, then targets::tar_make_future() could do it too.

I am not familiar with the details of future, but the Docker examples in parallelly::makeClusterPSOCK look very similar to what I'm proposing to do. Some sort of tar_*() function that abstracts away the details would be great... but again I'm not familiar enough with how future or parallelly work to know precisely what would be needed.

Docker is blocked in my work's HPC environment, so I do not have hands-on practice with the technique you describe.

As you're probably aware, there are options to run containers that don't require root permissions, like udocker and singularity (both included in the parallelly::makeClusterPSOCK examples). Perhaps you could try one of those? I have used udocker before, but not singularity.

[gorkang]

I wonder if future could handle multiple containers

Not sure if I might be misunderstanding some details, but this is what I've been doing recently (see #437) with https://github.com/gorkang/jsPsychMonkeys. In there I am using tar_make_future to launch a dozen parallel docker containers so each of them can run an experimental task (jsPsych) in a browser in a clean isolated environment.

If this is useful, I can upload a testing paradigm, so it can be easily reproducible.

[joelnitta]

@gorkang I had a look at your code and I agree it's a similar situation, but I'm not sure if it's quite the same.

One difference with tar_render() is that tar_render() has functionality to track dependencies (other targets) in the Rmd and re-build it appropriately. I would like my hypothetical tar_render_docker() to do the same. I don't think tar_target() would be able to do that currently.

Also, I think my use case more closely matches @wlandau's response to your question:

Ideally, a target should return an R object and not create side effects (with format = "file" for output files being the main exception; see https://books.ropensci.org/targets/practices.html#how-to-define-good-targets). So I recommend doing everything you need to do with the Docker image within a single target and then returning the R object you get from all that processing at the end.

The above is basically what I'm proposing, but thinking there might be some way to abstract away the details involved with launching the container, mounting volumes, etc.

Anyways, a testing paradigm does sound like it might be useful!

[wlandau]

I am not familiar with the details of future, but the Docker examples in parallelly::makeClusterPSOCK look very similar to what I'm proposing to do. Some sort of tar_*() function that abstracts away the details would be great... but again I'm not familiar enough with how future or parallelly work to know precisely what would be needed.

If you only need a single container, future + targets should already be able to handle it. Sketch:

# _targets.R file
library(targets)
library(future)
library(parallely)
cl <- makeClusterPSOCK(
  rep("localhost", times = 2L),
  ## Launch Rscript inside Docker container
  rscript = c(
    "docker", "run", "--net=host", "rocker/r-parallel",
    "Rscript"
  )
  dryrun = TRUE
)
plan(cluster, workers = cl)
list(
  tar_target(...),
  ...
)

# R console
library(targets)
tar_make_future(workers = 2)

But I don't know if you can make a PSOCK cluster that wraps multiple containers, and even then I do not think you can direct individual futures to individual containers.

Docker requires a fair bit of configuration and system dependencies outside R, which makes me hesitate to immediately build something that specific directly into tarchetypes. But there are other things we can try first. What do you think about the proposed tar_hook_outer() from ropensci/tarchetypes#44? This would allow you to create your own wrapper function to run R code inside your container, then apply it to entire collections of targets in your pipeline.

[joelnitta]

tar_hook_outer(), including tidyselect to choose which targets to apply the hooks and allowing for arbitrary expressions looks perfect!

BTW the link above to #44 is pointing to the wrong repo: it should be #44 from tarchetypes, not targets.

[wlandau]

tar_hook_outer(), including tidyselect to choose which targets to apply the hooks and allowing for arbitrary expressions looks perfect!

Glad to hear it.

BTW the link above to #44 is pointing to the wrong repo: it should be #44 from tarchetypes, not targets.

Right, thanks. Now fixed.