[ideas] Help make deployment be "main" for filesystem-dependent tasks

[billdenney]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

I'm starting to move toward more use of remote systems in my targets pipeline. (And, I'm excited about crew/mirai to help make it easier.) The remote systems do not always (or even usually, for me) have the files that need to be loaded in the same filesystem location. So, I'd like to have file loading and unloading occur via deployment = "main" and the rest occur via a "worker" deployment.

I can definitely set deployment manually for all relevant targets, but I was wondering if there may be a reasonable way to simplify that.

My first brainstorm of how to simplify that would be to have a new targets::tar_option_get("deployment_filesystem") which would default to targets::tar_option_get("deployment") if it were NULL or it could be set (I'd assume typically to "main").

The targets where this would mostly apply in my use cases are the ones immediately downstream of tar_file() or tar_files() and also tar_read_file() and tar_render() targets. The hard part is for targets immediately downstream of tar_file() and tar_files() where inheritance of the deployment seems infeasible.

Something that may be much simpler would be to have some form of deployment-modifier function like tar_set_deployment(x, deployment) where it would take a list of targets and set all of their deployments. I tend to make target lists like l_load_files, so I could easily wrap that in tar_set_deployment() to say that all file loading should happen on the main system.

A more generalized thought rather than tar_set_deployment() could be something like tar_set() which could set any of the arguments to tar_targets() other than name and command for a list of targets.

Does something like this seem feasible and useful?

[billdenney]

It's not clear to me if this would fit better in targets or tarchetypes.

[wlandau]

I'm starting to move toward more use of remote systems in my targets pipeline. (And, I'm excited about crew/mirai to help make it easier.) The remote systems do not always (or even usually, for me) have the files that need to be loaded in the same filesystem location. So, I'd like to have file loading and unloading occur via deployment = "main" and the rest occur via a "worker" deployment.

If everything is stored locally, then storage = "main" and retrieval = "main" will also be important so those remote targets ship with all the data they need. If you use AWS/GCP buckets (e.g. repository = "aws"), then storage = "worker"andretrieval = "worker"` are also possible (and in fact recommended for efficiency reasons).

I can definitely set deployment manually for all relevant targets, but I was wondering if there may be a reasonable way to simplify that.

tar_option_set(deployment = "main") sets the default deployment argument of tar_target(), so you could do something like this.

# _targets.R file
library(targets)
tar_option_set(deployment = "worker")
on_worker <- list(
  tar_target(worker_1, run())
  tar_target(worker_2, run())
}
tar_option_set(deployment = "main")
on_main <- list(
  tar_target(main_1, run())
  tar_target(main_2, run())
}
list(on_worker, on_main)

[billdenney]

@wlandau Great, thanks! I didn't realize that tar_option_set() had an effect on the next targets created and not on everything. That solves the problem without a new function.