[help] How to handle mixed local and cloud storage

[tarensanders]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

I've been using cloud storage with great success, but ran into an issue when using a mix of cloud storage and local storage when trying to move between machines.

I'm using mixed storage methods to try to reduce the cost of cloud storage. I have a few targets that take a very long time to generate which get sent to the cloud, and then some much faster targets (some of which generate large objects) which make sense to keep locally rather than have to pay for them to be checked on each pipeline run. I'm using GCS, so as a sketch the pipeline looks something like:

library(targets)
library(tarchetypes)

tar_option_set(
  resources = tar_resources(
    gcp = tar_resources_gcp(bucket = "mybucket", prefix = "mybucket")
  )
)

list(
  tar_target(slow_target, foo(), repository = "gcp"), 
  tar_target(fast_target, bar(slow_target))
)

I then keep _targets/meta/meta under version control. This works great, but the issue is when I then go to setup another computer. I needed to use a HPC to rerun the slow targets so I cloned the repo and tried to rerun the pipeline, but I ran into an error that targets was trying to access a file that doesn't exist. In this example, targets would be looking for _targets/objects/fast_target, but because /objects doesn't exist this fails. In the end I just ended up copying in the local objects to resolve the issue, but that seems hacky.

In retrospect I probably should have used tar_invalidate on the local targets. But is there any other way to deal with this?

[Aariq]

What about instead of syncing _targets/meta/meta with version control, use tar_meta_sync()? I'm not sure how that handles non-cloud targets, but maybe it already handles this issue?

[tarensanders]

That's a good suggestion. The function description doesn't address this case:

For each metadata file, if the local file does not exist or is older than the cloud file, then the cloud file is downloaded to the local file path. Conversely, if the cloud file is older or does not exist, then the local file is uploaded to the cloud.

Looking at the source code for the sync function, it doesn't look like this would be handled.