[help] Using tar_destroy with cloud storage

[jdlom]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

Hi all targeters,

I would like to know the best strategy to use tar_destroy.

For the context, I store the data on cloud store and I want to know how to use the most efficient way.
For now, I defined a tar_resources_aws in a script in the subfolder R.

minio_resource = tar_resources_aws(
  endpoint = endpoint,
  bucket = bucket,
  prefix = prefix
)

It works like a charm with tar_make() because of the tar_source().
Unfortunately, tar_destroy doesn't work if

tar_option_set(
  resources = tar_resources(
    aws = minio_resource
  ),
  repository = "aws"
)

is not run on the globalenv.

Moreover, I constat that cleaning the "local" store before "cloud" store does not work properly.

tar_destroy("local") # works fine. cleans the local store
tar_destroy("cloud") # doesn't clean the store in the cloud
tar_make()
tar_destroy("all") # works correctly. cleans both

Maybe I do the wrong way or I miss something.

Thanks in advance,

[wlandau]

Unfortunately, tar_destroy doesn't work if ... is not run on the globalenv.

That's surprising because targets stores the endpoint, bucket, and prefix in _targets/meta/meta and should not need to look at _targets.R to remove objects from the bucket. Have you tried a simple example that isolates the problem, as opposed to a fully scaled-up pipeline which may have other things that cause the problem? When it doesn't work, do you see the endpoint in tar_meta()$path?

Moreover, I constat that cleaning the "local" store before "cloud" store does not work properly.

The local _targets/meta/meta has information targets needs in order to delete objects from the cloud. So if you are not using "all", then please use "cloud" before "local".