Why doesn't /meta get sent to aws bucket?

[riazarbi]

Hello,

Why is it that, when I set an s3 storage backend, it sends all my _targets/objects contents, but not my _targets/meta objects?

This is an issue in my (probably super niche) use case.

We typically spin up a fresh R docker container, clone in a git repo, execute relevant code, and then spin down the container. This works well for projects where we can cleanly separate code and data, and store them in git and s3, respectively.

So a typical script would do something along the lines of

pull_s3_data()
load_data()
perform_function()
send_result_to_s3()

This is very close to the targets workflow, with the notable exception that a tar_make() call will skip up-to-date targets.

I am now realizing that this skipping is only possible if one has a record of the object hashes, which are stored in meta. And if my meta objects are not sent to s3, then my workflow will recompute from scratch each time, which is inefficient.

I could commit meta to git each time, I suppose, but it feels weird to send data to a code repo. I could manually send meta to s3, as well. But then I might as well dispense with the entire cloud backend functionality of the targets package and just run

pull_objects_and_meta_from_s3()
tar_make()
send_objects_and_meta_to_s3()

Which is easy and fine, but leaves me wondering - is there a particular reason meta is not sent to s3?

[wlandau]

The pattern I had envisioned is for users to select a small number of large data objects to push to S3 and then keep the rest of _targets/ on Git/GitHub, including metadata. Also, in theory you could have different targets going to different buckets, and it is unclear which bucket _targets/meta/meta would go to in that case.

If you want all of _targets/ to live in a single bucket, including metadata, I suggest something like aws.s3::s3sync().

[riazarbi]

Cool, thanks for the reasoning. I'll wrap in a s3sync