cp --from-fs --to-bucket

[asilvas]

Would be super handy for transferring data into S3.

[shanebdavis]

Yes! This is definitely on my list. You'd think it would be easy... but the way S3P does it's massively parallel bucket-scanning doesn't translate very well to the local file system. I was able to make a workable solution from-bucket-to-fs, but it's harder the other way around.

Still, it can and should be done!

[mbolek]

I was able to make a workable solution from-bucket-to-fs, but it's harder the other way around.

Could it be made available? :) @shanebdavis

[shanebdavis]

There is a 'quick and dirty' way to make this happen, but it has scaling limitations. Basically, I could load the list of all the source files to copy in memory and then simulate the S3 listBucket API against that list.

Would it be helpful if this was available, but it might blow up if you have too many local files? I'm guessing under 100,000 would be no problem. Possibly even under a million files would work. More than a million files might start to get out of control.

@mbolek @asilvas What do you think?

[mbolek]

@shanebdavis sorry, I might've hijacked the thread a bit, I was thinking about the S3 to local FS download specifically, which you've mentioned as I was able to make a workable solution from-bucket-to-fs :)
Or is this already the part you've just mentioned? Then I guess, millions of files might not be enough since buckets grow way to easily... 🙄 ;)