object_store: Conditional put and rename_if_not_exist on S3

[wjones127]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

This is now supported on S3: https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3-conditional-writes/

Describe the solution you'd like

We should make sure this works out-of-the-box. Also need to maintain compatibility with Minio, Localstack, and CloudFlare R2.

Describe alternatives you've considered

Additional context

[bentsku]

We are implementing it on LocalStack, should be available in latest by tomorrow: localstack/localstack#11402

From what I've seen, it's less powerful than R2, as you cannot specify any value other than * in If-None-Match (or it raises an exception), so you might need different code path between S3 and R2, for example. Also, they didn't implement it for CopyObject apparently, but it is also implemented for CompleteMultipartUpload. The logic is somewhat a bit complicated for multiparts, especially the "delete after initiation", and you'll often need to recreate a complete multipart upload.

I was already looking for the PR/feature request here when I read about that feature 😄

[rtyler]

@wjones127 👋 if you find yourself too busy with other things, I could probably take a crack at this 😄

[wjones127]

I haven't finished this.

But from what's I've looked at, it's already implemented. It's just a matter of changing the default behavior / configuration and documenting and testing that.

[fhilgers]

How do you plan on implementing ObjectStore::copy_if_not_exists. The new conditional features from aws only allow checking etag on the source for copy operations. x-amz-copy-source-if-match and x-amz-copy-source-if-match only check the source, not the destination as far as i understand (https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html). Is there a workaround through the conditonal Put operations?

Minio already supported these headers prior to AWS, but has anyone tried to use Minio for the conditional operations without dynamodb?

[fhilgers]

One idea maybe would be to use the conditional Put with the if-none-match header to create a dummy file and if that succeeds, copy the file. However this is not atomic and in the case where a client willingly overwrote the dummy file prior to the copy completing, their file would be overwritten by the copy?

Is this that bad? In the cases where clients use conditonal operations everything is safeguarded and in the cases where clients already just overwrite a file it does not change anything because there is no guarantee that another client will not overwrite anything. In terms of clients making sure that there is no overwriting happening, conditional put on a client and then force copying should semantically be the same as conditional copy.

Deleting and other operations would be a problem though. What happens if the dummy file is deleted prior to copying? Get with have empty files, etc.

Clients need to then ignore markers on list, delete and get and only honor them for put and copy.

[fhilgers]

After playing around a bit, I found out that this can be worked around just by using multipart copy:

1. Initiate Multipart Upload
2. Upload Part Copy
3. Supply if-none-match to Complete Multipart Upload

Using the header just for CopyObject results in an NotImplemented error.

[benesch]

I put up #6682, which implements both PutMode::Create and copy_if_not_exists for native Amazon S3. I'm unfamiliar with the maintainership for object_store so if someone can tag in the right folks to review, I'd be much obliged!

After playing around a bit, I found out that this can be worked around just by using multipart copy:

1. Initiate Multipart Upload
2. Upload Part Copy
3. Supply if-none-match to Complete Multipart Upload

@fhilgers thanks for the pointer here. Indeed that seems to work as expected.

[tustvold]

FYI the support has been broadened to not just wildcards

#6799

[benesch]

PRs are up! #6799 (comment)