S3 Put IfMatch

[tustvold]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

S3 finally support this, so we can enable this by default 🎉

Describe the solution you'd like

Describe alternatives you've considered

Additional context

https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3-functionality-conditional-writes/

[bentsku]

Do they support the full IfNoneMatch header? From the documentation, it seems IfNoneMatch is still limited to *, but IfMatch is the one with the ETag support? I suppose the combination of the two still enables the use-case, right? I'll work to implement this in LocalStack shortly 😄

[tustvold]

I think the docs haven't been updated yet, they often lag press releases by a day or so.

Edit: I typoed the wrong header

[bentsku]

I think I'll be able to release this feature probably only next week, as we're going to only release patch changes before re:invent. It should be available in the latest image after this Friday probably. I'll try to see if I can get it before that, but I doubt it 😄 I'll update you if anything changes

[benesch]

I put up two PRs that should tackle this on the object-store side:

• object-store: support real S3's If-Match semantics #6801 tweaks the implementation of S3ConditionalPut::ETag to support S3's specific If-Match semantics. I've tested this against real S3 and it works, so this is ready to be merged (pending review, of course).
• object-store: remove S3ConditionalPut::ETagPutIfNotExists #6802 rips out S3ConditionalPut::ETagIfNotExists. This will require that CI can exercise S3ConditionalPut::ETag against localstack, so this is blocked until we have a version of localstack that supports If-Match (thanks, @bentsku!).

[bentsku]

I've started working on the LocalStack implementation of S3 PutObject and CompleteMultipartUpload IfMatch, I have everything working now, just a matter of updating our internal specs, so it might be ready tomorrow evening.

I just wanted to share a finding I had around the CompleteMultipartUpload behavior that might be unexpected, I haven't looked at your implementation and you might already cover this.

Similarly to IfNoneMatch behavior with deletion of the object between a CreateMultipartUpload and CompleteMultipartUpload which still requires a full "restart" of the process as documented on the docs in AWS:

Conditional writes with If-None-Match header
[...]
When using CompleteMultipartUpload, the entire multipart upload must be re-initiated with CreateMultipartUpload to upload the object again after receiving a 409 Conflict error.

However, something that is undocumented for IfMatch is that it has a similar behavior for PutObject requests between Create and Complete:

If you create an object with a A key with the fictional ETag 123, create a multipart upload, upload a new object still with the A key with either a different ETag 456 or even with same content and thus 123 ETag, the Complete operation will fail with 409. The object needs to be "untouched" between the Create and Complete part for the operation to succeed.
AWS says that 409 exceptions should be retried, but I've tried and this one also necessitates a "restart" of the multipart upload operation.
I suspect they are also comparing the "initiated" time of the multipart upload to the current object, and if the current object is newer, they are raising a 409 exception.

I'm not too familiar with other cloud providers and I don't know their behavior, and I'm also not too familiar with conditional writes in themselves so maybe this behavior is expected and actually wanted, but it surprised me, so I wanted to share 😄

[benesch]

Ah, that is weird! Thanks very much for flagging.

The good news for us is that right now the multipart put operation exposed by this crate doesn't support conditions. The one place where we do use conditional multipart puts is in the implementation of copy_if_not_exists, but there we'll just propagate the 409 Conflict, if it occurs, back up to the user. Which kind of sucks, and maybe is the kind of thing we want to learn to handle and retry automatically, but it seems ~fine for the first version of this.

[bentsku]

Awesome, I'm glad to hear that! Great, I think it is simpler that way, the multipart uploads are a bit tricky. And it does seem fine, yes!

Also wanted to flag the fact that we do not really trigger 409 exception on PutObject, because right now the behavior is that we lock the object via a read/write lock, which allows us to check and prevent concurrent writes, and was easy to implement the following behavior for IfNoneMatch:

If multiple conditional writes occur for the same object name, the first write operation to finish succeeds. Amazon S3 then fails subsequent writes with a 412 Precondition Failed response.

But it's going to be harder to implement for IfMatch:

You can also receive a 409 Conflict response in the case of concurrent requests.

We'll just lock it and raise 412 if the provided ETag is not right. We'd need to implement semaphores and raise instant exception if multiple concurrent writers, only if IfMatch is provided, which is a bit out of scope, and I'm not too sure there's a concrete use-case except the retrying logic. As the locking seems to be working now (we've had issue before, which you greatly helped realize and fix thanks to your test suite, see #5283), I think I'll leave it as it is for now 😄

Anyway, just wanted to share the possible differences you might encounter.

On another note, the latest image now contains the IfMatch feature, and will be released in 4.0.3 on Friday. Feel free to give it a try! I tried it myself with the #6802 branch, and tests are passing with the AWS_CONDITIONAL_PUT=etag and AWS_COPY_IF_NOT_EXISTS=multipart values set.

[alamb]

label_issue.py automatically added labels {'object-store'} from #6682