Error when using AWS SDK 2.18 for S3 operations

[bpfoster]

Upgrading to the newly release SDK 2.18, S3Mock is returning an error when our tests create a bucket on setup:

software.amazon.awssdk.services.s3.model.S3Exception: null (Service: S3, Status Code: 405, Request ID: null).

Our client is created like:

S3Client.builder()
       .region(Region.US_EAST_1)
       .credentialsProvider(
               StaticCredentialsProvider.create(AwsBasicCredentials.create("foo", "bar")))
       .endpointOverride(URI.create(s3Mock.getHttpsEndpoint()))
       .httpClient(new DefaultSdkHttpClientBuilder().buildWithDefaults(attributeMap))
       .build();

The SDK release notes for 2.18.0 mention:

It is important to know that EndpointProviders and the endpoint override on the client are not mutually exclusive. In all existing cases, the endpoint override is passed as a parameter to the the provider and the provider may modify it. For example, the S3 provider may add the bucket name as a prefix to the endpoint override for virtual bucket addressing.

There may be some behavior changes related to endpoint resolution as as the the endpoint resolution is now standardised across SDKs so this release also bumps the minor version of the SDK to 2.18.

Notably, there are some changes in S3: when including and endpoint override, the SDK will use virtual bucket addressing as long as the bucket is a valid DNS label. This is different from the previous behavior where this only happens if the endpoint override begins with 's3'.

Turning on request logging, I see a difference between requests issued with client 2.17:

[main] DEBUG org.apache.http.wire:73 - http-outgoing-0 >> "PUT /bucket HTTP/1.1[\r][\n]"
[main] DEBUG org.apache.http.wire:73 - http-outgoing-0 >> "Host: localhost:49405[\r][\n]"

and those issued with client 2.18:

[main] DEBUG org.apache.http.wire:73 - http-outgoing-0 >> "PUT / HTTP/1.1[\r][\n]"
[main] DEBUG org.apache.http.wire:73 - http-outgoing-0 >> "Host: bucket.localhost:49402[\r][\n]"

Note that the bucket name is now in the DNS address and not the HTTP path. The S3 docs also now reflect that.
I suspect, but haven't confirmed, that other operations besides CreateBucket will be affected by this as well.

Looks like dependabot has created you a PR to upgrade the client, which appears to be failing with the same error:
#878

[afranken]

@bpfoster S3Mock currently does not support bucket access by subdomain, only path style access is supported.
See #144 - I have an idea on how to implement this, but it may be a breaking change as that I may have to disable path style access for this to work.
This is currently tentatively scheduled for the last update of v2 in S3Mock, see CHANGELOG

It seems that up until 2.18, the SDKv2 used path style access as a default, or maybe used some negotiation to find out if path style access is needed.
Starting with 2.18, path style access needs to be enabled when creating the client:

S3Client.builder()
.serviceConfiguration(S3Configuration.builder().pathStyleAccessEnabled(true).build())
.build();

This is not a breaking change for clients of the SDK if you use S3 as a backend, but it is if you are using any backend that does not support subdomain access like S3Mock...

[afranken]

Just realized: this one will be a pain in the neck - it will essentially break every integration test in existence where the service accessing S3 is using AWS SDKv2 for Java... :(

[bpfoster]

Ah, sorry I missed #144, that's definitely related. Please feel free to close this if you see it as a duplicate.

That's good to know that we can force path style access like in your sample, though the further my test client configuration gets from the production client configuration, the less faith I have in the test...

If I read the release notes correctly, it seems that prior to 2.18, SDKv2 would use subdomain style access as a default, unless an endpoint override was configured - which obviously all S3Mock uses would have set.

I think you're right that SDK v2.18+ will break all existing tests using S3Mock which is not great.

If I completely ignore the complexities of actually doing it 😄 , ideally I think you would want to support both subdomain and path style access.
Specifically I say this because the way one of our CI runners executes, the s3Mock.getHttpsEndpoint() returns an IP address (not localhost or other hostname). This is at least one case where the SDK seems to automatically fall back to path style access regardless of other configuration. The same exact code, when running locally, will use localhost and therefore use subdomain style access. So, I'm not sure you could disable path style access.

[afranken]

I just updated the usages of AWS SDKv2 in S3Mock, fixing our immediate problem allowing us to update the SDK version.
I have started working on subdomain support, but can't say yet if the implementation is backwards compatible or not.