Compatibility with latest aws s3 standards (signature v4)

[kmarkow]

Spent a decent amount of time trying to get it work with official [email protected] library for nodejs.
Below I describe all problems I encountered and how I worked around them.

1. Started with docker-compose.yml and initialized my-bucket

  s3:
    image: adobe/s3mock
    ports:
      - 9090:9090
    command: --initialBuckets=my-bucket

2. Created aws s3 client

const aws = require('aws-sdk');

const s3 = new aws.S3({
    endpoint: 'http://localhost:9090',
})

await s3.putObject({
    ContentType: 'image/png',
    Key: 'my-key/image.png',
    Body: imageBuffer, // The actual image buffer
    Bucket: 'my-bucket',
}).promise()

3. After trying to upload my object to the bucket I would get the following error:

[error] message: The specified bucket does not exist., stack: NoSuchBucket: The specified bucket does not exist.

This error message was very confusing because the bucket was there and I could see it with listBuckets and I was able to upload to it with my postman query. What I found out later is that the aws-sdk is using subdomain bucket naming convention which adobe/s3mock is not compatible with. I was able to fix it with s3ForcePathStyle: true option on my s3 client configuration.

4. The next error I encountered was

Internal Server Error, stack: 406: null

After long investigation I found out that adobe/s3-mock is not compatible with latest version of aws signatures so I had to add signatureVersion: 'v3' to my s3 configuration.

5. Final working configuration looked like

const aws = require('aws-sdk');

const s3 = new aws.S3({
    endpoint: 'http://localhost:9090',
    s3ForcePathStyle: true,
    signatureVersion: 'v3',
})

await s3.putObject({
    ContentType: 'image/png',
    Key: 'my-key/image.png',
    Body: imageBuffer, // The actual image buffer
    Bucket: 'my-bucket',
}).promise()

TLDR:
It's a great s3 mock server but is not compatible with the latest s3 standards

1. It does not support subdomain bucket naming conventions which is used as the default in aws sdk
2. It does not support v4 of signature generation which is also used as the default in aws sdk

[davidvuong]

I'm experiencing a similar issue:

"aws-sdk": "2.355.0",

services:
  s3:
    container_name: s3
    image: "adobe/s3mock:2.1.0"
    ports:
      - "9090:9090"
      - "9191:9191"
    environment:
      - initialBuckets=bucket

import * as AWS from 'aws-sdk';

const s3Client = new AWS.S3({
  endpoint: 'http://localhost:9090',
  s3ForcePathStyle: true,
  signatureVersion: 'v3',
});

s3Client.listObjects({ Bucket: 'bucket' }, (err, data) => {
  console.log(err, data);
});

However, I get a different error.

org.springframework.web.util.NestedServletException: Request processing failed; nested exception is java.lang.StringIndexOutOfBoundsException: String index out of range: -1

When I don't specify s3ForcePathStyle and signatureVersion, I can't seem to even connect.

UnknownEndpoint: Inaccessible host: `bucket.localhost'. This service may not be available in the `us-east-1' region.

[agudian]

Let's track this as a issue to support V4 signing, and handle the pathStyleAccess=false support in a separate issue.

[agudian]

I've created #144 for the bucket-subdomains support.

[haffo]

Any update on signature v4 support ?

[agudian]

The funny part is, that we’re using V4 signature by default already for a long long time, as I saw recently in the code.

No one has looked into the actual problem at hand, though. A reproducible failing test case (in Java) would help a lot.

[agudian]

This was in fact a bug, the same as described in #121.