source.store resolves before public access is set to the S3 file

[and2]

Calling source.store on AWS S3 resolves likely before the ACL policy is set, resulting in a 403 error when accessing the image straight away

Steps to replicate

const source = tinify.fromFile("large.jpg");
await source.store({ 
  service: "s3",
  aws_access_key_id: "AKIAIOSFODNN7EXAMPLE",
  aws_secret_access_key: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
  region: "us-west-1",
  headers: {
    Cache-Control: "public, max-age=31536000"
  },
path: "example-bucket/my-images/optimized.jpg"
});

Open path_to_example_bucket/my-images/optimized.jpg

Expected:
source.store to resolve when the optimized file is public.

[rolftimmermans]

Hi! The ACL that is used is the public-read canned ACL. This policy is set as soon as the optimised image is uploaded to S3.

Maybe the bucket is configured to disallowed public reads? Are you able to share (a screenshot of) the bucket permissions with us via [email protected]?

[and2]

Thanks for getting back to me.

So, the image will eventually be uploaded and made public, there's no issue with that.

Our problem is when this happens.

As it stands, source.store resolves before the tinified / resized image is actually available on the bucket.

From our tests, it can take anything from a couple of hundred milliseconds to 10+ seconds (depending on image size) until the image is available in S3. Again, this is after source.store resolved.

The immediate result is a 403 error which is why I assumed the ACL.

Hope this explains it a bit better.

This becomes an issue when one attempts to load that image immediately after a call.

[rolftimmermans]

Thanks for the clarification. A 403 error could indicate the file is (still) missing. As far as we're aware, this is expected behaviour of AWS S3. After upload succeeds, the image may not be immediately available as it propagates through the S3 infrastructure. However, to be sure it might be advisable to contact AWS support about this.

In any case, the Tinify API should not return a successful response until the underlying AWS S3 API call has completed successfully. And as long as you await on the response with await source.store(...), which you appear to do, then the returned promise should not resolve until the AWS S3 API call was successful.

The ACL is applied when the image is created, so I do not expect this is a factor.

[and2]

Thanks for looking into this further.

Could be an S3 issue, but we don't experience this when we upload to S3 directly.

The following works as expected:

const source = tinify.fromFile(filePath)
await source.toFile(newFilePath)
await s3.uploadFile(newFilePath)

Where s3.upload file is a function that returns a fulfilled promise when the S3 client completes the upload. The image url is available straight away.

[rolftimmermans]

Thanks for the follow-up. I agree that it's suspicious that you do not observe this effect if you upload to S3 yourself.

We'll try to investigate further to see if we can find something that explains this behaviour.