AWS: Add support for enabling access to S3 Requester Pays bucket

[blitzmohit]

Took a stab at resolving #11912

• Adds a new option s3.requester-pays-enabled which defaults to false.
• Modify the default aws client factories to add the requester pays header via override configuration if the option is enabled

Note will also need to document the option, hoping to get some feedback on the PR before that esp. since this is my first time working with iceberg.

[steveloughran]

This isn't the way to do it. Just call withRequesterPays(true) when creating the aws s3 client and let it do all the work

[blitzmohit]

This isn't the way to do it. Just call withRequesterPays(true) when creating the aws s3 client and let it do all the work

Hi,

Thank you for reviewing the PR. I was unable to locate withRequesterPays(true) or any other built-in option to set the RequesterPays flag globally on the S3 client builder.

I do observe this method on individual requests such as UploadPartRequest and ListObjectsV2Request, etc.

However, in my opinion, this approach would be more challenging to maintain as it would require setting the option correctly before each request, including those that may be added in the future.

[blitzmohit]

Hi @steveloughran

I just wanted to follow up on my previous comment and would love to hear any additional thoughts or suggestions you might have.

I looked into the Hadoop S3A implementation, and it seems to use a similar approach by setting the requester-pays header, which I believe you had also reviewed.

Ref:

• HADOOP-14661. Add S3 requester pays bucket support to S3A hadoop#3962
• https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java#L245-L248