support AWS SSO authentication

[drmorr0]

The aws extension doesn't correctly work with the sso credential provider, as noted in a comment in #31 (and possibly also #42? It's unclear in that report which credential provider the user is using). As noted in PR #45, the aws-sdk-cpp library needs to include the sts feature for that credential provider to work; similarly, it needs the sso feature for this provider to work.

I've confirmed that my change fixes the behaviour (at least for me):

~/src/duckdb_aws > make release
<snip>
~/src/duckdb_aws > aws sso login
<snip>
~/src/duckdb_aws > build/release/duckdb
v1.0.0 1f98600c2c
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
D CREATE SECRET(TYPE S3, PROVIDER CREDENTIAL_CHAIN);
┌─────────┐
│ Success │
│ boolean │
├─────────┤
│ true    │
└─────────┘
D SELECT * FROM read_parquet('s3://my_bucket/*.parquet');
100% ▕████████████████████████████████████████████████████████████▏
...

I tried to look at the other credential providers that duckdb_aws (claims to) provide in the docs: afaict, config and env work now. sts was fixed (I believe) by #45, and this PR should fix sso. I "assume" that process works correctly, so the only one I'm not sure about is instance but this is just hitting the instance metadata store on an EC2 instance so I feel like it "should" work? The web identity token method described in #16 and #31 maybe just requires some other aws-sdk-cpp feature to be enabled, but I'm not really sure which one and don't have a good way to test that in any case.

Hope this is helpful!

[samansmink]

full disclaimer: I did not try this out, looks good though. Thanks!

[drmorr0]

I've been using it locally and it works fine. I'm not sure if you need/want the docs updated? You have to run aws sso login before you create the secret (and if your creds expire/change you need to recreate the secret).

[aksel]

Thanks a lot for this! I was pulling my hair out, trying to figure out why it wasn't working 😄

Do you expect this to be included in the next release, pending 2nd of September?

[samansmink]

@aksel yes, but you can already try this out on duckdb v1.0.0 by switching to the nightly build of aws using: force install aws from core_nightly

[aksel]

Cool, thanks a lot!

[antbz]

Hey! I seem to still be facing this issue with the SSO chain on v1.1.0

Sequence of commands I am running after I aws sso login

LOAD aws;
LOAD httpfs;

CREATE OR REPLACE SECRET awscredentials (
        TYPE S3,
        PROVIDER CREDENTIAL_CHAIN,
        CHAIN 'sso'
);

SELECT secret_string FROM duckdb_secrets();

Still, the output of secret_string is:

>>> c.sql("SELECT secret_string FROM duckdb_secrets()").show()
┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                                                  secret_string                                                                  │
│                                                                     varchar                                                                     │
├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ name=awscredentials;type=s3;provider=credential_chain;serializable=true;scope=s3://,s3n://,s3a://;endpoint=s3.amazonaws.com;region=eu-central-1 │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

Using the env chain works, but I'd like to understand why this doesn't. I have tried installing the nightly version of the aws extension, but I run into this error:

>>> c.sql(
...         """
...         FORCE INSTALL aws FROM 'http://nightly-extensions.duckdb.org';
...         """
...     )
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
duckdb.duckdb.HTTPException: HTTP Error: Failed to download extension "aws" at URL "http://nightly-extensions.duckdb.org/v1.1.0/osx_arm64/aws.duckdb_extension.gz" (HTTP 403)
Extension "aws" is an existing extension.

>>> c.sql(
...         """
...         force install aws from core_nightly;
...         """
...     )
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
duckdb.duckdb.HTTPException: HTTP Error: Failed to download extension "aws" at URL "http://nightly-extensions.duckdb.org/v1.1.0/osx_arm64/aws.duckdb_extension.gz" (HTTP 403)
Extension "aws" is an existing extension.

I did manage to do the same setup, with nightly aws extension, on a v0.10.1 environment, but using SSO also does not populate the secret correctly.

Any help? Am I missing something?

[drmorr0]

I've been meaning to come back to this as well, I've noticed that using the nightly version of the AWS plugin isn't working with SSO still. I don't quite understand why not, because the version that I compiled locally definitely did! I haven't had a chance to do any investigation yet, though :(

[antbz]

@drmorr0 thanks for your help! Confirming it's not just me already rules out other issues at least. For now I think I can make due with the env chain.