Support AWS SSO #114

iainelder · 2021-04-12T15:57:06Z

Does awscurl already support AWS SSO?

I tried just now and I seem to be missing something.

I'm using the version installed today directly from Github.

I have a config file with a profile that uses AWS SSO to get temporary credentials.

https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html

[profile example]
sso_start_url = https://d-1111111111.awsapps.com/start
sso_region = eu-west-1
sso_account_id = 111111111111
sso_role_name = AdministratorAccess
region = eu-west-1
output = json

The profile works with AWS CLI v2.

$ aws sts get-caller-identity --profile example
{
    "UserId": "AROAAAAAAAAAAAEXAMPLE:example",
    "Account": "111111111111",
    "Arn": "arn:aws:sts::111111111111:assumed-role/AWSReservedSSO_AdministratorAccess_1111111111111111/example"
}

But when I try to use it with awscurl I get a message about the token being expired.

$ export AWS_PROFILE=example
$ awscurl --region=eu-west-1 --service=s3 http://example.s3.eu-west-1.amazonaws.com/
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>ExpiredToken</Code><Message>The provided token has expired.</Message><Token-0>IQoJ[...]96F0=</Token-0><RequestId>D[...]W</RequestId><HostId>GeFN[...]kfUTs=</HostId></Error>
Traceback (most recent call last):
  File "/home/isme/.local/bin/awscurl", line 8, in <module>
    sys.exit(main())
  File "/home/isme/.local/pipx/venvs/awscurl/lib/python3.8/site-packages/awscurl/awscurl.py", line 500, in main
    inner_main(sys.argv[1:])
  File "/home/isme/.local/pipx/venvs/awscurl/lib/python3.8/site-packages/awscurl/awscurl.py", line 494, in inner_main
    response.raise_for_status()
  File "/home/isme/.local/pipx/venvs/awscurl/lib/python3.8/site-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http://example.s3.eu-west-1.amazonaws.com/

The text was updated successfully, but these errors were encountered:

okigan · 2021-04-12T16:10:50Z

Hmm, i think SSO was added after awscurl was created so I think that not supported yet

…

On Apr 12, 2021, at 8:57 AM, Iain Samuel McLean Elder ***@***.***> wrote: Does awscurl already support AWS SSO? I tried just now and I seem to be missing something. I'm using the version installed today directly from Github. I have a config file with a profile that uses AWS SSO to get temporary credentials. https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html [profile example] sso_start_url = https://d-1111111111.awsapps.com/start sso_region = eu-west-1 sso_account_id = 111111111111 sso_role_name = AdministratorAccess region = eu-west-1 output = json The profile works with AWS CLI v2. $ aws sts get-caller-identity --profile example { "UserId": "AROAAAAAAAAAAAEXAMPLE:example", "Account": "111111111111", "Arn": "arn:aws:sts::111111111111:assumed-role/AWSReservedSSO_AdministratorAccess_1111111111111111/example" } But when I try to use it with awscurl I get a message about the token being expired. $ export AWS_PROFILE=example $ awscurl --region=eu-west-1 --service=s3 http://example.s3.eu-west-1.amazonaws.com/ <?xml version="1.0" encoding="UTF-8"?> <Error><Code>ExpiredToken</Code><Message>The provided token has expired.</Message><Token-0>IQoJ[...]96F0=</Token-0><RequestId>D[...]W</RequestId><HostId>GeFN[...]kfUTs=</HostId></Error> Traceback (most recent call last): File "/home/isme/.local/bin/awscurl", line 8, in <module> sys.exit(main()) File "/home/isme/.local/pipx/venvs/awscurl/lib/python3.8/site-packages/awscurl/awscurl.py", line 500, in main inner_main(sys.argv[1:]) File "/home/isme/.local/pipx/venvs/awscurl/lib/python3.8/site-packages/awscurl/awscurl.py", line 494, in inner_main response.raise_for_status() File "/home/isme/.local/pipx/venvs/awscurl/lib/python3.8/site-packages/requests/models.py", line 943, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http://example.s3.eu-west-1.amazonaws.com/ — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

iainelder · 2021-04-12T17:35:59Z

It looks like it's supported in boto3 1.14.0/botocore 1.17.

boto/botocore#1988

https://github.com/boto/botocore/blob/develop/CHANGELOG.rst#1170

https://github.com/boto/boto3/blob/develop/CHANGELOG.rst#1140

I don't see a version specified in your requirements.txt. Which version are you using?

okigan · 2021-04-12T17:43:02Z

Boto is a “soft dependency” it will use what you’ve got into the system (it will check before invoking it)

…

On Apr 12, 2021, at 10:36 AM, Iain Samuel McLean Elder ***@***.***> wrote: It looks like it's supported in boto3 1.14.0/botocore 1.17. boto/botocore#1988 https://github.com/boto/botocore/blob/develop/CHANGELOG.rst#1170 https://github.com/boto/boto3/blob/develop/CHANGELOG.rst#1140 I don't see a version specified in your requirements.txt. Which version are you using? — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

iainelder · 2021-04-12T19:13:19Z

Hm, now I'm confused.

I installed awscurl in a virtualenv with pipx.

It appears the botocore package isn't actually installed in the virtualenv.

Am I missing something?

I'm not sure if this is the right place to look for the botocore verison; I'm not very sure about how Python dependencies work in this context.

$ pipx runpip awscurl freeze
awscurl @ git+https://github.com/okigan/awscurl@f60961b4c6725a21bc4532a04d36c7f756636054
certifi==2020.12.5
cffi==1.14.5
chardet==4.0.0
ConfigArgParse==1.4
configparser==5.0.2
cryptography==3.4.7
idna==2.10
pkg-resources==0.0.0
pycparser==2.20
pyOpenSSL==20.0.1
requests==2.25.1
six==1.15.0
urllib3==1.26.4

okigan · 2021-04-12T19:51:32Z

Have not used pipx (so can't comment too much about that) -- since awscurl does not specifically depends on boto you'd need to specify / install boto libarary

iainelder · 2021-04-12T20:24:14Z

According to requirements.txt awscurl seems to depend not on boto but on botocore.

awscurl/requirements.txt

Line 4 in f60961b

botocore

I don't understand why botocore isn't an installed dependency when it appears in requirements.txt.

Indeed botocore is optionally loaded in the code that parses the credentials.

awscurl/awscurl/awscurl.py

Lines 379 to 392 in f8f7664

    
           # try to load instance credentials using botocore 
        
           if access_key is None or secret_key is None: 
        
               try: 
        
                   __log("loading botocore package") 
        
                   import botocore 
        
               except ImportError: 
        
                   __log("botocore package could not be loaded") 
        
                   botocore = None 
        
               if botocore: 
        
                   import botocore.session 
        
                   session = botocore.session.get_session() 
        
                   cred = session.get_credentials() 
        
                   access_key, secret_key, security_token = cred.access_key, cred.secret_key, cred.token

That part comes after some custom parsing of the AWS config file.

Why is there custom parsing of the config file before using botocore?

I thought (but I might be wrong!) that botocore was capable of parsing the credentials file already.

If botocore were used, it might be possible to support AWS SSO profiles without any custom code (but I haven't tested that).

okigan · 2021-04-12T20:41:01Z

botocore would be used if you have it installed and don't have any other credentials already set.

Do you have environment variables set with old access_key, secret_key, security_token?

iainelder · 2021-04-12T20:55:53Z

I was quite sure of it. I will try to produce a repro for you when I have a moment.

iainelder · 2021-06-10T14:09:26Z

I hadn't forgotten about this issue. I just took a long time to find some time :-)

I've got an explanation for the behavior that I reported and a solution to my request. But first I want to explain something that was confusing me before.

I don't understand why botocore isn't an installed dependency when it appears in requirements.txt.

Because the installation process appears to ignore requirements.txt. Instead it follows setup.py.

In setup.py botocore is declared as an optional dependency.

awscurl/setup.py

Lines 20 to 28 in f60961b

    
           install_requires=[ 
        
               'requests', 
        
               'configargparse', 
        
               'configparser', 
        
               'urllib3[secure]' 
        
           ], 
        
           extras_require={ 
        
               'awslibs': ["botocore"] 
        
           }

To install optional dependencies you need to change the installation command.

To test variations of the installation commands, I will use the same command as before with the --verbose flag for some debugging output that makes it easier to see what's happening inside the script. The complete test command looks like this:

awscurl \
--verbose \
--profile example \
--region=eu-west-1 \
--service=s3 \
http://example.s3.eu-west-1.amazonaws.com/

Originally I had installed awscurl with just the required dependencies like this:

pipx install awscurl

The test command's verbose output contains lines like these:

'loading botocore package'
'botocore package could not be loaded'

So botocore could not be loaded because the package was not installed as a dependency.

Now I know to install awscurl with the optional awslibs feature like this:

pipx install awscurl[awslibs]

The test commands verbose output now contains lines like these:

'loading botocore package'

Implicitly we see that botocore was successfully loaded. This is because the package was installed as a dependency.

Have not used pipx (so can't comment too much about that)

Pipx is for avoiding dependency version conflicts for Python-based applications. Each package is installed to its own virtual environment. It's my first choice for installing awscurl and many other tools written in Python.

iainelder · 2021-06-10T14:40:00Z

But when I try to use it with awscurl I get a message about the token being expired.

Do you have environment variables set with old access_key, secret_key, security_token?

Yes, I surely did.

When I use tools that don't understand AWS SSO profiles (Terraform pre-0.14.6 is another example), I use a tool called aws2-wrap to bridge the gap. It generates temporary credentials from the SSO profile and writes them to the shared credentials file. As a command line tool, it allows me to use a single script to automate the credential refresh process for all SSO profiles.

The AWS CLI v2 ignores the shared credentials file for SSO profiles. It uses its own cache for temporary credentials. Those credentials have a different expiry time. That explains why I was seeing expired credentials for awscurl but not for AWS CLI v2.

My initial feature request was a little unclear because at the time I had forgotten about that detail.

Now that I understand more about how awscurl's dependency management and how it loads credentials, I'm ready to provide a PR to add support for AWS SSO.

The good news is, as awscurl already depends on botocore, it's a small change to make.

andreacavagna01 · 2021-07-26T13:50:08Z

As a workaround, you can use Leapp.
it generates AWS access, secret key, and an access token directly from AWS SSO credentials, making this automatically support AWS sso:
https://docs.leapp.cloud/use-cases/aws_sso/

azadsagar · 2023-05-29T08:46:07Z

But when I try to use it with awscurl I get a message about the token being expired.

Do you have environment variables set with old access_key, secret_key, security_token?

Yes, I surely did.

When I use tools that don't understand AWS SSO profiles (Terraform pre-0.14.6 is another example), I use a tool called aws2-wrap to bridge the gap. It generates temporary credentials from the SSO profile and writes them to the shared credentials file. As a command line tool, it allows me to use a single script to automate the credential refresh process for all SSO profiles.

The AWS CLI v2 ignores the shared credentials file for SSO profiles. It uses its own cache for temporary credentials. Those credentials have a different expiry time. That explains why I was seeing expired credentials for awscurl but not for AWS CLI v2.

My initial feature request was a little unclear because at the time I had forgotten about that detail.

Now that I understand more about how awscurl's dependency management and how it loads credentials, I'm ready to provide a PR to add support for AWS SSO.

The good news is, as awscurl already depends on botocore, it's a small change to make.

Where are we on this ? I would like to jump in if not started already.

azadsagar · 2023-05-29T09:54:17Z

I ended up using AWS Sigv4 Proxy. It's pretty simple and straight forward. All you need is docker container running in the background. Your credentials directory mounted inside the container (though I am sure binary can be used independently). Enviroment variable pointing which profile to use. (This is one time task).

Going forward make a normal curl request to localhost and make sure to manually add host header in the request. That's all.

okigan · 2023-05-29T12:58:09Z

I am traveling this week - I’ll follow up with this thread when I return. On May 29, 2023, at 4:54 AM, azadsagar ***@***.***> wrote: I ended up using AWS Sigv4 Proxy. It's pretty simple and straight forward. All you need is docker container running in the background. Your credentials directory mounted inside the container (though I am sure binary can be used independently). Enviroment variable pointing which profile to use. (This is one time task). Going forward make a normal curl request to localhost and make sure to manually add host header in the request. That's all. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>

iainelder · 2023-05-30T12:42:12Z

Where are we on this ?

I moved on from awscurl to use other tools. I haven't found a use case for awscurl since then.

Thanks for the tip about AWS Sigv4 Proxy.

I don't use aws2-wrap any more either. Now I recommend the AWS SSO CLI. Its exec command allows you to execute other commands in a subshell with session environment variables.

joewragg · 2024-04-16T09:23:02Z

+1 on this

A good workaround is to go to the AWS SSO start page and grab a temporary key and secret from there

eriweb · 2024-04-19T11:32:17Z

For any mac-users who have installed awscurl with brew; you can add botocore to its virtualenv like this: $(brew --prefix awscurl)/libexec/bin/python -m pip install botocore

sirianni · 2024-05-08T19:56:10Z

@iainelder thank you so much for this detailed comment #114 (comment) !!!

I did spent a bit of time debugging myself, but this oneliner saved me a bunch of extra time

pipx install 'awscurl[awslibs]'

okigan · 2024-05-08T20:37:26Z

Originally 'botocore' was selected to be optional to reduce the number dependencies installed.

Overtime, I have not heard complains of installing too many dependencies, but users do run into issues of not having 'botocore', so on next release I am inclined to make 'botocore' to be included by default...

TheFynx mentioned this issue Sep 14, 2023

[BUG] AWS Signing Requests Broken when URL has ports (SSH Tunnel) opensearch-project/opensearch-go#370

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support AWS SSO #114

Support AWS SSO #114

iainelder commented Apr 12, 2021

okigan commented Apr 12, 2021 via email

iainelder commented Apr 12, 2021

okigan commented Apr 12, 2021 via email

iainelder commented Apr 12, 2021

okigan commented Apr 12, 2021

iainelder commented Apr 12, 2021 •

edited

Loading

okigan commented Apr 12, 2021

iainelder commented Apr 12, 2021

iainelder commented Jun 10, 2021

iainelder commented Jun 10, 2021

andreacavagna01 commented Jul 26, 2021

azadsagar commented May 29, 2023

azadsagar commented May 29, 2023

okigan commented May 29, 2023 via email

iainelder commented May 30, 2023

joewragg commented Apr 16, 2024

eriweb commented Apr 19, 2024

sirianni commented May 8, 2024

okigan commented May 8, 2024

Support AWS SSO #114

Support AWS SSO #114

Comments

iainelder commented Apr 12, 2021

okigan commented Apr 12, 2021 via email

iainelder commented Apr 12, 2021

okigan commented Apr 12, 2021 via email

iainelder commented Apr 12, 2021

okigan commented Apr 12, 2021

iainelder commented Apr 12, 2021 • edited Loading

okigan commented Apr 12, 2021

iainelder commented Apr 12, 2021

iainelder commented Jun 10, 2021

iainelder commented Jun 10, 2021

andreacavagna01 commented Jul 26, 2021

azadsagar commented May 29, 2023

azadsagar commented May 29, 2023

okigan commented May 29, 2023 via email

iainelder commented May 30, 2023

joewragg commented Apr 16, 2024

eriweb commented Apr 19, 2024

sirianni commented May 8, 2024

okigan commented May 8, 2024

iainelder commented Apr 12, 2021 •

edited

Loading