The AWS CLI ignores attempts to use UTF-8 on Windows,

[Zaita]

Describe the bug

The AWS CLI ignores attempts to use UTF-8 on Windows, this causes failures when interacting with the SecurityHub API as this is returning UTF characters that then fail encoding.

This has been discussed previously here:
https://stackoverflow.com/questions/39301253/can-i-force-aws-cli-v1-to-use-utf-encoding

And there are instructions regarding encoding here:
https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html

The issue is that because the AWS CLI is packaging and launching it's own Python interpreter, it won't recognise standard environment variables.

I am using latest AWS CLI, the following command is used from Windows Command Prompt:
aws securityhub get-findings --filters "{\"WorkflowStatus\": [{\"Value\":\"NEW\",\"Comparison\":\"EQUALS\"}], \"SeverityLabel\": [{\"Value\": \"HIGH\", \"Comparison\":\"EQUALS\"}], \"RecordState\": [{\"Value\":\"ACTIVE\", \"Comparison\":\"EQUALS\"}] }" --sort-criteria "{\"Field\":\"FirstObservedAt\", \"SortOrder\":\"desc\"}" --page-size=100 --profile=SecurityReadOnly --debug > high_findings.json

This is the error that I get.

I have previously raised a support case with AWS, the outcome was to log a GitHub issue after multiple back-n-forth attempts to fix it.

Expected Behavior

The AWS CLI can successfully call the SecurityHub endpoint without an encoding error when the user is specifying UTF-8 as the encoding scheme.

Current Behavior

If the AWS SecurityHub Get-Findings call returns a UTF-8 character, an encoding error occurs:

Reproduction Steps

aws securityhub get-findings --filters "{\"WorkflowStatus\": [{\"Value\":\"NEW\",\"Comparison\":\"EQUALS\"}], \"SeverityLabel\": [{\"Value\": \"HIGH\", \"Comparison\":\"EQUALS\"}], \"RecordState\": [{\"Value\":\"ACTIVE\", \"Comparison\":\"EQUALS\"}] }" --sort-criteria "{\"Field\":\"FirstObservedAt\", \"SortOrder\":\"desc\"}" --page-size=100 --profile=SecurityReadOnly --debug > high_findings.json

On an account with a security finding that includes a UTF-8 character.

Possible Solution

Add a command line parameter to allow for encoding selection.

Additional Information/Context

At the moment, the workaround is the decompress the packaged Python and alter the cp1252.py file to force UTF-8 then re-compress it and ensure the CLI is using it. This is a very hacky workaround.

Support Case: 12249326911 has a bunch of context for this.

CLI version used

aws-cli/2.13.32 Python/3.11.6 Windows/10 exe/AMD64 prompt/off

Environment details (OS name and version, etc.)

Windows 10, command prompt

[RyanFitzSimmonsAK]

Hi @Zaita, thanks for reaching out. This issue has come up a couple times recently, and there are few things that might fix it. I will note that the AWS CLI uses the character encoding provided by your system.

• Change your system locale (per this Stack Overflow thread)
• Set the environment variable PYTHONUTF8=1
• Set the environment variable PYTHONIOENCODING=UTF-8

Let me know if any of those work for you. Thanks!

[Zaita]

Thanks. Can confirm using PYTHONUTF8=1 works.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.