Error when copying to s3

[mattf10]

Describe the bug

Since updating to the latest version (aws-cli/2.14.5 Python/3.11.6 Linux/5.15.0-89-generic exe/x86_64.ubuntu.22 prompt/off), I can no longer copy to s3 via the command line. The error is:

module '_awscrt' has no attribute 's3_is_crt_s3_optimized_for_system'

Expected Behavior

I expected files to be copied to s3 bucket

Current Behavior

Files are not copied

Reproduction Steps

From the comman line, issue the following command:

aws s3 cp 'path to folder' s3://'bucket' --recursive

Possible Solution

Problem started happening after upgrading aws-cli v2 to latest version (2.14.5)

Additional Information/Context

Same problem started happening this morning on two different VPS, both updated to this version before running s3 cp command (other VPS on older versions are fine).

Using previous version (2.13.38) fixes the problem.

CLI version used

2.14.5

Environment details (OS name and version, etc.)

Ubuntu 22.04.3

[ghnp5]

Getting the same here, but for ls:

aws s3 ls --recursive s3://bucket-xyz/path-xyz/

module '_awscrt' has no attribute 's3_is_crt_s3_optimized_for_system'

RC=255

--

on AlmaLinux 8

[fkhn]

Same error for s3 sync

aws --version => aws-cli/2.14.5 Python/3.11.6 Linux/6.1.0-13-amd64 exe/x86_64.debian.12 prompt/off

[tim-finnigan]

Hi @mattf10 thanks for reaching out. The AWS CLI CHANGELOG notes a few recent changes related to awscrt. Starting in AWS CLI 2.14.2, version 0.19.18+ of awscrt is required. (Also here are the release notes for that dependency.)

Can you confirm that you meet those version requirements? I just tested installing 2.14.5 of the AWS CLI using the documented methods for Mac and Linux and was able to run S3 commands.

[ghnp5]

Thank you.
awscrt doesn't seem to be mentioned in the install instructions you linked (the last link of your comment).

I don't seem to have awscrt installed in pip/python3 at all.

I'd have thought the awscli install would add any required dependencies.

[mattf10]

I don't have awscrt installed - it's never been needed until now. To confirm, I download the latest version of awscli via curl, like so curl -s "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip".

So you're saying that this method is no longer sufficient, and that I now need to install pip and also add/update the awscrt package? If this is a required package, shouldn't it be included in the awscli zip?

Thanks

[tim-finnigan]

Thanks for following up. How did you install/update the AWS CLI? Have you tried reinstalling the AWS CLI using one of the officially supported methods linked earlier? Here is where the the version floor was bumped for awscrt. Installing/updating the AWS CLI should install the required dependency version unless you're pinning to an earlier version.

[corybryant]

@tim-finnigan That is the officially supported method that was linked. The zip file is the pre-compiled binary containing three .so files for awscrt. Dealing with dependencies on our end is not the officially recommended method.

[corybryant]

Version 2.14.4 is working correctly. 2.14.5 does not list any changes for awscrt in the changelog.

Also, the signature file for the officially supported install method has expired so we can't run verification checks when installing/updating the CLI.

[tim-finnigan]

I installed 2.14.5 on Mac, Amazon Linux 2023, Ubuntu 22 and was able to run s3 commands like cp and ls fine. Can you provide more steps to reproduce this or additional details on your environment? Do you have something else installed like https://github.com/aws/aws-iot-device-sdk-python-v2 that might have a conflicting CRT version?

[corybryant]

Nothing else installed, Alma Linux 8:

aws --version

aws-cli/2.14.5 Python/3.11.6 Linux/4.18.0-477.27.2.el8_8.x86_64 exe/x86_64.almalinux.8 prompt/off

/bin/curl -s -o awscliv2.sig https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip.sig
/bin/curl -s "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
/bin/gpg --verify awscliv2.sig awscliv2.zip && /bin/unzip -uo awscliv2.zip && ./aws/install --update

[corybryant]

It seems our automated system tool is leaving the aws install directory around and unzipping on top of it when it's time to update so something in 2.14.5 must be conflicting with an old version. Clearing out the install directory and doing a full reinstall seems to have fixed the s3_is_crt_s3_optimized_for_system error for us.

I can open a new issue for the expired SIG file. Ignore this... I see the updated GPG key in the docs.

[ghnp5]

It seems our automated system tool is leaving the aws install directory around and unzipping on top of it when it's time to update so something in 2.14.5 must be conflicting with an old version. Clearing out the install directory and doing a full reinstall seems to have fixed the s3_is_crt_s3_optimized_for_system error for us.

Looks like that was it for me, too!! Thanks!

Added rm -rf aws to the update script,
then also cleared /usr/local/aws-cli/v2/ so I could re-install the same version,
and looks like I'm no longer getting the issue.

[tim-finnigan]

Thanks for letting us know, glad you were able to resolve the issue. @mattf10 can you confirm if removing the aws install directory solves the issue for you as well?

[fkhn]

it does for me. thanks.

[mattf10]

@tim-finnigan Yes. After first removing all versions from the install directory, then installing the latest version (2.14.5), I can now issue commands without the error.

Related: #5979. Over time, the install directory becomes huge (it was > 10G on my VPS). I added the following code to the end of my update script to fix: find /usr/local/aws-cli/v2/ -maxdepth 1 -type d -mtime +14 -exec rm -fr {} \; - removes all versions more than 2 weeks old.

I suggest that the officially support update method add 'rm -fr' to a) avoid the old dependency problem and b) not to take up disk space. Or some flag to that effect.

Thanks

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

[ghnp5]

Related: #5979. Over time, the install directory becomes huge (it was > 10G on my VPS). I added the following code to the end of my update script to fix: find /usr/local/aws-cli/v2/ -maxdepth 1 -type d -mtime +14 -exec rm -fr {} \; - removes all versions more than 2 weeks old.

Yeah, I had noticed this a few months ago, and had to do add a similar command to all my servers!

[mattf10]

@tim-finnigan - this is still a problem on all of my VPS that use version 2.14.5. I cleared all old versions and tested by checking with 'aws s3 ls' on a bucket. This command worked so I thought the issue was fixed.

However, my nightly back-ups failed to run with the original error message. So there must be some other reason why the 'awscrt' lib is missing or outdated.

The command that fails is: aws s3 cp 'path to folder' s3://'bucket' --recursive.

The error message is: module '_awscrt' has no attribute 's3_is_crt_s3_optimized_for_system'

The awscli version is: aws-cli/2.14.5 Python/3.11.6 Linux/5.15.0-89-generic exe/x86_64.ubuntu.22 prompt/off

Server version: Ubuntu 22.04.3 LTS (GNU/Linux 5.15.0-89-generic x86_64)

The commands used to update awscli:
curl -s "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip -oq awscliv2.zip
./aws/install --bin-dir /usr/local/bin --install-dir /usr/local/aws-cli --update

I checked the install dist folder '/usr/local/aws-cli/v2/current/dist' and can see three files related to 'awscrt' - here are the results of ls -la:
-rwxr-xr-x 1 root root 24738475 Dec 5 09:27 _awscrt.abi3.so
-rwxr-xr-x 1 root root 23710739 Dec 5 09:27 _awscrt.cpython-311-x86_64-linux-gnu.so
-rwxr-xr-x 1 root root 23313825 Dec 5 09:27 _awscrt.cpython-39-x86_64-linux-gnu.so

Are these the correct versions? Is there some way to check why they're missing the attribute in the error above?

I ran a 'find' on the entire disk and those were the only three files with 'awscrt' in the name, so it doesn't appear to be a version conflict.

In other words, it appears the problem is with the awscrt lib that's shipped in the zip.

[mattf10]

Today I upgraded to the latest version 2.15.0 and I can now run aws s3 commands as normal.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

[cash168]

I encountered the same problem when I didn’t notice that the old version of aws-cli had previously been unpacked into the aws directory
Run rm -rf aws before unzip -oq awscliv2.zip