ignition download error with Access Denied when using s3:// and arn schema on AWS

[HuijingHei]

Describe the bug

Start vm on aws using fedora-coreos-40.20240709.2.0-x86_64, download ignition config from the bucket using arn {"ignition":{"config":{"replace":{"source":"arn:aws:s3:::hhei-test/ssh.ign"}},"version":"3.4.0"}} or s3, failed with "AccessDenied: Access Denied"

Reproduction steps

1. Create vm using fedora-coreos-40.20240709.2.0-x86_64
2. Add user data with {"ignition":{"config":{"replace":{"source":"arn:aws:s3:::hhei-test/ssh.ign"}},"version":"3.4.0"}}

Expected behavior

Failed to boot with failed to fetch config: AccessDenied: Access Denied

Actual behavior

failed logs:

:/root# journalctl -u ignition-fetch | cat
Jul 30 13:13:44 ip-10-0-2-27 ignition[748]: Ignition 2.19.0
Jul 30 13:13:44 ip-10-0-2-27 ignition[748]: Stage: fetch
Jul 30 13:13:44 ip-10-0-2-27 ignition[748]: reading system config file "/usr/lib/ignition/base.d/00-core.ign"
Jul 30 13:13:44 ip-10-0-2-27 ignition[748]: parsing config with SHA512: ff6a5153be363997e4d5d3ea8cc4048373a457c48c4a5b134a08a30aacd167c1e0f099f0bdf1e24c99ad180628cd02b767b863b5fe3a8fce3fe1886847eb8e2e
Jul 30 13:13:44 ip-10-0-2-27 ignition[748]: reading system config file "/usr/lib/ignition/base.d/30-afterburn-sshkeys-core.ign"
Jul 30 13:13:44 ip-10-0-2-27 ignition[748]: parsing config with SHA512: a30a1921169d5a3b58ef9b25de60783be1add6ea8d05fd44a0746cb60dd1b8a8b34ab51eec5eb14eecc2df2ab6ba1cd3fd7351eed65793d22316ab262a857d95
Jul 30 13:13:44 ip-10-0-2-27 ignition[748]: reading system config file "/usr/lib/ignition/base.platform.d/aws/20-aws-nm-cloud-setup.ign"
Jul 30 13:13:44 ip-10-0-2-27 ignition[748]: parsing config with SHA512: ecbe8e22f0d809d43786977d41f937a67a0d6a5ecbd7e3e40385e57daacd3a973cce677aa7c8fc58bd99d85c92d730745b96895ab8ca86d038c89f2c278b82cd
Jul 30 13:13:44 ip-10-0-2-27 ignition[748]: PUT http://169.254.169.254/latest/api/token: attempt #1
Jul 30 13:13:44 ip-10-0-2-27 ignition[748]: PUT result: OK
Jul 30 13:13:44 ip-10-0-2-27 ignition[748]: parsed url from cmdline: ""
Jul 30 13:13:44 ip-10-0-2-27 ignition[748]: no config URL provided
Jul 30 13:13:44 ip-10-0-2-27 ignition[748]: reading system config file "/usr/lib/ignition/user.ign"
Jul 30 13:13:44 ip-10-0-2-27 ignition[748]: no config at "/usr/lib/ignition/user.ign"
Jul 30 13:13:44 ip-10-0-2-27 ignition[748]: PUT http://169.254.169.254/latest/api/token: attempt #1
Jul 30 13:13:44 ip-10-0-2-27 ignition[748]: PUT result: OK
Jul 30 13:13:44 ip-10-0-2-27 ignition[748]: GET http://169.254.169.254/2019-10-01/user-data: attempt #1
Jul 30 13:13:44 ip-10-0-2-27 ignition[748]: GET result: OK
Jul 30 13:13:44 ip-10-0-2-27 ignition[748]: parsing config with SHA512: c3a668f4637f9685bf8bbe4aa81a96a36dc3a0dc222169a4550e79401014a330c1110c2054cd442fd8c9036c2d6ed9419d410de08e44095af0f118e47615370f
Jul 30 13:13:44 ip-10-0-2-27 ignition[748]: failed to fetch config: AccessDenied: Access Denied
                                                    status code: 403, request id: YD5P04RQ6JEY5VH3, host id: uT43Lls93HlL7LWkqNnf2v4ecSCs1V/bhNVORIOFXxf1DoBKhD+WqPgkWMbRBLtpdOQvGrL8Rls=
Jul 30 13:13:44 ip-10-0-2-27 ignition[748]: failed to acquire config: AccessDenied: Access Denied
                                                    status code: 403, request id: YD5P04RQ6JEY5VH3, host id: uT43Lls93HlL7LWkqNnf2v4ecSCs1V/bhNVORIOFXxf1DoBKhD+WqPgkWMbRBLtpdOQvGrL8Rls=
Jul 30 13:13:44 ip-10-0-2-27 ignition[748]: Ignition failed: AccessDenied: Access Denied
                                                    status code: 403, request id: YD5P04RQ6JEY5VH3, host id: uT43Lls93HlL7LWkqNnf2v4ecSCs1V/bhNVORIOFXxf1DoBKhD+WqPgkWMbRBLtpdOQvGrL8Rls=
Jul 30 13:13:43 ip-10-0-2-27 systemd[1]: Starting ignition-fetch.service - Ignition (fetch)...
Jul 30 13:13:44 ip-10-0-2-27 (ignition)[748]: ignition-fetch.service: Referenced but unset environment variable evaluates to an empty string: IGNITION_ARGS
Jul 30 13:13:44 ip-10-0-2-27 systemd[1]: ignition-fetch.service: Main process exited, code=exited, status=1/FAILURE
Jul 30 13:13:44 ip-10-0-2-27 systemd[1]: ignition-fetch.service: Failed with result 'exit-code'.
Jul 30 13:13:44 ip-10-0-2-27 systemd[1]: Failed to start ignition-fetch.service - Ignition (fetch).
Jul 30 13:13:45 ip-10-0-2-27 systemd[1]: ignition-fetch.service: Triggering OnFailure= dependencies.

System details

FCOS version: fedora-coreos-40.20240709.2.0-x86_64

Butane or Ignition config

No response

Additional information

Related issue https://issues.redhat.com/browse/OCPBUGS-31525

[travier]

We document S3 setup in https://docs.fedoraproject.org/en-US/fedora-coreos/provisioning-aws/#_remote_ignition_configuration. Can you give this a try?

If the "plain" S3 support works then it's something specific to the arn logic: https://docs.aws.amazon.com/fr_fr/IAM/latest/UserGuide/reference-arns.html

[travier]

https://github.com/coreos/ignition/blob/main/internal/resource/url.go#L161

[HuijingHei]

We document S3 setup in https://docs.fedoraproject.org/en-US/fedora-coreos/provisioning-aws/#_remote_ignition_configuration. Can you give this a try?

Test with S3 url s3://hhei-test/ssh.ign ( using 40.20240728.2.1), vm failed to boot with fetch config: AccessDenied: Access Denied, same logs as above. And the presigned url works.

[HuijingHei]

We document S3 setup in https://docs.fedoraproject.org/en-US/fedora-coreos/provisioning-aws/#_remote_ignition_configuration. Can you give this a try?

Test with S3 url s3://hhei-test/ssh.ign ( using 40.20240728.2.1), vm failed to boot with fetch config: AccessDenied: Access Denied, same logs as above. And the presigned url works.

The root cause is the VM missing IAM instance profile with s3:GetObject permission, after add related role, it works. Refer to the iam roles in bootstrap https://github.com/openshift/installer/blob/master/upi/aws/cloudformation/04_cluster_bootstrap.yaml#L107-L136

[travier]

Thanks! Can you update the docs?