ec2: Support double encoded userdata #4276
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There was a problem hiding this comment.
Changes here LGTM! Thanks for the contribution.
I haven't added my name to tools/.github-cla-signers, but can do so if you need it
Yes please. We'll need that along with fixes to the failing lint job in order for this PR to pass CI. Once those two things happen, we can merge this.
added 2 commits
August 4, 2023 07:13
This function can be used elsewhere and is not specific to Hetzner Signed-off-by: Noah Meyerhans <[email protected]>
Experience has shown that there are a number of tools and common patterns that effectively doubly base64 encode userdata. This change replaces a patch previously carried by Amazon Linux that was incorrectly ported to cloud-init 22.2.2 in Amazon Linux 2023 add addresses amazonlinux/amazon-linux-2023#401 Signed-off-by: Noah Meyerhans <[email protected]>
nmeyerhans
force-pushed
the
ec2-double-encoded-userdata
branch
from
313243f
to
bb72e8a
Compare
Thanks, updated. |
nmeyerhans
pushed a commit
to nmeyerhans/cloud-init
that referenced
this pull request
Aug 24, 2023
canonical#4276 uncovered an issue with the initialization of the return value for get_instance_userdata(). The return value was initialized with user_data = "" which is a str class. It then calls url_helper.read_file_or_url(), which attempts to retrieve user-data content from IMDS. read_file_or_url() returns its results as a bytes object, which is then passed directly up to the caller. In the event that read_file_or_url() does not successfully retrieve content (e.g. because it was given a file:// path to a nonexistent file or an http:// path that generates a 404 code), an exception is raised an get_instance_userdata returns the string object initially stored in user_data. Rather than make the caller cope with return data potentially encoded as either bytes or str, this commit changes the initialization of user_data to an empty bytes object, ensuring type consistency in get_instance_userdata()'s return value. Fixes canonical#4386 Signed-off-by: Noah Meyerhans <[email protected]>
nmeyerhans
pushed a commit
to nmeyerhans/cloud-init
that referenced
this pull request
Aug 24, 2023
canonical#4276 uncovered an issue with the initialization of the return value for get_instance_userdata(). The return value was initialized with user_data = "" which is a str class. It then calls url_helper.read_file_or_url(), which attempts to retrieve user-data content from IMDS. read_file_or_url() returns its results as a bytes object, which is then passed directly up to the caller. In the event that read_file_or_url() does not successfully retrieve content (e.g. because it was given a file:// path to a nonexistent file or an http:// path that generates a 404 code), an exception is raised an get_instance_userdata returns the string object initially stored in user_data. Rather than make the caller cope with return data potentially encoded as either bytes or str, this commit changes the initialization of user_data to an empty bytes object, ensuring type consistency in get_instance_userdata()'s return value. Fixes canonical#4386 Signed-off-by: Noah Meyerhans <[email protected]>
3 tasks
holmanb
pushed a commit
that referenced
this pull request
Aug 24, 2023
#4276 uncovered an issue with the initialization of the return value for get_instance_userdata(). The return value was initialized with user_data = "" which is a str class. It then calls url_helper.read_file_or_url(), which attempts to retrieve user-data content from IMDS. read_file_or_url() returns its results as a bytes object, which is then passed directly up to the caller. In the event that read_file_or_url() does not successfully retrieve content (e.g. because it was given a file:// path to a nonexistent file or an http:// path that generates a 404 code), an exception is raised an get_instance_userdata returns the string object initially stored in user_data. Rather than make the caller cope with return data potentially encoded as either bytes or str, this commit changes the initialization of user_data to an empty bytes object, ensuring type consistency in get_instance_userdata()'s return value. Fixes GH-4386 Signed-off-by: Noah Meyerhans <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed Commit Message
Additional Context
Amazon Linux has long carried a patch to cloud-init to handle this case of double base64 encoding. This patch broke when it was forward ported to python3 and a more recent cloud-init during the Amazon Linux 2023 development. By submitting the fixed version here, we hope to remove this custom patch and also make this functionality work for users of other distros.
Test Steps
Start with any valid cloud-config. This can be passed to the AWS CLI, which will take care of base64 encoding. This works as expected.
Base64 encode the userdata, and pass the encoded results to the AWS CLI, which will re-encode it with a second layer of base64 encoding. Example:
Launch an instance with the base64 encoded data. Without this change, cloud-init fails to process userdata and records a message similar to the following in
cloud-init.log:With this change, the userdata is decoded and executed as intended.
Tested on Amazon Linux 2023 and Debian 12 and unstable.
Checklist:
Note that AWS has already signed the CLA on behalf of employees, which should cover this contribution. I haven't added my name to
tools/.github-cla-signers, but can do so if you need it.