Code relying on AWS IMDSv1 blocking cloud env hardening 😓

[laluka]

Is your feature request related to a problem? Please describe.

AWS IMDSv2 (metadata url - 169.254.169.254) seems to be supported in google/auth/aws.py but some files like google/auth/compute_engine/_metadata.py still rely on IMDSv1, blocking aws cloud hardening.

V1 Legacy code:

google-auth-library-python/google/auth/compute_engine/_metadata.py

Lines 49 to 54 in 9c87ad0

	_METADATA_IP_ROOT = "http://{}".format(
	os.getenv(environment_vars.GCE_METADATA_IP, "169.254.169.254")
	)
	_METADATA_FLAVOR_HEADER = "metadata-flavor"
	_METADATA_FLAVOR_VALUE = "Google"
	_METADATA_HEADERS = {_METADATA_FLAVOR_HEADER: _METADATA_FLAVOR_VALUE}

google-auth-library-python/google/auth/compute_engine/_metadata.py

Lines 153 to 241 in 9c87ad0

	def get(
	request,
	path,
	root=_METADATA_ROOT,
	params=None,
	recursive=False,
	retry_count=5,
	headers=None,
	):
	"""Fetch a resource from the metadata server.
	Args:
	request (google.auth.transport.Request): A callable used to make
	HTTP requests.
	path (str): The resource to retrieve. For example,
	``'instance/service-accounts/default'``.
	root (str): The full path to the metadata server root.
	params (Optional[Mapping[str, str]]): A mapping of query parameter
	keys to values.
	recursive (bool): Whether to do a recursive query of metadata. See
	https://cloud.google.com/compute/docs/metadata#aggcontents for more
	details.
	retry_count (int): How many times to attempt connecting to metadata
	server using above timeout.
	headers (Optional[Mapping[str, str]]): Headers for the request.
	Returns:
	Union[Mapping, str]: If the metadata server returns JSON, a mapping of
	the decoded JSON is return. Otherwise, the response content is
	returned as a string.
	Raises:
	google.auth.exceptions.TransportError: if an error occurred while
	retrieving metadata.
	"""
	base_url = urlparse.urljoin(root, path)
	query_params = {} if params is None else params
	headers_to_use = _METADATA_HEADERS.copy()
	if headers:
	headers_to_use.update(headers)
	if recursive:
	query_params["recursive"] = "true"
	url = _helpers.update_query(base_url, query_params)
	retries = 0
	while retries < retry_count:
	try:
	response = request(url=url, method="GET", headers=headers_to_use)
	break
	except exceptions.TransportError as e:
	_LOGGER.warning(
	"Compute Engine Metadata server unavailable on "
	"attempt %s of %s. Reason: %s",
	retries + 1,
	retry_count,
	e,
	)
	retries += 1
	else:
	raise exceptions.TransportError(
	"Failed to retrieve {} from the Google Compute Engine "
	"metadata service. Compute Engine Metadata server unavailable".format(url)
	)
	if response.status == http_client.OK:
	content = _helpers.from_bytes(response.data)
	if response.headers["content-type"] == "application/json":
	try:
	return json.loads(content)
	except ValueError as caught_exc:
	new_exc = exceptions.TransportError(
	"Received invalid JSON from the Google Compute Engine "
	"metadata service: {:.20}".format(content)
	)
	six.raise_from(new_exc, caught_exc)
	else:
	return content
	else:
	raise exceptions.TransportError(
	"Failed to retrieve {} from the Google Compute Engine "
	"metadata service. Status: {} Response:\n{}".format(
	url, response.status, response.data
	),
	response,
	)

V2 Supported code:

google-auth-library-python/google/auth/aws.py

Lines 418 to 470 in 9c87ad0

	def retrieve_subject_token(self, request):
	"""Retrieves the subject token using the credential_source object.
	The subject token is a serialized `AWS GetCallerIdentity signed request`_.
	The logic is summarized as:
	Retrieve the AWS region from the AWS_REGION or AWS_DEFAULT_REGION
	environment variable or from the AWS metadata server availability-zone
	if not found in the environment variable.
	Check AWS credentials in environment variables. If not found, retrieve
	from the AWS metadata server security-credentials endpoint.
	When retrieving AWS credentials from the metadata server
	security-credentials endpoint, the AWS role needs to be determined by
	calling the security-credentials endpoint without any argument. Then the
	credentials can be retrieved via: security-credentials/role_name
	Generate the signed request to AWS STS GetCallerIdentity action.
	Inject x-goog-cloud-target-resource into header and serialize the
	signed request. This will be the subject-token to pass to GCP STS.
	.. _AWS GetCallerIdentity signed request:
	https://cloud.google.com/iam/docs/access-resources-aws#exchange-token
	Args:
	request (google.auth.transport.Request): A callable used to make
	HTTP requests.
	Returns:
	str: The retrieved subject token.
	"""
	# Fetch the session token required to make meta data endpoint calls to aws.
	if (
	request is not None
	and self._imdsv2_session_token_url is not None
	and self._should_use_metadata_server()
	):
	headers = {"X-aws-ec2-metadata-token-ttl-seconds": "300"}
	imdsv2_session_token_response = request(
	url=self._imdsv2_session_token_url, method="PUT", headers=headers
	)
	if imdsv2_session_token_response.status != 200:
	raise exceptions.RefreshError(
	"Unable to retrieve AWS Session Token",
	imdsv2_session_token_response.data,
	)
	imdsv2_session_token = imdsv2_session_token_response.data
	else:
	imdsv2_session_token = None

Describe the solution you'd like

A full support of IMDSv2 as default metadata fetching service, follwing: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-metadata-v2-how-it-works.html

[clundin25]

Cursory look:

The code in google-auth-library-python/google/auth/compute_engine/_metadata.py is for supporting Google Compute Engine.

The code in google-auth-library-python/google/auth/aws.py is for AWS metadata servers.

IMDSv2 is an implementation detail for the AWS metadata server. With this information, does your question remain? Disclaimer: I did not refresh myself on the code, and I may be mistaken.

[laluka]

Hmmmm, that totally makes sense, but then I wonder what's happening on our side, might be a default behavior or misconfig, I'll keep you posted!
Thank you for the fast answer! 🫶

[laluka]

Solved, it definitely was a a misunderstanding + misconfig on our side, thanks so much!

[zchenyu]

I ran into this as well. I posted my solution here: #1364 (comment)