Default credentials fail to detect running on GCP when ADC env var is empty #983

mbrancato · 2023-08-19T04:44:30Z

In the final layer of a container, it is pretty common to clear out env vars by setting them to empty. When GOOGLE_APPLICATION_CREDENTIALS exists but is empty (e.g. after pulling packages from Google Artifact Registry), the default credentials for the pubsub transport fails to use the assigned service account / metadata server.

Environment details

OS type and version: Linux 5.10.176+
Python version: Python 3.11.2
pip version: pip 23.0.1 from /usr/lib/python3/dist-packages/pip (python 3.11)
google-cloud-pubsub version: 2.18.3

Steps to reproduce

Set the application creds env var to an empty value
Attempt to create a publisher client while on GCP / GKE with a service account assigned

Code example

import asyncio
from google.pubsub_v1.services.publisher.async_client import PublisherAsyncClient
async def setup():
  publisher_client = PublisherAsyncClient(transport="grpc_asyncio")
  return publisher_client

loop = asyncio.new_event_loop()
loop.run_until_complete(setup())

Stack trace

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "<stdin>", line 2, in setup
  File "/usr/local/lib/python3.11/dist-packages/google/pubsub_v1/services/publisher/async_client.py", line 215, in __init__
    self._client = PublisherClient(
                   ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/google/pubsub_v1/services/publisher/client.py", line 492, in __init__
    self._transport = Transport(
                      ^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/google/pubsub_v1/services/publisher/transports/grpc_asyncio.py", line 198, in __init__
    super().__init__(
  File "/usr/local/lib/python3.11/dist-packages/google/pubsub_v1/services/publisher/transports/base.py", line 104, in __init__
    credentials, _ = google.auth.default(
                     ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/google/auth/_default.py", line 659, in default
    credentials, project_id = checker()
                              ^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/google/auth/_default.py", line 652, in <lambda>
    lambda: _get_explicit_environ_credentials(quota_project_id=quota_project_id),
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/google/auth/_default.py", line 272, in _get_explicit_environ_credentials
    credentials, project_id = load_credentials_from_file(
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/google/auth/_default.py", line 116, in load_credentials_from_file
    raise exceptions.DefaultCredentialsError(
google.auth.exceptions.DefaultCredentialsError: File  was not found.

The text was updated successfully, but these errors were encountered:

liuyunnnn · 2023-08-22T02:54:41Z

Hi @mbrancato,

Could you describe how you attached service account? This doc provides a few ways: https://cloud.google.com/docs/authentication/provide-credentials-adc#containerized

For example, if you use work identity for GKE and network policy is enabled, this troubleshooting page might be helpful: https://cloud.google.com/knowledge/kb/defaultcredentialserror-with-workload-identity-000004712.

liuyunnnn · 2023-08-22T03:57:01Z

Also wondering what does GOOGLE_APPLICATION_CREDENTIALS exists but is empty means exactly. Set to the path of invalid service account file or None? From the stack trace, it's very likely that your var is not None so it errors out here

mbrancato · 2023-08-22T11:49:50Z

Yes, we're using GKE workload identity. Everything works fine if I unset the GOOGLE_APPLICATION_CREDENTIALS env var. Inside GKE, it acts like a normal user-assigned service account and can access the service account endpoints in the metadata api.

You can create an empty env var by doing this:

export GOOGLE_APPLICATION_CREDENTIALS=""

The Dockerfile spec does not have a way to "unset" an env var, only clear its value from the layer making it empty. I was setting the ADC env var to a file path, then later clearing it. This left the env var set in the environment, but empty. Manually running unset GOOGLE_APPLICATION_CREDENTIALS fixes the issue. I've worked around this by not "exporting" the env var, and just passing it on the command line where needed (since its just a file path anyway).

I ran the following in a temporary python container to illustrate the issue. All commands were performed after running gcloud auth login --update-adc. The example.py script contains the example in my original post.

root@9fb1332e5023:~# gcloud auth list
    Credentialed Accounts
ACTIVE  ACCOUNT
*       m*********************

To set the active account, run:
    $ gcloud config set account `ACCOUNT`

root@9fb1332e5023:~# env
HOSTNAME=9fb1332e5023
PYTHON_VERSION=3.11.4
PWD=/root
PYTHON_SETUPTOOLS_VERSION=65.5.1
...
root@9fb1332e5023:~# python example.py 
root@9fb1332e5023:~# export GOOGLE_APPLICATION_CREDENTIALS=""
root@9fb1332e5023:~# env
HOSTNAME=9fb1332e5023
PYTHON_VERSION=3.11.4
PWD=/root
GOOGLE_APPLICATION_CREDENTIALS=
PYTHON_SETUPTOOLS_VERSION=65.5.1
...
root@9fb1332e5023:~# python example.py 
Traceback (most recent call last):
  File "/root/example.py", line 8, in <module>
    loop.run_until_complete(setup())
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/root/example.py", line 4, in setup
    publisher_client = PublisherAsyncClient(transport="grpc_asyncio")
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/google/pubsub_v1/services/publisher/async_client.py", line 215, in __init__
    self._client = PublisherClient(
                   ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/google/pubsub_v1/services/publisher/client.py", line 492, in __init__
    self._transport = Transport(
                      ^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/google/pubsub_v1/services/publisher/transports/grpc_asyncio.py", line 198, in __init__
    super().__init__(
  File "/usr/local/lib/python3.11/site-packages/google/pubsub_v1/services/publisher/transports/base.py", line 104, in __init__
    credentials, _ = google.auth.default(
                     ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/google/auth/_default.py", line 659, in default
    credentials, project_id = checker()
                              ^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/google/auth/_default.py", line 652, in <lambda>
    lambda: _get_explicit_environ_credentials(quota_project_id=quota_project_id),
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/google/auth/_default.py", line 272, in _get_explicit_environ_credentials
    credentials, project_id = load_credentials_from_file(
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/google/auth/_default.py", line 116, in load_credentials_from_file
    raise exceptions.DefaultCredentialsError(
google.auth.exceptions.DefaultCredentialsError: File  was not found.

liuyunnnn · 2023-08-22T15:10:42Z

I'm not an expert on Dockerfile, but when you export var="", and read var in python by var = os.environ['var'], var is not empty (var is not None) but an empty string. That's the reason your script ended up here and threw error here.

I'm going to close this as this is not an issue for pubsub python client library.

mbrancato · 2023-08-22T16:15:08Z

Wow ok. I'm saying this is a behavioral bug in how the pubsub auth process determines what creds to use. If the ADC env var exists but is empty, the auth should ignore it as if it does not exist.

I'm not sure why this is so confusing. I'm not taking about a programming language's interpretation of an env var, all env vars are strings.

liuyunnnn · 2023-08-22T16:44:25Z

Well the authentication is actually taken care of by google-auth(https://github.com/googleapis/google-auth-library-python). You can open an issue there if you think it is a bug.

(Oh you already did this so nvm :))

product-auto-label bot added the api: pubsub Issues related to the googleapis/python-pubsub API. label Aug 19, 2023

liuyunnnn self-assigned this Aug 21, 2023

liuyunnnn closed this as completed Aug 22, 2023

mbrancato mentioned this issue Aug 22, 2023

fix: detect correct auth when ADC env var is set but empty googleapis/google-auth-library-python#1374

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Default credentials fail to detect running on GCP when ADC env var is empty #983

Default credentials fail to detect running on GCP when ADC env var is empty #983

mbrancato commented Aug 19, 2023

liuyunnnn commented Aug 22, 2023

liuyunnnn commented Aug 22, 2023

mbrancato commented Aug 22, 2023

liuyunnnn commented Aug 22, 2023

mbrancato commented Aug 22, 2023

liuyunnnn commented Aug 22, 2023 •

edited

Loading

Default credentials fail to detect running on GCP when ADC env var is empty #983

Default credentials fail to detect running on GCP when ADC env var is empty #983

Comments

mbrancato commented Aug 19, 2023

Environment details

Steps to reproduce

Code example

Stack trace

liuyunnnn commented Aug 22, 2023

liuyunnnn commented Aug 22, 2023

mbrancato commented Aug 22, 2023

liuyunnnn commented Aug 22, 2023

mbrancato commented Aug 22, 2023

liuyunnnn commented Aug 22, 2023 • edited Loading

liuyunnnn commented Aug 22, 2023 •

edited

Loading