Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix: Generate Signed Urls through a service account by providing service_account_email and access_token #1427

Open
wants to merge 17 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions docs/backends/gcloud.rst
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,23 @@ For development use cases, or other instances outside Google infrastructure:
#. Ensure the key is mounted/available to your running Django app.
#. Set an environment variable of GOOGLE_APPLICATION_CREDENTIALS to the path of the json file.

**Note Regarding Authentication**

There is currently a limitation in the GCS client for Python which by default requires a service account private key file to be
present when generating signed urls. **This is important to realize**. The service account private key is unavailable when running on a compute service.
Compute Services (App Engine, Cloud Run, Cloud Functions, Compute Engine...) fetch access tokens from a metadata service.
Those services do not have access to the service account private key. That means you must use one of the IAM sign functions (SignBlob, SignJwt)
to have Google sign using their managed private key. This means, you can't directly sign data. You must use the Cloud IAM API to do signing.

Luckily this can be worked around by passing `service_account_email` and `access_token` to the generate_signed_url function.
When both of those args are provided, generate_signed_url will use the IAM SignBlob API to sign the url and no private key file is needed.
yohannes15 marked this conversation as resolved.
Show resolved Hide resolved

Google also now recommends avoiding service account json key files as they are insecure, risky and hard to manage. This avoids the need for that
when developing locally.

`GS_SA_EMAIL` will be what is what provided to generate_signed_url param: service_account_email. Note, this service account will need credentials to
sign and download/upload files as necessary. Read more `here <https://cloud.google.com/storage/docs/access-control/signing-urls-with-helpers>`__

Alternatively, you can use the setting ``credentials`` or ``GS_CREDENTIALS`` as described below.


Expand Down Expand Up @@ -219,3 +236,12 @@ Settings
It supports `timedelta`, `datetime`, or `integer` seconds since epoch time.

Note: The maximum value for this option is 7 days (604800 seconds) in version `v4` (See this `Github issue <https://github.com/googleapis/python-storage/issues/456#issuecomment-856884993>`_)

``sa_email`` or ``GS_SA_EMAIL``

default: ``None``

This is the service account email to be used for signing the url. Signing urls requires a service account key file to be present in the env or IAM SignBlob/JWT API call
through a provided service account email. Compute services (App Engine, Cloud Run, Cloud Functions, Compute Engine...) for example don't have access to the key file in the env.
Providing, sa_email, will use the IAM API in order to sign the URL thus avoiding the need for a private service account json key file. If using `v4` of generate_signed_url,
`google-cloud-storage>=v1.36.1 <https://github.com/googleapis/python-storage/releases/tag/v1.36.1>`_ is required .
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ dropbox = [
"dropbox>=7.2.1",
]
google = [
"google-cloud-storage>=1.27",
"google-cloud-storage>=1.36.1",
]
libcloud = [
"apache-libcloud",
Expand Down
19 changes: 17 additions & 2 deletions storages/backends/gcloud.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,9 @@
from storages.utils import to_bytes

try:
from google import auth
from google.auth.transport import requests
from google.auth.credentials import TokenState
from google.cloud.exceptions import NotFound
from google.cloud.storage import Blob
from google.cloud.storage import Client
Expand Down Expand Up @@ -141,11 +144,20 @@ def get_default_settings(self):
# roll over.
"max_memory_size": setting("GS_MAX_MEMORY_SIZE", 0),
"blob_chunk_size": setting("GS_BLOB_CHUNK_SIZE"),
"sa_email": setting("GS_SA_EMAIL")
}

@property
def client(self):
if self._client is None:
if self.project_id is None or self.credentials is None:
self.credentials, self.project_id = auth.default(
scopes=['https://www.googleapis.com/auth/cloud-platform']
)
if not self.credentials.token_state == TokenState.FRESH:
self.credentials.refresh(requests.Request())
yohannes15 marked this conversation as resolved.
Show resolved Hide resolved
if not hasattr(self.credentials, "service_account_email") and self.sa_email:
self.credentials.service_account_email = self.sa_email
self._client = Client(project=self.project_id, credentials=self.credentials)
return self._client

Expand Down Expand Up @@ -323,12 +335,15 @@ def url(self, name, parameters=None):
quoted_name=_quote(name, safe=b"/~"),
)
else:
params = parameters or {}
default_params = {
"bucket_bound_hostname": self.custom_endpoint,
"expiration": self.expiration,
"version": "v4",
"version": "v4"
yohannes15 marked this conversation as resolved.
Show resolved Hide resolved
}
params = parameters or {}
if hasattr(self.credentials, "service_account_email"):
default_params["access_token"] = self.credentials.token
default_params["service_account_email"] = self.credentials.service_account_email

for key, value in default_params.items():
if value and key not in params:
Expand Down
Loading