Skip to content

Commit

Permalink
[gcloud] use a different checksum method to fix downloading in FIPS m…
Browse files Browse the repository at this point in the history
…ode (#1473)

Python 3.10 and later versions rely on OpenSSL 1.1.1 or newer, which includes FIPS-compliance checks.

MD5 is not an approved algorithm in FIPS mode, so attempting to instantiate self.blob.download_to_file(self._file) will fail when the system is running in FIPS mode.

The change configures the `download_to_file` function to use an alternative algorithm provided by gcloud storage SDK - 'crc32c' - for checksum calculation.
Configurable checksumming is available in the google-storage lib since v1.31.0, but pinning to >=1.32 for the retry import.

Co-authored-by: markesha <>
  • Loading branch information
markesha authored Nov 28, 2024
1 parent fda4e23 commit b3513ec
Show file tree
Hide file tree
Showing 3 changed files with 4 additions and 4 deletions.
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ dropbox = [
"dropbox>=7.2.1",
]
google = [
"google-cloud-storage>=1.27",
"google-cloud-storage>=1.32",
]
libcloud = [
"apache-libcloud",
Expand Down
2 changes: 1 addition & 1 deletion storages/backends/gcloud.py
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ def _get_file(self):
)
if "r" in self._mode:
self._is_dirty = False
self.blob.download_to_file(self._file)
self.blob.download_to_file(self._file, checksum="crc32c")
self._file.seek(0)
if self._storage.gzip and self.blob.content_encoding == "gzip":
self._file = self._decompress_file(mode=self._mode, file=self._file)
Expand Down
4 changes: 2 additions & 2 deletions tests/test_gcloud.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ def test_open_read(self):
self.filename, chunk_size=None
)

f.blob.download_to_file = lambda tmpfile: tmpfile.write(data)
f.blob.download_to_file = lambda tmpfile, **kwargs: tmpfile.write(data)
self.assertEqual(f.read(), data)

def test_open_read_num_bytes(self):
Expand All @@ -55,7 +55,7 @@ def test_open_read_num_bytes(self):
self.filename, chunk_size=None
)

f.blob.download_to_file = lambda tmpfile: tmpfile.write(data)
f.blob.download_to_file = lambda tmpfile, **kwargs: tmpfile.write(data)
self.assertEqual(f.read(num_bytes), data[0:num_bytes])

def test_open_read_nonexistent(self):
Expand Down

0 comments on commit b3513ec

Please sign in to comment.