Nightly build failure 2024-01-26

[jdangerx]

Overview

What do you think happened, if anything?

Seems like a Zenodo upload failure again.

Next steps

What next steps do we need to do to understand or remediate the issue?

It looks like an actual Zenodo file upload failed, instead of just the publication - we should check the actual Zenodo draft and see if everything made it. If not, we should:

Tasks

Beta Give feedback

1. add retry logic to zenodo_data_release.py
2. add fsspec logic to zenodo_data_release.py
3. tell zenodo_data_release.py to pull from the nightly build bucket

Verify that everything is fixed!

Tasks

Beta Give feedback

1. validation tests passed
2. S3 distribution bucket was updated at the expected time
3. GCP distribution bucket was updated at the expected time
4. GCP internal bucket was updated at the expected time
5. Datasette PUDL version points at the same hash as nightly
6. Zenodo sandbox record was updated to the record number in the logs (search for zenodo_data_release.py and Using metadata from in the logs, to see what the new record number should be!)

Relevant logs

link to build logs from internal distribution bucket

2024-01-26 12:46:25,321: INFO - Uploading file to https://sandbox.zenodo.org/api/files/e728d1ee-73e4-4364-af3b-b5e97ca21663/ferc2_dbf.sqlite.gz (zenodo_data_release.py:162)
Traceback (most recent call last):
  File "/home/mambauser/env/lib/python3.11/site-packages/urllib3/connectionpool.py", line 467, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/home/mambauser/env/lib/python3.11/site-packages/urllib3/connectionpool.py", line 462, in _make_request
    httplib_response = conn.getresponse()
                       ^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.11/http/client.py", line 1386, in getresponse
    response.begin()
  File "/home/mambauser/env/lib/python3.11/http/client.py", line 325, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.11/http/client.py", line 286, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.11/socket.py", line 706, in readinto
    return self._sock.recv_into(b)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.11/ssl.py", line 1315, in recv_into
    return self.read(nbytes, buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.11/ssl.py", line 1167, in read
    return self._sslobj.read(len, buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TimeoutError: The read operation timed out