Nightly Build Failure 2024-01-24

[jdangerx]

Relevant logs

2024-01-24 13:29:59,338: INFO - Using metadata from 23602 to publish 23676... (zenodo_data_release.py:303)
Traceback (most recent call last):
  File "/home/mambauser/env/lib/python3.11/site-packages/urllib3/connectionpool.py", line 467, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/home/mambauser/env/lib/python3.11/site-packages/urllib3/connectionpool.py", line 462, in _make_request
    httplib_response = conn.getresponse()
                       ^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.11/http/client.py", line 1386, in getresponse
    response.begin()
  File "/home/mambauser/env/lib/python3.11/http/client.py", line 325, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.11/http/client.py", line 286, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.11/socket.py", line 706, in readinto
    return self._sock.recv_into(b)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.11/ssl.py", line 1315, in recv_into
    return self.read(nbytes, buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.11/ssl.py", line 1167, in read
    return self._sslobj.read(len, buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TimeoutError: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/mambauser/env/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.11/site-packages/urllib3/connectionpool.py", line 799, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.11/site-packages/urllib3/util/retry.py", line 550, in increment
    raise six.reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.11/site-packages/urllib3/packages/six.py", line 770, in reraise
    raise value
  File "/home/mambauser/env/lib/python3.11/site-packages/urllib3/connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.11/site-packages/urllib3/connectionpool.py", line 469, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/home/mambauser/env/lib/python3.11/site-packages/urllib3/connectionpool.py", line 358, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='sandbox.zenodo.org', port=443): Read timed out. (read timeout=5)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/mambauser/pudl/devtools/zenodo/zenodo_data_release.py", line 372, in <module>
    pudl_zenodo_data_release()
  File "/home/mambauser/env/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/pudl/devtools/zenodo/zenodo_data_release.py", line 365, in pudl_zenodo_data_release
    completed_draft.publish()
  File "/home/mambauser/pudl/devtools/zenodo/zenodo_data_release.py", line 317, in publish
    self.zenodo_client.publish_deposition(self.record_id)
  File "/home/mambauser/pudl/devtools/zenodo/zenodo_data_release.py", line 176, in publish_deposition
    response = requests.post(
               ^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.11/site-packages/requests/api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.11/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.11/site-packages/requests/adapters.py", line 532, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='sandbox.zenodo.org', port=443): Read timed out. (read timeout=5)

Next steps

Looks like a Zenodo timeout. Potentially this draft has already been published, in which case - great.

We should verify that the distribution/internal buckets are up-to-date and that fly.io did indeed update our datasette instance:

TODO

Beta Give feedback

1. check that pudl.core_pudl__codes_datasources.pudl_version points at the same hash as nightly
2. check that s3 and gcp buckets were updated

[jdangerx]

closed as dupe of #3284