Twine upload fails with large file, SSLEOFError, 'EOF occurred in violation of protocol' and ConnectionError: RemoteDisconnected

[akkana]

Problem description

I'm trying to make a test release of my PyTopo project (current git:
https://github.com/akkana/pytopo/tree/05262ed7484bd37efb7aaa4e76c401f523bd1814) but for months, I've been unable to. I try to upload with:
python -m twine upload --repository testpypi dist/*
and first it fails and retries:

WARNING  Retrying (Retry(total=9, connect=5, read=None, redirect=None,          
         status=None)) after connection broken by 'SSLEOFError(8, 'EOF occurred 
         in violation of protocol (_ssl.c:2406)')': /legacy/            

and then it goes to 100% but then sits there for 5-10 minutes with no output before finally failing with
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
(I'll attach the full error log).

I worked with SnoopJ on #python for a long time today, trying a Python-3.12.6 source build to make sure it wasn't a problem with Debian's 3.12.6, and a Python-3.11.2 build because that was the last build before a patch related to OP_IGNORE_UNEXPECTED_EOF went in, but always got the same result.

I wondered if it had to do with the size of the upload. I had one large data file (test/files/Surface_Ownership_10_14_2015.geojson, 36M) in the tarball, so I tried removing it (it's only used for unit tests) and the error went away.

I've had that file in the source tarball for several years, and it used to work fine (I've released several versions of the package during that time), so something has apparently changed that has made testpypi and/or twine less tolerant of large uploads, and maybe of OpenSSL 3.x protocol errors. And the error message gives no hint of what the real problem is.

This may be related to the following other issues:
pypa/twine#273
#696
python/cpython#87960

Full error log here;
twineerrs.txt

[SnoopJ]

a patch related to OP_IGNORE_UNEXPECTED_EOF went in

To expand on the OpenSSL parts of this a bit:

OpenSSL changed how they respond to an unexpected EOF between 1.1.1 and 3.x (the observed error is from OpenSSL 3.3.2 3 Sep 2024 specifically), the behavior used to be non-fatal and now it is fatal. See this discussion on the OpenSSL repo: openssl/openssl#22690 (comment)

I suspect this means that Test PyPI is running a TLS implementation that has some gear-grinding with OpenSSL 3.x. That is, it sometimes sends EOF without first sending close_notify. I'm not sure how the large file interacts with that.

I suggested that @akkana try a build of 3.11.2 because of an upstream change that was present in 3.11.2 but removed in 3.11.3 that enabled the SSL_OP_IGNORE_UNEXPECTED_EOF option, which the OpenSSL discussion seems to suggest would work around the new behavior. Debian sid (the distro at play here) switched from Python 3.11.2 to Python 3.11.4 on 25 June 2023, and she said on IRC that this lines up with the timeline when she started seeing the problem.

We confirmed that her build of 3.11.2 did set this flag in the default SSLContext, but it didn't seem to have any effect on the behavior of twine. Possibly some interaction between twine, requests, urllib3 means that the SSL context does not include this option even when the underlying Python's default context has it.

I think at this point someone from PyPA who knows their TLS better than we do should have a look.

[janden]

I'm observing the same issue with our package cufinufft https://github.com/flatironinstitute/cufinufft. I've tried this with two configurations:

• Python 3.10.12 (Ubuntu) and twine version 6.0.1 (keyring: 23.5.0, pkginfo: 1.8.3, requests: 2.25.1, requests-toolbelt: 0.9.1, urllib3: 1.26.5)
• Python 3.12.1 (compiled from source) and twine version 6.0.1 (keyring: 25.5.0, pkginfo: 1.12.0, requests: 2.32.3, requests-toolbelt: 1.0.0, urllib3: 2.2.3)

Both give me the following error

$ twine upload --verbose *
Uploading distributions to https://upload.pypi.org/legacy/
INFO     cufinufft-2.3.1-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (159.1 MB)                                               
INFO     finufft-2.3.1-py3-none-macosx_13_0_arm64.whl (2.4 MB)                                                                            
INFO     finufft-2.3.1-py3-none-macosx_13_0_x86_64.whl (4.0 MB)                                                                           
INFO     finufft-2.3.1-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.0 MB)                                                   
INFO     finufft-2.3.1-py3-none-win_amd64.whl (5.0 MB)                                                                                    
INFO     cufinufft-2.3.1.tar.gz (3.7 MB)                                                                                                  
INFO     finufft-2.3.1.tar.gz (3.7 MB)                                                                                                    
INFO     username set by command options                                                                                                  
INFO     Querying keyring for password                                                                                                    
Enter your API token: 
INFO     username: __token__                                                                                                              
INFO     password: <hidden>                                                                                                               
Uploading cufinufft-2.3.1-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
WARNING  Retrying (Retry(total=9, connect=5, read=None, redirect=None, status=None)) after connection broken by 'SSLEOFError(8, 'EOF      
         occurred in violation of protocol (_ssl.c:2406)')': /legacy/                                                                     
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 166.8/166.8 MB • 04:35 • 502.8 kB/s
Traceback (most recent call last):
  File "/home/janden/venv/twine312/lib/python3.12/site-packages/urllib3/connectionpool.py", line 789, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/home/janden/venv/twine312/lib/python3.12/site-packages/urllib3/connectionpool.py", line 536, in _make_request
    response = conn.getresponse()
               ^^^^^^^^^^^^^^^^^^
  File "/home/janden/venv/twine312/lib/python3.12/site-packages/urllib3/connection.py", line 507, in getresponse
    httplib_response = super().getresponse()
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/lib/python3.12/http/client.py", line 1419, in getresponse
    response.begin()
  File "/opt/python/lib/python3.12/http/client.py", line 331, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "/opt/python/lib/python3.12/http/client.py", line 300, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

Full error log attached.

Note that this is also quite a large file, which may be triggering the same issue. However we didn't have this issue when uploading our last release (2024-09-10).