Pip with Python > 3.8.17 fails to download large-ish packages

[byt3bl33d3r]

Description

PIp seems to not be able to download large packages (200Mb+). I can consistently reproduce this in Python 3.10 using the official python docker image. There is no issue with my network or my docker installation as downloading the wheel directly with curl or wget works fine (from both the docker container and my host OS).

Pip with Python > 3.10 gives an ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. even tho it's clear by the download progress bar that the package never fully downloaded.

In Python < 3.10, it does actually throw a pip._vendor.urllib3.exceptions.ProtocolError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))

Pip with older versions of Python seem to have a greater chance of success of successfully installing large packages. Pip with Python 3.8 seems fine most times. Pip with Python 3.9 fails occasionally and throws the connection reset error as shown above.

Update: Python 3.8.17 + pip 23.0.1 seem to work just fine. Any version of Python > 3.8.17 seems to have this issue with large packages.

This seems to be affecting other people and projects (its definitely not just me), see the below issues:

pytorch/pytorch#104598
python-poetry/poetry#7881

My gut instinct is that this is some network issue directly with PyPi servers or a bug in urllib3 as it seems to also be affecting Poetry ?

Expected behavior

Pip should be able to download any package regardless of size

pip version

23.0.1 & 23.2.1

Python version

3.10.12

OS

Docker (python:3.10 Intel image)

How to Reproduce

1. docker run --rm -it python:3.10
2. pip3 install torch

Output

Sometimes it failes directly downloading the PyTorch wheel

❯ docker run --rm -it python:3.10 /bin/bash
root@beac14d39a1a:/# pip3 install torch
Collecting torch
  Downloading torch-2.0.1-cp310-cp310-manylinux1_x86_64.whl (619.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 518.1/619.9 MB 104.2 MB/s eta 0:00:01
ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    torch from https://files.pythonhosted.org/packages/8c/4d/17e07377c9c3d1a0c4eb3fde1c7c16b5a0ce6133ddbabc08ceef6b7f2645/torch-2.0.1-cp310-cp310-manylinux1_x86_64.whl:
        Expected sha256 8ced00b3ba471856b993822508f77c98f48a458623596a4c43136158781e306a
             Got        1969020faa6e8513ef18cb8def2c7b3407c2719c2ec84f3b36b8e792ede57bcd


[notice] A new release of pip is available: 23.0.1 -> 23.2.1
[notice] To update, run: pip install --upgrade pip

Other times it fails on subdepedencies that are also large:

root@7f44044175dc:/# pip --version
pip 23.2.1 from /usr/local/lib/python3.10/site-packages/pip (python 3.10)
root@7f44044175dc:/# pip install torch
Collecting torch
  Downloading torch-2.0.1-cp310-cp310-manylinux1_x86_64.whl (619.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 619.9/619.9 MB 4.3 MB/s eta 0:00:00
Collecting filelock (from torch)
  Obtaining dependency information for filelock from https://files.pythonhosted.org/packages/00/45/ec3407adf6f6b5bf867a4462b2b0af27597a26bd3cd6e2534cb6ab029938/filelock-3.12.2-py3-none-any.whl.metadata
  Downloading filelock-3.12.2-py3-none-any.whl.metadata (2.7 kB)
Collecting typing-extensions (from torch)
  Obtaining dependency information for typing-extensions from https://files.pythonhosted.org/packages/ec/6b/63cc3df74987c36fe26157ee12e09e8f9db4de771e0f3404263117e75b95/typing_extensions-4.7.1-py3-none-any.whl.metadata
  Downloading typing_extensions-4.7.1-py3-none-any.whl.metadata (3.1 kB)
Collecting sympy (from torch)
  Downloading sympy-1.12-py3-none-any.whl (5.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.7/5.7 MB 65.7 MB/s eta 0:00:00
Collecting networkx (from torch)
  Downloading networkx-3.1-py3-none-any.whl (2.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 75.1 MB/s eta 0:00:00
Collecting jinja2 (from torch)
  Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 133.1/133.1 kB 27.0 MB/s eta 0:00:00
Collecting nvidia-cuda-nvrtc-cu11==11.7.99 (from torch)
  Downloading nvidia_cuda_nvrtc_cu11-11.7.99-2-py3-none-manylinux1_x86_64.whl (21.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 21.0/21.0 MB 69.3 MB/s eta 0:00:00
Collecting nvidia-cuda-runtime-cu11==11.7.99 (from torch)
  Downloading nvidia_cuda_runtime_cu11-11.7.99-py3-none-manylinux1_x86_64.whl (849 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 849.3/849.3 kB 81.5 MB/s eta 0:00:00
Collecting nvidia-cuda-cupti-cu11==11.7.101 (from torch)
  Downloading nvidia_cuda_cupti_cu11-11.7.101-py3-none-manylinux1_x86_64.whl (11.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.8/11.8 MB 72.8 MB/s eta 0:00:00
Collecting nvidia-cudnn-cu11==8.5.0.96 (from torch)
  Downloading nvidia_cudnn_cu11-8.5.0.96-2-py3-none-manylinux1_x86_64.whl (557.1 MB)
     ━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━ 233.0/557.1 MB 100.0 MB/s eta 0:00:04
ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    nvidia-cudnn-cu11==8.5.0.96 from https://files.pythonhosted.org/packages/dc/30/66d4347d6e864334da5bb1c7571305e501dcb11b9155971421bb7bb5315f/nvidia_cudnn_cu11-8.5.0.96-2-py3-none-manylinux1_x86_64.whl (from torch):
        Expected sha256 402f40adfc6f418f9dae9ab402e773cfed9beae52333f6d86ae3107a1b9527e7
             Got        f06b20a13c305047f4b7a90fa0f8cf5476804ac70654dc0b5197e2cd6f362be5

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[byt3bl33d3r]

These issues seem to be related.

#11153
#11348

[uranusjr]

Downloading works for me. I would suspect this has something to do with your network setup. Merging this into other issues since there is no useful information not present elsewhere.