About pip 20.3+ proxy setting problem

[schwarzichet]

pip version

21.0.1

Python version

3.9.1

OS

Windows 10

I have known there exists some changes in pip 20.3+ proxy usage. I have read the #9216 and some related issues. I have the same problem with this comment and I can solve my problem by setting a HTTPS_PROXY="http://proxy.server:port"environment variable.
However, I wonder whether there exists a way to write this config in the pip config file.
I tried to write these in file but they all cannot work.

[global]
proxy = proxy.server:port
proxy = http://proxy.server:port
https.proxy = http://proxy.server:port
http.proxy = http://proxy.server:port

also, I cannot make it by using the --proxy command-line option. That's kind of wired.
I wonder the config file way, because I will use the proxy every time.

[isgallagher]

I am also experiencing this issue. Proxy option was working on pip-18.0 and quit working immediately after upgrading to pip-21.0.1.

I am able to work around the issue by setting the HTTPS_PROXY environment variable. Thanks @schwarzichet!

[hjoukl]

I can confirm problems with --proxy and --cert when pip-installing to a venv (pip 21.0.1, Python 3.6).

Looks to me like the situation is bit more subtle: I can successfully install several packages (e.g. pip, setuptools, jmespath, ...) behind our corporate proxy using --proxy $PROXY --cert $CERT.

However certain packages, namely atlassian-python-api in my case, trigger a mysterious setuptools>=40,0.8 dependency when pip-installing. I've checked the atlassian-python-api repo and can't find such setuptools dependency.

Note that I already upgraded setuptools to 53.0.0 in this venv before trying to install said package so this doesn't make sense in the 1st place.

The mysterious setuptools dependency leads to this command invocation (pip -v output, original abs. paths edited out):

$TMPDIR/test-venv/bin/python3 $TMPDIR/test-venv/lib/python3.6/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-ea24dejv/overlay --no-warn-script-location -v --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.8.0' wheel 

As you can see, neither --cert nor --proxy are propagated to this command, so it is bound to fail - no connection without the proxy, and SSL errors due to the necessity to provide custom CA certificate acceptance (also proxy-related, insane certificate provisioning here :-()

Workaround (as (partly) mentioned in the ticket comments): Use http_proxy, https_proxy and REQUESTS_CA_BUNDLE environment variables instead of --cert and --proxy.

So here's the full fun (slightly edited for brevity):

0 holger@pydev .../tmp $ env|grep -i proxy
1 holger@pydev .../tmp $ PROXY=proxy.example.internal.de:8080
0 holger@pydev .../tmp $ CERT=$HOME/ca-bundle.crt
0 holger@pydev .../tmp $ python3 --version
Python 3.6.5
0 holger@pydev .../tmp $ rm -fr ~/.cache/pip  # make sure we download from the net
0 holger@pydev .../tmp $ python3 -m venv test-venv
0 holger@pydev .../tmp $ $TMPDIR/test-venv/bin/pip install --proxy $PROXY --cert $CERT --timeout=120 --upgrade pip
Collecting pip
  Downloading https://files.pythonhosted.org/packages/fe/ef/60d7ba03b5c442309ef42e7d69959f73aacccd0d86008362a681c4698e83/pip-21.0.1-py3-none-any.whl (1.5MB)
Installing collected packages: pip
  Found existing installation: pip 9.0.3
    Uninstalling pip-9.0.3:
      Successfully uninstalled pip-9.0.3
Successfully installed pip-21.0.1
0 holger@pydev .../tmp $ $TMPDIR/test-venv/bin/pip install --proxy $PROXY --cert $CERT --timeout=120 --upgrade setuptools
Requirement already satisfied: setuptools in ./test-venv/lib/python3.6/site-packages (39.0.1)
Collecting setuptools
  Downloading setuptools-53.0.0-py3-none-any.whl (784 kB)
     |████████████████████████████████| 784 kB 10.4 MB/s 
Installing collected packages: setuptools
  Attempting uninstall: setuptools
    Found existing installation: setuptools 39.0.1
    Uninstalling setuptools-39.0.1:
      Successfully uninstalled setuptools-39.0.1
Successfully installed setuptools-53.0.0
0 holger@pydev .../tmp $ $TMPDIR/test-venv/bin/pip --version
pip 21.0.1 from /ae/data/tmp/holger/test-venv/lib/python3.6/site-packages/pip (python 3.6)
0 holger@pydev .../tmp $ $TMPDIR/test-venv/bin/pip install --proxy $PROXY --cert $CERT --timeout=120 atlassian-python-api -v
Using pip 21.0.1 from /ae/data/tmp/holger/test-venv/lib/python3.6/site-packages/pip (python 3.6)
Non-user install because user site-packages disabled
Created temporary directory: /tmp/pip-ephem-wheel-cache-tt3c0yyc
Created temporary directory: /tmp/pip-req-tracker-ivgdrv_i
Initialized build tracking at /tmp/pip-req-tracker-ivgdrv_i
Created build tracker: /tmp/pip-req-tracker-ivgdrv_i
Entered build tracker: /tmp/pip-req-tracker-ivgdrv_i
Created temporary directory: /tmp/pip-install-zxkt7vt5
1 location(s) to search for versions of atlassian-python-api:
* https://pypi.org/simple/atlassian-python-api/
Fetching project page and analyzing links: https://pypi.org/simple/atlassian-python-api/
Getting page https://pypi.org/simple/atlassian-python-api/
Found index url https://pypi.org/simple
Looking up "https://pypi.org/simple/atlassian-python-api/" in the cache
Request header has "max_age" as 0, cache bypassed
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/atlassian-python-api/ HTTP/1.1" 200 14313
Updating cache with response from "https://pypi.org/simple/atlassian-python-api/"
Caching due to etag
  Found link https://files.pythonhosted.org/packages/7d/5c/995fbd1ab86e3b3352fa8e0dba12a174bbe19290afcd85bbb8371ec8ce9c/atlassian-python-api-0.11.5.tar.gz#sha256=cb1c667c188420f45c501705625028f2deb21eaa0cfc98c0973a888a3fbb63af (from https://pypi.org/simple/atlassian-python-api/), version: 0.11.5
...
Given no hashes to check 156 links for project 'atlassian-python-api': discarding no candidates
Collecting atlassian-python-api
  Created temporary directory: /tmp/pip-unpack-7v5vynpb
  Looking up "https://files.pythonhosted.org/packages/45/35/a3e9be8951048b79c70caf25e1028e47271019aa335adebf3dfdd8a871e5/atlassian-python-api-3.5.2.tar.gz" in the cache
  No cache entry available
  Starting new HTTPS connection (1): files.pythonhosted.org:443
  https://files.pythonhosted.org:443 "GET /packages/45/35/a3e9be8951048b79c70caf25e1028e47271019aa335adebf3dfdd8a871e5/atlassian-python-api-3.5.2.tar.gz HTTP/1.1" 200 115878
  Downloading atlassian-python-api-3.5.2.tar.gz (115 kB)
     |███████████████████████████████ | 112 kB 29.1 MB/s eta 0:00:01  Ignoring unknown cache-control directive: immutable
  Updating cache with response from "https://files.pythonhosted.org/packages/45/35/a3e9be8951048b79c70caf25e1028e47271019aa335adebf3dfdd8a871e5/atlassian-python-api-3.5.2.tar.gz"
  Caching due to etag
     |████████████████████████████████| 115 kB 29.1 MB/s 
  Added atlassian-python-api from https://files.pythonhosted.org/packages/45/35/a3e9be8951048b79c70caf25e1028e47271019aa335adebf3dfdd8a871e5/atlassian-python-api-3.5.2.tar.gz#sha256=acd9a0e66ca56f403df2f047bd7efde1271fd7842a7382ed82db2ebaae72d3c5 to build tracker '/tmp/pip-req-tracker-ivgdrv_i'
  Created temporary directory: /tmp/pip-build-env-ea24dejv
  Running command /ae/data/tmp/holger/test-venv/bin/python3 /ae/data/tmp/holger/test-venv/lib/python3.6/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-ea24dejv/overlay --no-warn-script-location -v --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.8.0' wheel
  Using pip 21.0.1 from /ae/data/tmp/holger/test-venv/lib/python3.6/site-packages/pip (python 3.6)
  Non-user install by explicit request
  Created temporary directory: /tmp/pip-ephem-wheel-cache-r19g2cwj
  Created build tracker: /tmp/pip-req-tracker-ivgdrv_i
  Entered build tracker: /tmp/pip-req-tracker-ivgdrv_i
  Created temporary directory: /tmp/pip-install-2tyzxzq9
  1 location(s) to search for versions of setuptools:
  * https://pypi.org/simple/setuptools/
  Fetching project page and analyzing links: https://pypi.org/simple/setuptools/
  Getting page https://pypi.org/simple/setuptools/
  Found index url https://pypi.org/simple
  Looking up "https://pypi.org/simple/setuptools/" in the cache
  Request header has "max_age" as 0, cache bypassed
  Starting new HTTPS connection (1): pypi.org:443
  Incremented Retry for (url='/simple/setuptools/'): Retry(total=4, connect=None, read=None, redirect=None, status=None)
  WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f00e3f5d828>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/setuptools/
  Starting new HTTPS connection (2): pypi.org:443
  Incremented Retry for (url='/simple/setuptools/'): Retry(total=3, connect=None, read=None, redirect=None, status=None)
  WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f00e3f802b0>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/setuptools/
  Starting new HTTPS connection (3): pypi.org:443
  Incremented Retry for (url='/simple/setuptools/'): Retry(total=2, connect=None, read=None, redirect=None, status=None)
  WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f00e3f80400>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/setuptools/
  Starting new HTTPS connection (4): pypi.org:443
  Incremented Retry for (url='/simple/setuptools/'): Retry(total=1, connect=None, read=None, redirect=None, status=None)
  WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f00e3f80550>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/setuptools/
  Starting new HTTPS connection (5): pypi.org:443
  Incremented Retry for (url='/simple/setuptools/'): Retry(total=0, connect=None, read=None, redirect=None, status=None)
  WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f00e3f806a0>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/setuptools/
  Starting new HTTPS connection (6): pypi.org:443
  Could not fetch URL https://pypi.org/simple/setuptools/: connection error: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/setuptools/ (Caused by NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f00e3f807f0>: Failed to establish a new connection: [Errno -2] Name or service not known',)) - skipping
  Given no hashes to check 0 links for project 'setuptools': discarding no candidates
  ERROR: Could not find a version that satisfies the requirement setuptools>=40.8.0
  ERROR: No matching distribution found for setuptools>=40.8.0
  Exception information:
  Traceback (most recent call last):
    File "/ae/data/tmp/holger/test-venv/lib/python3.6/site-packages/pip/_vendor/resolvelib/resolvers.py", line 171, in _merge_into_criterion
      crit = self.state.criteria[name]
  KeyError: 'setuptools'

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/ae/data/tmp/holger/test-venv/lib/python3.6/site-packages/pip/_vendor/resolvelib/resolvers.py", line 318, in resolve
      name, crit = self._merge_into_criterion(r, parent=None)
    File "/ae/data/tmp/holger/test-venv/lib/python3.6/site-packages/pip/_vendor/resolvelib/resolvers.py", line 173, in _merge_into_criterion
      crit = Criterion.from_requirement(self._p, requirement, parent)
    File "/ae/data/tmp/holger/test-venv/lib/python3.6/site-packages/pip/_vendor/resolvelib/resolvers.py", line 83, in from_requirement
      raise RequirementsConflicted(criterion)
  pip._vendor.resolvelib.resolvers.RequirementsConflicted: Requirements conflict: SpecifierRequirement('setuptools>=40.8.0')

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/ae/data/tmp/holger/test-venv/lib/python3.6/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 122, in resolve
      requirements, max_rounds=try_to_avoid_resolution_too_deep,
    File "/ae/data/tmp/holger/test-venv/lib/python3.6/site-packages/pip/_vendor/resolvelib/resolvers.py", line 453, in resolve
      state = resolution.resolve(requirements, max_rounds=max_rounds)
    File "/ae/data/tmp/holger/test-venv/lib/python3.6/site-packages/pip/_vendor/resolvelib/resolvers.py", line 320, in resolve
      raise ResolutionImpossible(e.criterion.information)
  pip._vendor.resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=SpecifierRequirement('setuptools>=40.8.0'), parent=None)]

  The above exception was the direct cause of the following exception:

  Traceback (most recent call last):
    File "/ae/data/tmp/holger/test-venv/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 189, in _main
      status = self.run(options, args)
    File "/ae/data/tmp/holger/test-venv/lib/python3.6/site-packages/pip/_internal/cli/req_command.py", line 178, in wrapper
      return func(self, options, args)
    File "/ae/data/tmp/holger/test-venv/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 317, in run
      reqs, check_supported_wheels=not options.target_dir
    File "/ae/data/tmp/holger/test-venv/lib/python3.6/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 127, in resolve
      six.raise_from(error, e)
    File "<string>", line 3, in raise_from
  pip._internal.exceptions.DistributionNotFound: No matching distribution found for setuptools>=40.8.0
  Removed build tracker: '/tmp/pip-req-tracker-ivgdrv_i'
  Installing build dependencies ... error
WARNING: Discarding https://files.pythonhosted.org/packages/45/35/a3e9be8951048b79c70caf25e1028e47271019aa335adebf3dfdd8a871e5/atlassian-python-api-3.5.2.tar.gz#sha256=acd9a0e66ca56f403df2f047bd7efde1271fd7842a7382ed82db2ebaae72d3c5 (from https://pypi.org/simple/atlassian-python-api/). Command errored out with exit status 1: /ae/data/tmp/holger/test-venv/bin/python3 /ae/data/tmp/holger/test-venv/lib/python3.6/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-ea24dejv/overlay --no-warn-script-location -v --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.8.0' wheel Check the logs for full command output.

... # intential ctrl-c here by me :-)

KeyboardInterrupt
Removed atlassian-python-api from https://files.pythonhosted.org/packages/45/35/a3e9be8951048b79c70caf25e1028e47271019aa335adebf3dfdd8a871e5/atlassian-python-api-3.5.2.tar.gz#sha256=acd9a0e66ca56f403df2f047bd7efde1271fd7842a7382ed82db2ebaae72d3c5 from build tracker '/tmp/pip-req-tracker-ivgdrv_i'
Removed atlassian-python-api from https://files.pythonhosted.org/packages/69/cb/316acfd5a91de2bd3dfa7fc077a5a070dec06e66b9726b0a5ed262c9c50d/atlassian-python-api-3.5.1.tar.gz#sha256=072a73bfcfeedcd57b3bb678be769f967e1faf19c013dc4de55690fbc5a3e13b from build tracker '/tmp/pip-req-tracker-ivgdrv_i'
Removed build tracker: '/tmp/pip-req-tracker-ivgdrv_i'


#
# If I set proxy and certificate as env variables, everything works:
#

1 holger@pydev .../tmp $ http_proxy=$PROXY https_proxy=$PROXY REQUESTS_CA_BUNDLE=$CERT $TMPDIR/test-venv/bin/pip install --timeout=120 atlassian-python-api
Collecting atlassian-python-api
  Using cached atlassian-python-api-3.5.2.tar.gz (115 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting six
  Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
...
Successfully installed atlassian-python-api-3.5.2 certifi-2020.12.5 chardet-4.0.0 deprecated-1.2.11 idna-2.10 oauthlib-3.1.0 requests-2.25.1 requests-oauthlib-1.3.0 six-1.15.0 urllib3-1.26.3 wrapt-1.12.1

Best regards, Holger

[uranusjr]

As you can see, neither --cert nor --proxy are propagated to this command, so it is bound to fail.

So this is #6018. Closing as a duplicate then. Thanks for the investigation! Hopefully someone affected would be willing to step up and work on this 🙂

[hjoukl]

Aha, didn't see #6018 when looking at the issue tracker before filing my comment - so IIUC the mere existence of pyproject.toml triggers the problem. Thanks @uranusjr for the pointer.

[hello-world65]

@schwarzichet : thank you - you made my day. It took me days until I found this post being the right one for this issue with pip.