Add option to bypass http proxy

[lordyavin]

• Pip version: 10.0.1
• Python version: 2.7.8 & 3.6.3
• Operating system: Windows 7

Description:

I'm trying to install packages from a local (SVN) repository. Since I'm behind a proxy I have configured the proxy in the pip.ini. For local resources pip should bypass the proxy.

Please add an option to specify a list of IP address ranges, domains and domain wildcards to connect directly.

What I've run:

prompt> pip install https://svnsrv.domain.net/svn/MyRepo/trunk/some-package-1.0.0.zip

Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', timeout('timed out',))': /svn/MyRepo/trunk/some-package-1.0.0.zip

[pradyunsg]

Hey @lordyavin! Thanks for filing this issue.

As a workaround, if you pass --proxy="", pip will not use the proxy for that run.

I'm not sure how simple/difficult it would be to provide this support. Do you happen to know how other package managers handle this? That might be a good reference to gain insight from.

[lordyavin]

Why looking at other package managers? Just look at your frontend to the internet. Every browser is capable to bypass a proxy as I requested.

[pfmoore]

Because pip is a package manager, not a browser?

Regardless, as far as I know, pip respects the usual http_proxy, https_proxy and no_proxy environment variables, so why not just use those?

[lordyavin]

This information is new to me. Didn't find that in the manual. I'll give it a try, but I would prefer to configure the no_proxy side by side with the proxy server settings in the ini file.

[pradyunsg]

You're referring to the user guide? We could improve the documentation for this.

[pfmoore]

Agreed we probably should have docs for this. It's a "standard" feature, probably implemented by requests or one of our dependencies rather than by pip itself, so it's not entirely clear where we'd put it in the pip docs, but it would make sense to do so. (When I say "standard" feature, it's standard on Unix, but not on Windows, so that makes documenting it as a cross-platform feature is even mode relevant).

[kishankarun]

@pradyunsg , the workaround option didn't work for me to disable proxy

`
$ pip install flask-socketio --proxy=""
Collecting flask-socketio

WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x0000000004313BA8>, 'Connection to <IP_ADDRESS> timed out. (connect timeout=15)')':
`

Can we list the currently used proxy in pip?

[omniproc]

Because pip is a package manager, not a browser?

Regardless, as far as I know, pip respects the usual http_proxy, https_proxy and no_proxy environment variables, so why not just use those?

Because real life configurations are much more complex then this. Proxy/No-Proxy can change depending on application context. Also those proxy ENV vars are probably the most heavily used and yet most uncouth defined pseudo-standard in IT. The syntax is not defined in a standard leading to multiple competing / supported syntaxes. One app may use "*.domain" while another one will use ".domain". One App expects domains to be seperated by , another one expects ; as seperator. One allows to ignore subnets by using CIDR style adressing another doesn't support IP ranges at all.
Great fun if you try to actually run more then one application on a system.

GitHub issues for hundrets of projects are full of issues around this not defined pseudo standard syntax. Most refer to this documentation when asked for the formal definition of the expected syntax. Note how loosely defined it is. Note also how this is not a standard or something worth to be called definition at all since it leaves so much stuff open to interpretation. It's simply how wget decided to handle it.
And still people write software around this crap and expect it to work. Sorry: this is designed to break.

In addition command line args should always have higher precedence then ENV var imho since they are more explicit. However: when those ENV vars are set the suggested --proxy="" option did not work in my tests (Windows platform). Also it seems to be possible to set that param in the pip.ini. It's ignored just like the commandline arg regardless.

IMHO using the http_proxy ENV vars is a nice to have default. But because of the lack of definition and app-level control really every app should at least define a app-specific way to control proxy settings that always should have precedence over the http_proxy system config. At least until we someday will maybe finally have an OS independed, strictly defined proxy setting that can provide those often needed level of control.

[ssbarnea]

In the absence of a --no-proxy option I can confirm that neither --no-proxy="" is available. Like any other CLI tools, the is a rule that a CLI option always take precedence over ENV or config-file options.

This does not work with pip, as pip refuses to disable proxy when that is defined using http_proxy env vars.

Some woud wonder why is such a big deal? Try to remember that the HTTP proxy may do SSL rewriting and pip does not respect the system installed certificates.

So that chain of bugs contribute to providing a poor experience.

[stefanoborini]

Because pip is a package manager, not a browser?

Anything that uses HTTP is, in the end, a browser.

Regardless, as far as I know, pip respects the usual http_proxy, https_proxy and no_proxy environment variables, so why not just use those?

Because your proxy may depend on the index url. For example, I have to use different proxies depending on which URLs I am connecting to. The browser generally handles this through a proxy.pac, that computes dynamically the proxy depending on the URL, but neither pip nor using http_proxy does that.

[pradyunsg]

Is there prior art for other package managers to have a --no-proxy option? That would be a fairly strong indicator. :)

[stefanoborini]

Is there prior art for other package managers to have a --no-proxy option? That would be a fairly strong indicator. :)

Package managers generally have fairly flexible configuration files. Pip is not that smart on that respect.

$ wget --help | grep proxy
--no-proxy explicitly turn off proxy
--proxy-user=USER set USER as proxy username
--proxy-password=PASS set PASS as proxy password

npm has it (can be passed as --option)

https://docs.npmjs.com/misc/config#noproxy

[maths-qureshi]

On windows you can also use the following commands to set the proxy to nothing and then use pip without issues:
set https_proxy=
set http_proxy=

Execute the above two commands. It works for me that way.

[pradyunsg]

Okie; I think it's perfectly reasonable to add a --no-proxy flag for this case. If someone wants to file a PR adding a --no-proxy flag, they're welcome to.

[deveshks]

Hi @pradyunsg

I wanted to work on this PR. From what I could understand from looking at the code on how to approach it is as follows.

We check if the --proxy option is set at https://github.com/pypa/pip/blob/master/src/pip/_internal/cli/req_command.py#L118 and set the session.proxies with the values provided.

I assume that we can do a similar thing with checking if the --noproxy option is provided, and if it is, set the session.proxies['no_proxy] to True which will be checked at https://github.com/pypa/pip/blob/master/src/pip/_vendor/requests/sessions.py#L293 and the proxy won't be used in further requests.

I am assuming that's all there is to it, but. feel free to add more to what is needed to be done.

[chrisinmtown]

Too bad this seems to have stalled, I could really use that --noproxy option today.

[uranusjr]

Contributions are always welcomed.

[hikigaya58]

Too bad this seems to have stalled, I could really use that --noproxy option today.

it seems that the no-proxy feature was added but not in style of CLI argument.
please refer to https://pip.pypa.io/en/stable/user_guide/#using-a-proxy-server

in my case(windows powershell), i can do:

>$Env:no_proxy = 1
>$Env:no_proxy
1
>pip list -o
Package    Version Latest Type
---------- ------- ------ -----
idna       2.10    3.2    wheel
requests   2.25.1  2.26.0 wheel
setuptools 56.0.0  57.4.0 wheel

given that my pip version is 21.1.3 and the issue #9972 , it work fine.
considered that the no-proxy is not a usual feature, i think hiding this on environment variable is a elegant design.
thanks for everyone's work.

[Solotov]

I still can't install any package with PIP

I need help!

PS C:\Users\AXA> pip install pyOpenSSL --proxy=off
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x0000000004D3D3D0>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed'))': /simple/pyopenssl/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x0000000004D3DA60>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed'))': /simple/pyopenssl/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x0000000004D3D670>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed'))': /simple/pyopenssl/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x0000000004D3D3A0>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed'))': /simple/pyopenssl/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x0000000004D79220>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed'))': /simple/pyopenssl/
ERROR: Operation cancelled by user
Traceback (most recent call last):
  File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_vendor\urllib3\connection.py", line 169, in _new_conn
    conn = connection.create_connection(
  File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_vendor\urllib3\util\connection.py", line 73, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "c:\users\axa\appdata\local\programs\python\python39\lib\socket.py", line 953, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 11001] getaddrinfo failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\users\axa\appdata\local\programs\python\python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\users\axa\appdata\local\programs\python\python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\AXA\AppData\Local\Programs\Python\Python39\Scripts\pip.exe\__main__.py", line 7, in <module>
  File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_internal\cli\main.py", line 71, in main
    return command.main(cmd_args)
  File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_internal\cli\base_command.py", line 104, in main
    return self._main(args)
  File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_internal\cli\base_command.py", line 221, in _main
    self.handle_pip_version_check(options)
  File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_internal\cli\req_command.py", line 147, in handle_pip_version_check
    pip_self_version_check(session, options)
  File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_internal\self_outdated_check.py", line 152, in pip_self_version_check
    best_candidate = finder.find_best_candidate("pip").best_candidate
  File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_internal\index\package_finder.py", line 879, in find_best_candidate
    candidates = self.find_all_candidates(project_name)
  File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_internal\index\package_finder.py", line 824, in find_all_candidates
    page_candidates = list(page_candidates_it)
  File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_internal\index\sources.py", line 134, in page_candidates
    yield from self._candidates_from_page(self._link)
  File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_internal\index\package_finder.py", line 783, in process_project_url
    html_page = self._link_collector.fetch_page(project_url)
  File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_internal\index\collector.py", line 512, in fetch_page
  File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_internal\index\collector.py", line 422, in _get_html_page
    resp = _get_html_response(url, session=session)
  File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_internal\index\collector.py", line 120, in _get_html_response
    resp = session.get(
  File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_vendor\requests\sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_internal\network\session.py", line 449, in request
    return super().request(method, url, *args, **kwargs)
  File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_vendor\requests\sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_vendor\requests\sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_vendor\cachecontrol\adapter.py", line 53, in send
    resp = super(CacheControlAdapter, self).send(request, **kw)
  File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_vendor\requests\adapters.py", line 439, in send
    resp = conn.urlopen(
  File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_vendor\urllib3\connectionpool.py", line 696, in urlopen
    self._prepare_proxy(conn)
  File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_vendor\urllib3\connectionpool.py", line 964, in _prepare_proxy
    conn.connect()
  File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_vendor\urllib3\connection.py", line 353, in connect
    conn = self._new_conn()
  File "c:\users\axa\appdata\local\programs\python\python39\lib\site-packages\pip\_vendor\urllib3\connection.py", line 169, in _new_conn
    conn = connection.create_connection(
KeyboardInterrupt

[colceagus]

any updates on this request?

[pradyunsg]

The maintainers have said that a PR would be welcome, and no one has filed a PR for this so far.

[pradyunsg]

I've labelled this, to reflect that.

[martinezlc99]

Hi all - I dont mind taking a crack at implementing this. My plan is:

• define --no-proxy option in cmdoptions.py
• process --no-proxy option in req_command.py
  -- from here its not super clear since it seems like pip is using the requests library.
  -- I plan on setting PipSession(requests.Session).proxies to proxies={"http": "", "https": ""} which seems to be the way to bypass proxies per here and here.

Any thoughts? Just go ahead and submit PR and solicit feedback?

[notatallshaw]

Any thoughts? Just go ahead and submit PR and solicit feedback?

I would carefully read the documentation, especially the warnings, and what the resquest devs have said is the supported way to do this, rather than relying on stack overflow answers:

• https://requests.readthedocs.io/en/latest/user/advanced/#proxies
• Strange difference between passing proxy as None vs an empty string psf/requests#6153 (comment)
• Strange difference between passing proxy as None vs an empty string psf/requests#6153 (comment)
• https://requests.readthedocs.io/en/latest/api/#requests.Session.trust_env

I suspect any PR would face significant pushback if it's using an undocumented way to achieve this behavior.

[martinezlc99]

@notatallshaw - thanks!

Yes, I agree on not relying on stack overflow answers. I'll take a look at your pointers and then generate a PR for review. I ran into the same issues you link when researching this issue.

Specifically, I need to be ensure any http_proxy and https_proxy enviromental variables are not used with --no-proxy option. Looks like its not clear how this can be accomplished.

The request docs are a bit confusing on this issues.

OK, I'll dig around some. Thanks again.

[martinezlc99]

I've submitted a PR for this - #12011 . Not sure what the process is for getting a review.