Add support for HTTPS proxies (available to trio/asyncio)

[karpetrosyan]

Refs #722

Related #732 (Required for sync support)

TODO

• Add proxy_ssl_context argument to AsyncForwardHTTPConnection, AsyncTunnelHTTPConnection and AsyncHTTPProxy classes
• Add changelog

[tomchristie]

Looks like we could do with a docs update here...

https://www.encode.io/httpcore/proxies/

We probably want to split "Proxy SSL and HTTP Versions" into "Proxy SSL" which documents the proxy_ssl_context argument, and HTTP Versions which notes the HTTP/1.1 constraint.

[tomchristie]

Okay, so attempting to test this...

I'm using trio in this example, so as not to be blocked by #732.

Installation...

$ pip install httpcore trio trustme proxy.py

Use trustme to generate certs...

$ python -m trustme
Generated a certificate for 'localhost', '127.0.0.1', '::1'
Configure your server to use the following files:
  cert=/Users/tomchristie/GitHub/encode/httpcore/server.pem
  key=/Users/tomchristie/GitHub/encode/httpcore/server.key
Configure your client to use the following files:
  cert=/Users/tomchristie/GitHub/encode/httpcore/client.pem

Running the proxy...

$ proxy --cert-file server.pem --key-file server.key 

Make the requests...

import ssl
import trio
import httpcore


ctx = ssl.create_default_context()
ctx.load_verify_locations('client.pem')

async def main():
    async with httpcore.AsyncHTTPProxy(proxy_url="http://127.0.0.1:8899/", proxy_ssl_context=ctx) as http:
        response = await http.request("GET", "https://www.example.com/")
        print(response)
        response = await http.request("GET", "http://www.example.com/")
        print(response)


trio.run(main)

Fails with...

[...]
ssl.SSLError: [SSL: HTTPS_PROXY_REQUEST] https proxy request (_ssl.c:992)

What am I getting wrong here?

[tomchristie]

Also, occurs to me that something we could do here would be decouple this from #732.

Indicate in the CHANGELOG/docs that HTTPS proxies are only supported with trio and asyncio, and ensure that we raise a runtime error if TLS-in-TLS is attempted with the sync backend.

[T-256]

Fails with...

[...]
ssl.SSLError: [SSL: HTTPS_PROXY_REQUEST] https proxy request (_ssl.c:992)

I think problem is here:

httpcore/httpcore/_async/connection.py

Line 140 in f7c515c

if self._origin.scheme == b"https":

Where you set proxy_url="http://127.0.0.1:8899/", so self._origin.scheme is http, and it won't do tls handshake.

[T-256]

I also had this problem on #734 and ended up by scheme checking.
IMO we should do ssl context checking instead of scheme checking. (e.g. here too)

[karpetrosyan]

What am I getting wrong here?

Just change the proxy url scheme to https.

[karpetrosyan]

Also, occurs to me that something we could do here would be decouple this from #732.

Indicate in the CHANGELOG/docs that HTTPS proxies are only supported with trio and asyncio, and ensure that we raise a runtime error if TLS-in-TLS is attempted with the sync backend.

We should probably merge #732 first, then go over this one.

[tomchristie]

Wonderful thanks, tested now and all working as expected.

Okay so, should we add the following error cases here?...

• If proxy_url is set to https and proxy_ssl_context is unset, then raise an exception.
• If proxy_url is set to http and proxy_ssl_context is set, then raise an exception.

I'd suggest that this PR is neat and uncontentious, so let's work to getting it merged, as "Add support for HTTPS proxies (available to trio/asyncio)". Then follow up with the more awkward sync support.

Remaining here is...

• Documentation.
• Raise an error in the sync backend if TLS-in-TLS is attempted.

[tomchristie]

These docs are really nice, thanks! 💚

[karpetrosyan]

If proxy_url is set to https and proxy_ssl_context is unset, then raise an exception.

Because most users do not interact with SSL contexts, we can allow this behavior to keep things as simple as possible.

When using the public HTTPS proxy, for example, the user must create the default SSL context and pass it to the HTTPProxy class without any modifications or configurations.

If proxy_url is set to http and proxy_ssl_context is set, then raise an exception.

Agree

[tomchristie]

Great - No objections to this.

Welcome to merge once you're also happy with it @karosis88.