PAC download with custom CA

[DBS-ST-VIT]

Hello,
we need to use PX with a PAC file. The PAC file in our company is provided via an internal web server, what is using a TLS cert, that was signed by the internal CA.

Actually, this isn't a problem (normally), as the CA is part of the trust store of our windows 11 machines. But it seems like px isn't using this trust store and fails with a stack trace. If i download the PAC file manually and specify the local path, everything works fine.

Heres the stacktrace i was talking about:

PS C:\Users\myuser> px --config=D:\px.ini
Serving at :3128 proc Process-3
Serving at :3128 proc Process-2
Serving at :3128 proc Process-6
Serving at :3128 proc Process-8
Serving at :3128 proc Process-7
Serving at :3128 proc Process-4
Serving at :3128 proc Process-1
Serving at :3128 proc Process-9
Serving at :3128 proc Process-5
Serving at :3128 proc Process-11
Serving at :3128 proc Process-10
----------------------------------------
Exception occurred during processing of request from ('127.0.0.1', 60842)
Traceback (most recent call last):
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python312\Lib\socketserver.py", line 692, in process_request_thread
    self.finish_request(request, client_address)
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python312\Lib\socketserver.py", line 362, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python312\Lib\socketserver.py", line 761, in __init__
    self.handle()
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python312\Lib\http\server.py", line 436, in handle
    self.handle_one_request()
  File "C:\Users\myuser\pipx\venvs\px-proxy\Lib\site-packages\px\handler.py", line 128, in handle_one_request
    http.server.BaseHTTPRequestHandler.handle_one_request(self)
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python312\Lib\http\server.py", line 424, in handle_one_request
    method()
  File "C:\Users\myuser\pipx\venvs\px-proxy\Lib\site-packages\px\handler.py", line 230, in do_GET
    self.do_curl()
  File "C:\Users\myuser\pipx\venvs\px-proxy\Lib\site-packages\px\handler.py", line 159, in do_curl
    ipport = self.get_destination()
             ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\myuser\pipx\venvs\px-proxy\Lib\site-packages\px\handler.py", line 255, in get_destination
    servers, netloc, path = STATE.wproxy.find_proxy_for_url(
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\myuser\pipx\venvs\px-proxy\Lib\site-packages\px\wproxy.py", line 468, in find_proxy_for_url
    servers, netloc, path = super().find_proxy_for_url(url)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\myuser\pipx\venvs\px-proxy\Lib\site-packages\px\wproxy.py", line 258, in find_proxy_for_url
    return parse_proxy(self.pac.find_proxy_for_url(url, netloc[0])), netloc, path
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\myuser\pipx\venvs\px-proxy\Lib\site-packages\px\pac.py", line 88, in find_proxy_for_url
    proxies = self._ctxt.eval("FindProxyForURL")(url, host)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_quickjs.JSException: ReferenceError: 'FindProxyForURL' is not defined
    at <eval> (<input>)

----------------------------------------

I am wondering, whether px can be "convinced" to use the system trust store or ignore the fact, that it cannot validate the TLS certificate (which is indeed insecure, but we are not using it in a production use case).

[genotrance]

The current version of Px on Windows uses libcurl and comes with its own cert bundle. You could append your internal CA to the crt file (in Lib/site-packages/px/libcurl/curl-ca-bundle.crt) in the PEM format.

The way to fix this would be to add CURLSSLOPT_NATIVE_CA to the SSL options but it's unclear if it will work - curl/curl#14869

Oh, and that's a really bad stack trace - it should say that the cert verification failed. That needs to be improved as well.

[genotrance]

Better answer in #219. Leveraging a build that uses schannel will solve this issue and is already on the roadmap - switch to pymcurl as the backend.

I do need to make sure I update pymcurl to not set CAINFO on Windows so that it uses the system CA but also have some way to use the bundled CA if preferred for some reason.

[genotrance]

This is being fixed in mcurl v8.9.1.0. Px v0.10.0 will use mcurl as the backend and use schannel so custom certs will get picked up when added to the OS.

CURLSSLOPT_NATIVE_CA is only relevant to WolfSSL, GnuTLS and OpenSSL so not relevant to us.

[DBS-ST-VIT]

Hello @genotrance,
many thanks for your investigation and your feedback! :)

We are happily awaiting the release v0.10.0 and going to provide some feedback, whether it solves our problem. If we should test something, also in alpha or beta stage, just give us a ping here.

[genotrance]

v0.10.0 is out!