px 0.9.0 responds with "URL malformed" on macOS #207
Labels
Comments
|
Can you please share the full log for both the http and https requests, including the libcurl version info at the top? Also, is this an x86_64 Mac or the new M1/2? I don't have a machine to test the newer models. |
|
Thanks for looking into this. Sure, please find the full logs below. I'm using an Apple Silicon device, rather than an Intel-based one. The version of curl included with macOS appears to be built for x86_64 instead of arm64 though. I'd be happy to contribute a fix however I'm not sure where to start with this given that I can't see any malformed URLs. |
|
Adding a data point about possible http --version
3.2.2
http -I --proxy http://localhost:3130 https://google.com --follow -h
HTTP/1.1 200 OKI am on M1 Sonoma 14.3 with below details: curl --version
curl 8.4.0 (x86_64-apple-darwin23.0) libcurl/8.4.0 (SecureTransport) LibreSSL/3.3.6 zlib/1.2.12 nghttp2/1.58.0
Release-Date: 2023-10-11
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS GSS-API HSTS HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz MultiSSL NTLM NTLM_WB SPNEGO SSL threadsafe UnixSocketsPx info: config [proxy]
port = 3130
hostonly = 1Startup HTTPS logs: curl --proxy 'http://localhost:3130' https://google.com
curl: (56) CONNECT tunnel failed, response 503
MainProcess: MainThread: 1711471618: /_handle_request_noblock/verify_request/dprint: Client address: 127.0.0.1
MainProcess: MainThread: 1711471618: /_handle_request_noblock/verify_request/dprint: Host-only IP allowed
MainProcess: Thread_0: 1711471618: /do_curl/do_client_auth/dprint: No client authentication required
MainProcess: Thread_0: 1711471618: /do_curl/__init__/dprint: 2cb1b69ede760e79fbf71613f360cee7e2670cc4: New curl instance
MainProcess: Thread_0: 1711471618: /__init__/_setup/dprint: 2cb1b69ede760e79fbf71613f360cee7e2670cc4: CONNECT google.com:443 using HTTP/1.1
MainProcess: Thread_0: 1711471618: /_setup/set_tunnel/dprint: 2cb1b69ede760e79fbf71613f360cee7e2670cc4: HTTP proxy tunneling = True
MainProcess: Thread_0: 1711471618: /do_CONNECT/do_curl/dprint: 2cb1b69ede760e79fbf71613f360cee7e2670cc4: Path = google.com:443
MainProcess: Thread_0: 1711471618: /find_proxy_for_url/get_netloc/dprint: netloc = ('google.com', 443), path = /
MainProcess: Thread_0: 1711471618: /do_curl/get_destination/dprint: 2cb1b69ede760e79fbf71613f360cee7e2670cc4: Proxy = [('1', 80)]
MainProcess: Thread_0: 1711471618: /do_CONNECT/do_curl/dprint: 2cb1b69ede760e79fbf71613f360cee7e2670cc4: Configuring proxy settings
MainProcess: Thread_0: 1711471618: /set_proxy/set_tunnel/dprint: 2cb1b69ede760e79fbf71613f360cee7e2670cc4: HTTP proxy tunneling = False
MainProcess: Thread_0: 1711471618: /do_curl/set_curl_auth/dprint: SSPI not available and no username configured - no auth
MainProcess: Thread_0: 1711471618: /do_curl/set_headers/dprint: 2cb1b69ede760e79fbf71613f360cee7e2670cc4: Adding header => Host: google.com:443
MainProcess: Thread_0: 1711471618: /set_headers/set_useragent/dprint: 2cb1b69ede760e79fbf71613f360cee7e2670cc4: Setting user agent to curl/8.4.0
MainProcess: Thread_0: 1711471618: /do_curl/set_headers/dprint: 2cb1b69ede760e79fbf71613f360cee7e2670cc4: Adding header => Proxy-Connection: Keep-Alive
MainProcess: Thread_0: 1711471618: /do_curl/set_headers/dprint: 2cb1b69ede760e79fbf71613f360cee7e2670cc4: Delaying headers
MainProcess: Thread_0: 1711471618: /do/add/dprint: 2cb1b69ede760e79fbf71613f360cee7e2670cc4: Handles = 0
MainProcess: Thread_0: 1711471618: /add/_add_handle/dprint: 2cb1b69ede760e79fbf71613f360cee7e2670cc4: Add handle
MainProcess: Thread_0: 1711471618: /add/_add_handle/dprint: 2cb1b69ede760e79fbf71613f360cee7e2670cc4: Added handle
MainProcess: Thread_0: 1711471618: /do_curl/do/dprint: 2cb1b69ede760e79fbf71613f360cee7e2670cc4: Getting active socket
MainProcess: Thread_0: 1711471618: /do_curl/do/dprint: 2cb1b69ede760e79fbf71613f360cee7e2670cc4: Failed to get active socket: 48, 0
MainProcess: Thread_0: 1711471618: /do_CONNECT/do_curl/dprint: 2cb1b69ede760e79fbf71613f360cee7e2670cc4: Connection failed: 3; URL malformedFailed to get active socket: 48, 0;
MainProcess: Thread_0: 1711471618: /log_error/log_message/dprint: code 503, message 3; URL malformedFailed to get active socket: 48, 0;
MainProcess: Thread_0: 1711471618: /log_request/log_message/dprint: "CONNECT google.com:443 HTTP/1.1" 503 -
MainProcess: Thread_0: 1711471618: /remove/_remove_handle/dprint: 2cb1b69ede760e79fbf71613f360cee7e2670cc4: Remove handle: 3; URL malformedFailed to get active socket: 48, 0;
MainProcess: MainThread: 1711471624: /_handle_request_noblock/verify_request/dprint: Client address: 127.0.0.1
MainProcess: MainThread: 1711471624: /_handle_request_noblock/verify_request/dprint: Host-only IP allowed
MainProcess: Thread_0: 1711471624: /do_curl/do_client_auth/dprint: No client authentication required
MainProcess: Thread_0: 1711471624: /do_curl/__init__/dprint: 2cb1b69ede760e79fbf71613f360cee7e2670cc4: New curl instance
MainProcess: Thread_0: 1711471624: /__init__/_setup/dprint: 2cb1b69ede760e79fbf71613f360cee7e2670cc4: CONNECT google.com:443 using HTTP/1.1
MainProcess: Thread_0: 1711471624: /_setup/set_tunnel/dprint: 2cb1b69ede760e79fbf71613f360cee7e2670cc4: HTTP proxy tunneling = True
MainProcess: Thread_0: 1711471624: /do_CONNECT/do_curl/dprint: 2cb1b69ede760e79fbf71613f360cee7e2670cc4: Path = google.com:443
MainProcess: Thread_0: 1711471624: /find_proxy_for_url/get_netloc/dprint: netloc = ('google.com', 443), path = /
MainProcess: Thread_0: 1711471624: /do_curl/get_destination/dprint: 2cb1b69ede760e79fbf71613f360cee7e2670cc4: Proxy = [('1', 80)]
MainProcess: Thread_0: 1711471624: /do_CONNECT/do_curl/dprint: 2cb1b69ede760e79fbf71613f360cee7e2670cc4: Configuring proxy settings
MainProcess: Thread_0: 1711471624: /set_proxy/set_tunnel/dprint: 2cb1b69ede760e79fbf71613f360cee7e2670cc4: HTTP proxy tunneling = False
MainProcess: Thread_0: 1711471624: /do_curl/set_curl_auth/dprint: SSPI not available and no username configured - no auth
MainProcess: Thread_0: 1711471624: /do_curl/set_headers/dprint: 2cb1b69ede760e79fbf71613f360cee7e2670cc4: Adding header => Host: google.com:443
MainProcess: Thread_0: 1711471624: /set_headers/set_useragent/dprint: 2cb1b69ede760e79fbf71613f360cee7e2670cc4: Setting user agent to curl/8.4.0
MainProcess: Thread_0: 1711471624: /do_curl/set_headers/dprint: 2cb1b69ede760e79fbf71613f360cee7e2670cc4: Adding header => Proxy-Connection: Keep-Alive
MainProcess: Thread_0: 1711471624: /do_curl/set_headers/dprint: 2cb1b69ede760e79fbf71613f360cee7e2670cc4: Delaying headers
MainProcess: Thread_0: 1711471624: /do/add/dprint: 2cb1b69ede760e79fbf71613f360cee7e2670cc4: Handles = 0
MainProcess: Thread_0: 1711471624: /add/_add_handle/dprint: 2cb1b69ede760e79fbf71613f360cee7e2670cc4: Add handle
MainProcess: Thread_0: 1711471624: /add/_add_handle/dprint: 2cb1b69ede760e79fbf71613f360cee7e2670cc4: Added handle
MainProcess: Thread_0: 1711471624: /do_curl/do/dprint: 2cb1b69ede760e79fbf71613f360cee7e2670cc4: Getting active socket
MainProcess: Thread_0: 1711471624: /do_curl/do/dprint: 2cb1b69ede760e79fbf71613f360cee7e2670cc4: Failed to get active socket: 48, 0
MainProcess: Thread_0: 1711471624: /do_CONNECT/do_curl/dprint: 2cb1b69ede760e79fbf71613f360cee7e2670cc4: Connection failed: 3; URL malformedFailed to get active socket: 48, 0;
MainProcess: Thread_0: 1711471624: /log_error/log_message/dprint: code 503, message 3; URL malformedFailed to get active socket: 48, 0;
MainProcess: Thread_0: 1711471624: /log_request/log_message/dprint: "CONNECT google.com:443 HTTP/1.1" 503 -
MainProcess: Thread_0: 1711471624: /remove/_remove_handle/dprint: 2cb1b69ede760e79fbf71613f360cee7e2670cc4: Remove handle: 3; URL malformedFailed to get active socket: 48, 0;HTTP request curl --proxy 'http://localhost:3130' http://google.com
<!DOCTYPE HTML>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Error response</title>
</head>
<body>
<h1>Error response</h1>
<p>Error code: 400</p>
<p>Message: 3; URL malformed.</p>
<p>Error code explanation: 400 - Bad request syntax or unsupported method.</p>
</body>
</html>
MainProcess: MainThread: 1711471785: /_handle_request_noblock/verify_request/dprint: Client address: 127.0.0.1
MainProcess: MainThread: 1711471785: /_handle_request_noblock/verify_request/dprint: Host-only IP allowed
MainProcess: Thread_0: 1711471785: /do_curl/do_client_auth/dprint: No client authentication required
MainProcess: Thread_0: 1711471785: /do_curl/__init__/dprint: 053dd22528d83912b0b9f8156ced668967e3de66: New curl instance
MainProcess: Thread_0: 1711471785: /__init__/_setup/dprint: 053dd22528d83912b0b9f8156ced668967e3de66: GET http://google.com/ using HTTP/1.1
MainProcess: Thread_0: 1711471785: /do_GET/do_curl/dprint: 053dd22528d83912b0b9f8156ced668967e3de66: Path = http://google.com/
MainProcess: Thread_0: 1711471785: /find_proxy_for_url/get_netloc/dprint: netloc = ('google.com', 80), path = /
MainProcess: Thread_0: 1711471785: /do_curl/get_destination/dprint: 053dd22528d83912b0b9f8156ced668967e3de66: Proxy = [('1', 80)]
MainProcess: Thread_0: 1711471785: /do_GET/do_curl/dprint: 053dd22528d83912b0b9f8156ced668967e3de66: Configuring proxy settings
MainProcess: Thread_0: 1711471785: /do_curl/set_curl_auth/dprint: SSPI not available and no username configured - no auth
MainProcess: Thread_0: 1711471785: /do_curl/bridge/dprint: 053dd22528d83912b0b9f8156ced668967e3de66: Setting up bridge
MainProcess: Thread_0: 1711471785: /do_curl/set_headers/dprint: 053dd22528d83912b0b9f8156ced668967e3de66: Adding header => Host: google.com
MainProcess: Thread_0: 1711471785: /set_headers/set_useragent/dprint: 053dd22528d83912b0b9f8156ced668967e3de66: Setting user agent to curl/8.4.0
MainProcess: Thread_0: 1711471785: /do_curl/set_headers/dprint: 053dd22528d83912b0b9f8156ced668967e3de66: Adding header => Accept: */*
MainProcess: Thread_0: 1711471785: /do_curl/set_headers/dprint: 053dd22528d83912b0b9f8156ced668967e3de66: Adding header => Proxy-Connection: Keep-Alive
MainProcess: Thread_0: 1711471785: /do_curl/set_headers/dprint: 053dd22528d83912b0b9f8156ced668967e3de66: Setting headers
MainProcess: Thread_0: 1711471785: /do/add/dprint: 053dd22528d83912b0b9f8156ced668967e3de66: Handles = 0
MainProcess: Thread_0: 1711471785: /add/_add_handle/dprint: 053dd22528d83912b0b9f8156ced668967e3de66: Add handle
MainProcess: Thread_0: 1711471785: /add/_add_handle/dprint: 053dd22528d83912b0b9f8156ced668967e3de66: Added handle
MainProcess: Thread_0: 1711471785: /do_GET/do_curl/dprint: 053dd22528d83912b0b9f8156ced668967e3de66: Connection failed: 3; URL malformed
MainProcess: Thread_0: 1711471785: /log_error/log_message/dprint: code 400, message 3; URL malformed
MainProcess: Thread_0: 1711471785: /log_request/log_message/dprint: "GET http://google.com/ HTTP/1.1" 400 -
MainProcess: Thread_0: 1711471785: /remove/_remove_handle/dprint: 053dd22528d83912b0b9f8156ced668967e3de66: Remove handle: 3; URL malformedI can confirm from Network capture there's no traffic going out so it seems to fail within px itself |
|
To try to figure out the cause, I attached a debugger to a working install on Linux and a non-working install on macOS. The arguments passed to functions in the The following minimal program, as executed from the import libcurl
easy = libcurl.easy_init()
libcurl.easy_setopt(easy, libcurl.CURLOPT_VERBOSE, True)
libcurl.easy_setopt(easy, libcurl.CURLOPT_URL, 'http://example.com/'.encode('utf-8'))
err = libcurl.easy_perform(easy)
if err == libcurl.CURLE_OK:
print('Success')
else:
print('Error: ' + str(err))
libcurl.easy_cleanup(easy)To verify that I think that the issue is not just related to strings. For example, the Curiously, I found no difference between |
|
Probably related : |
|
I got same issue. here's curl response. here's px log another wired things is I set password in the ini file, log showes it loaded. But I can see 'Blank password for user' in the output log, looks like the password I set isn't working. |
|
Looks like we are not getting an active socket: Lines 998 to 1005 in d40f03f Error 48 = CURLE_UNKNOWN_OPTION but this is libcurl v8.6.0 and the feature was added in 7.45. I don't see why this should fail so will need additional debugging on libcurl side. It's possible that the new Macs get a wrong value for CURLINFO_ACTIVESOCKET. Line 2976 in d40f03f Meanwhile, Px is moving to pymcurl as the backend going forward. It will be great if that can be tested on the new Macs to see if this issue can be recreated. Definitely will try to test this case myself when porting is completed but any help will be appreciated. @chengwhynot - password has to be stored in keyring or env var, not in the configuration file. Please see the documentation on that. |
|
Just to let you guys know: I have the same issue on my Mac (MacBook Air, M2, macOS 14.5): MainProcess: Thread_0: 1726124894: /do_curl/set_headers/dprint: 5531a55ed452e842f5caf1eb07dfb1a19b1a5dab: Delaying headers
MainProcess: Thread_0: 1726124894: /do/add/dprint: 5531a55ed452e842f5caf1eb07dfb1a19b1a5dab: Handles = 0
MainProcess: Thread_0: 1726124894: /add/_add_handle/dprint: 5531a55ed452e842f5caf1eb07dfb1a19b1a5dab: Add handle
MainProcess: Thread_0: 1726124894: /add/_add_handle/dprint: 5531a55ed452e842f5caf1eb07dfb1a19b1a5dab: Added handle
MainProcess: Thread_0: 1726124894: /do_curl/do/dprint: 5531a55ed452e842f5caf1eb07dfb1a19b1a5dab: Getting active socket
MainProcess: Thread_0: 1726124894: /do_curl/do/dprint: 5531a55ed452e842f5caf1eb07dfb1a19b1a5dab: Failed to get active socket: 48, 0
MainProcess: Thread_0: 1726124894: /do_CONNECT/do_curl/dprint: 5531a55ed452e842f5caf1eb07dfb1a19b1a5dab: Connection failed: 3; URL malformedFailed to get active socket: 48, 0; |
|
Same issue:
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When running px 0.9.0 using Python 3.11.7 on macOS 14.3.1, requests are responded to with a "URL malformed" error.
px starts successfully, as shown below.
Upon requesting a HTTP URL, the following behaviour is observed.
Upon requesting a HTTPS URL, the following behaviour is observed.
No errors are written to stderr/stdout by px, however the following debug logs appear to be relevant.
The text was updated successfully, but these errors were encountered: