You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Please check the following items before reporting a bug, otherwise it may be closed immediately.
This is NOT a site-related "bugs", e.g. some site blocks me when using curl_cffi,
UNLESS it has been verified that the reason is missing pieces in the impersonation.
A code snippet that can reproduce this bug is provided, even if it's a one-liner.
Version information will be pasted as below.
Describe the bug
When using concurrent.futures.ThreadPoolExecutor to run multiple threads with the same impersonate='chrome' session (from requests_curl.Session(impersonate='chrome')), the script often crashes during the first few session.get() calls. However, no exception is raised when this happens.
The bug is difficult to reproduce consistently, but I find a pattern:
If the script hasn't been run for a long period (e.g., 10 hours), it always crashes the first time it's executed afterward (same computer, no restarts).
After that initial crash, any subsequent runs work fine without issues.
To Reproduce
My script is similar to the one below:
fireattack
changed the title
Cold starting curl_cffi session with with ThreadPoolExecutor crash silently
Cold (?) starting curl_cffi session with with ThreadPoolExecutor crashes silently
Oct 29, 2024
After that initial crash, any subsequent runs work fine without issues.
Do you mean that the second time you run the script, it works? It's really weird that a exited process would have effect on a non-related second process.
Yes, that's what I mean. I agree this is definitely weird. But it only happens after I switched to use curl_cffi so I have to assume it's related to it.
A small correction: it's not always the very first session.get(). As shown in newly added console output example, it fails after 3 threads finished successfully.
Also I'm using it against a CF-protected website. This is irrelevant, I can reproduce with https://example.com
fireattack
changed the title
Cold (?) starting curl_cffi session with with ThreadPoolExecutor crashes silently
Cold (?) starting curl_cffi session with ThreadPoolExecutor crashes silently
Oct 31, 2024
Please check the following items before reporting a bug, otherwise it may be closed immediately.
UNLESS it has been verified that the reason is missing pieces in the impersonation.
Describe the bug
When using
concurrent.futures.ThreadPoolExecutor
to run multiple threads with the sameimpersonate='chrome'
session (fromrequests_curl.Session(impersonate='chrome')
), the script often crashes during the first fewsession.get()
calls. However, no exception is raised when this happens.The bug is difficult to reproduce consistently, but I find a pattern:
If the script hasn't been run for a long period (e.g., 10 hours), it always crashes the first time it's executed afterward (same computer, no restarts).
After that initial crash, any subsequent runs work fine without issues.
To Reproduce
My script is similar to the one below:
Expected behavior
It should run without issue.
Observed:
Versions
pip freeze
dump dump.txtAdditional context
requests
,httpx
or real browsers. have never seen similar crash before.The text was updated successfully, but these errors were encountered: