Timeouts (short dns timeout value?) #2
Comments
|
Same here i.e. Thanks @threatpatrols for creating this tool! Cheers |
|
Thanks for the report(s) @gregsadetsky and @wildegnux I've not had a chance to look into this yet, however I've been able to compare Python 3.10 vs 3.11 and somewhere in between something has broken :( My current limited check is that I've just used Not entirely sure the root cause is a DNS lookup issue yet, as I get slightly different messages in my 3.11 test environment |
|
Note to self
|
|
New release v0.3.0 that might resolve this issue, unfortunately it's been really hard to replicate the errors on this to chase it down - https://github.com/threatpatrols/hibp-downloader/releases/tag/0.3.0 There are two key changes that I'm expecting should move the needle on this issue though -
The code is more robust around handling worker threads when they have HTTPX exceptions so we should see less infinite loop scenarios, however clean thread exiting (eg SIGINT with ^C) is still not a thing :( Please do send feedback so we can gauge if this resolves. |
|
Occasional errors still occur on Python 3.11.6 (on EndeavourOS): Debug log doesn't seem to tell me anything else about those threads that might be useful. But the failing threads do exit cleanly now so there's no more infinite wait. And to workaround the errors once could just run it in a loop until it stops giving errors. |
|
I can also reproduce the same errors on a different machine (in a different network) with Python 3.10.12 on Ubuntu 22.04.3 LTS (running under WSL) |
|
Okay, that helps - are you able to provide a list of the installed Python packages in each virtual environment you have there? |
|
Python 3.11.6: Python 3.10.12: |
|
I am also experiencing the |
|
Yes, retries are not a thing in the current code - would surely help Current thinking is that the timeouts would make sense if you consider this thing creates a lot connections which may in-turn cause an upstream network component to grok - for example a small home router may not have enough NAT state table and then intermittently show connection problems. This line of thinking might explain why I've rarely been able to replicate this most of the time. Could you give it another spin and reduce the --processes and --chunk-size values - if this is the underlaying issue then a feature to dynamically scale back to suit the local network conditions might be the thing needed here |
|
New release 0.3.1 implements retries and provides CLI options to adjust the default values Feedback would be great |
|
Sorry for the lack of feedback! Will run some test now! |
|
I still run into some threads crashing: And now some retries (nice 👍 ) Unclear to me if that crashed thread is one of the retries... That test was run on my laptop in my office (wifi) network using the default options. Running the same right now on my home network. Will also run a test on my home desktop (WSL). |
|
Test from home network: No crashed threads, but more of these failed downloads: But all failures succeeded on attempt 2. 👍 |
|
Test in WSL (home network): With default options (which on this machine means 32 processes) it runs full boar for ~30-60 sec, then completely stops for ~30 sec (matching the http-timeout value). Then all the active download fails and it continues going at full speed again for another minute and then stops again and so on. Reducing processes / chunks size does improve things, but I had to go down to 8 processes and 1 chunk for getting it to not occasionally lock up. Might be some funky stuff in the WSL network stack I guess, since it worked mostly fine one the linux laptop in the same network. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I'm repeatedly getting the following errors running the downloader:
The thread managing also does not handle the threads crashing, making those threads stall. Eventually every thread crashes and the main process gets stuck waiting for the now crashed threads to close.
The text was updated successfully, but these errors were encountered: