Parse / Donwloader stuck in inf loop if it did not reach max_num

[zMynxx]

When using the crawler I've encountered with issue regarding the max_num.
If cases where less images found than the provided "max_num", a infinite loop begin, stop any previous result from actually providing information.
Expected result should be immidiate stop when there nothing left to download, and exit safely.

The following example is for the greedy crawler with a url provided to use flicker search engine like so:

• search_phrase set to "gripper"
• max_num set to 30

def test_flicker(search_phrase: str, max_num: int) -> None:
    print("start testing FlickerImageCrawler")
    greedy_crawler = GreedyImageCrawler(parser_threads=4, storage={"root_dir": root_dir})
    greedy_crawler.crawl(f"https://www.flickr.com/search/?q={search_phrase}", max_num=max_num, min_size=(100, 100))

result: (downloaded image #27 and then inf loop)

INFO - downloader - image #27\thttps://combo.staticflickr.com/pw/images/favicons/f>
INFO - parser - parser-001 is waiting for new page urls
INFO - parser - parser-002 is waiting for new page urls
INFO - parser - parser-004 is waiting for new page urls
INFO - parser - parser-003 is waiting for new page urls
INFO - parser - parser-001 is waiting for new page urls
INFO - parser - parser-002 is waiting for new page urls
INFO - parser - parser-004 is waiting for new page urls
INFO - parser - parser-003 is waiting for new page urls
INFO - parser - parser-001 is waiting for new page urls
INFO - downloader - downloader-001 is waiting for new download tasks
INFO - parser - parser-002 is waiting for new page urls
INFO - parser - parser-003 is waiting for new page urls
INFO - parser - parser-004 is waiting for new page urls
INFO - parser - parser-001 is waiting for new page urls
INFO - parser - parser-002 is waiting for new page urls
INFO - parser - parser-003 is waiting for new page urls
INFO - parser - parser-004 is waiting for new page urls

[ZhiyuanChen]

Thank you for raising this issue, this seems very interesting. I'll look into it

[zMynxx]

Any updates?
If not, in the mean time can you please introduce a timeout mechanism? say 30s or so. This way the crawler will still be functional / operational. At the moment it is unstable due to that, to the point where I cannot use it :(

[ZhiyuanChen]

Sorry for this late reply.

Can you try if the latest commit fixes your issue?