Pool is faster than ThreadPool in some situations

[apcamargo]

Not really a bug, but I'm posting this here to make it easier to find. We talked about it over email before, but someone brought it up to me again. Figured it'd be good to make this information searchable.

Pyrodigal's CLI command uses multiprocessing.pool.ThreadPool for parallel gene calling. The documentation suggests using ThreadPool too. But in my benchmarks, multiprocessing.pool.Pool was way faster. I mentioned this to @althonos, who did some tests and found that Pool is faster than ThreadPool when there are lots of CPUs available (though we didn't do super in-depth tests to figure out why).

If Pyrodigal is running slow for you, try using multiprocessing.pool.Pool instead. There's an example you can check out here.

[althonos]

Thanks for the report! Not sure in which case which one of threads and process pools is better, otherwise I'd select the best one by default when possible... For now I'm keeping threads as a default, but:

• I've added a CLI flag to switch to process instead of thread pools
• I've added a small section to the documentation describing this: https://pyrodigal.readthedocs.io/en/latest/guide/parallel.html