Slow processing many (>100) Process tasks with a maximum running in parallel (~10)

[Plastonick]

It seems it's somewhat unnecessarily slow running a few tasks in parallel from a larger pool. Take ~1000 ProcessCallbackTasks loaded into the manager, with a rule forcing at most 10 running in parallel (from 0 -> 100 processor load).

The 'tick' method will loop through all 1000 tasks and determine if the task can run. The first ten will be set off running and added to $this->running, each of the 990 afterwards will not run since there are ten processes running already. In each of those 990 cases though, it will perform the moderately expensive task of creating a Profile for each of the ten running processes. The profile will set the number of running processes at ten even if the process is not still running.

Only once looping through all 990, will it then remove non-running processes from the pool, and allow further processes to run, but this again allows 10 processes to run before needlessly looping through 980 more.

Am I right in my thinking here? It seems to me, that the only change necessary is to attempt to remove non-running processes from the running list at an earlier date. It feels like there may be a solution already to my use-case though that I'm not aware of.

If not, then I'd be happy to look deeper into this and contribute some form of fix/new feature, though.

I also think I may be slightly abusing the processor load functionality!