Improve performance with large number of queued prompts

[miabrahams]

Mitigates Comfy-Org/ComfyUI_frontend#2435

ComfyUI's web app exhibits significant lag when a large number of prompts are queued. I verified what was found in the linked thread, pinpointing the slowdown to the specific call api.getQueue() as the number of prompts grows. Every time a prompt is submitted, the front-end synchronizes the entire queue state by making this call.

Profiling the back-end, I found most of the time is spent executing copy.deepcopy inside PromptQueue while a mutex is held.

This PR adds a new version of PromptQueue's get_current_queue method that performs copy.copy instead of copy.deepcopy. Allowing read-only shared access is safe as long as PromptQueue treats its queue items as immutable, which it does by returning deep copies in its other methods. I named the new version get_current_queue_volatile to remind callers they must only read and release these values.

Using the updated method allows Comfy to respond much faster to requests to the /queue route. The web app remains usable on my machine with several hundred queue items. I enqueued 300 prompts for these benchmarks.

Notice how the whole deepcopy block is gone, and you can actually see Comfy working on other stuff:

Caching the JSON representations may be another 80/20 fix. I could look into that in the future. A more complex but optimized solution would be to rewrite state sync so the front end only requests a diff with updates to the queue instead of a full sync. Still, this quick fix helps a lot.

[catboxanon]

Nice work looking into this with benchmarks to show.

I had something similar opened months ago with zero activity but I'll go ahead and close that since I think your implementation makes more sense.

[christian-byrne]

This is great.

Testing Network & UI Performance

1. Copy and paste the default workflow on top of itself 10 times to make large graph
2. Queue 100 times

With this PR:

1. Response times on the /queue and /history endpoints stays under 100ms
2. Takes ~3-5s for all jobs to be queued fully
3. Performance is slightly laggy

Without this PR:

1. Response times on the /queue and /history endpoints gradually increase from 100ms to 500ms, eventually going over 1000ms
2. Gradually decreases speed as jobs are queued. Is not even able to queue all 100 jobs eargerly, as the response times and lag become too much
3. Performance on client is slightly worse, not super noticeable difference on my machine though. Did not compare inference speed, only UI responsiveness

I can do some other tests later with more features that interact with the prompt queue.

[christian-byrne]

Testing Queue Operations/Features from UI

• Deleting individual items from history
• Cancelling individual items items in queue
• Cancelling all items in queue
• Queueing large batch at once
• Cancelling large batch then re-queueing batch

test-queue-pr-queue-button-features.mp4

Everything seems to work the same as before, with better performance in the case where we queue a large batch at once.

[0X-JonMichaelGalindo]

I queue ~400 gens before work (assets, game elements, & prototypes, doing best of 100 generation). This is epic. Thank you!