Improve single-instance rendering performance

[tzvc]

We’ve discovered that running Remotion renderer on powerful VPSs (e.g., 48 cores, 350GB RAM) with increased concurrency does not consistently improve performance and can sometimes make rendering slower. Investigations point to two main bottlenecks:

Chrome Resource Allocation

Simply opening more Chrome tabs doesn’t linearly increase resource usage.
Running multiple browser instances (rather than multiple tabs in a single instance) can yield better (but still limited) performance gains.

OffthreadVideo Frame Extraction

Videos are extracted frame-by-frame, making it difficult to parallelize.
Composition with heavy OffthreadVideos can see significant slowdowns at high concurrency.

A thorough discussion of these findings, along with logs and performance charts, is available in the existing issue.

Goal
The goal of this issue is spark a conversation on solutions to improve single instance rendering performance, with for example:

New concurrency strategies (e.g., multi-browser instances)
Possible refactors for more efficient frame extraction (e.g., multi-threading/process compositor, preprocessing, optimistic caching, open the same video streams concurrently).

We use Remotion professionally and this would have a great impact for us, I'll open a $500 bounty for a working solution.

[algora-pbc]

💎 $500 bounty • zigg.team

⏸️ Bounty Status: ON HOLD

This bounty has been placed on hold to ensure any solutions align with the project's long-term direction. It might be reactivated in the future if the issue is ready to accept external contributions.

[abhishek818]

/attempt #4664

Algora profile	Completed bounties	Tech	Active attempts	Options
@abhishek818	3 remotion bounties + 20 bounties from 5 projects	Go, TypeScript, Rust & more		Cancel attempt

[JonnyBurger]

Thanks for filing and posting a bounty!

Although appreciated, I cancelled the bounty because there are tons of contributors who would attempt this without any context.
The hard issues we don't wanna outsource without also specifying how we expect the issue to be solved, otherwise random things happen which leads to overhead.

The feature request is valid but it concerns two completely separate things.

Frame extraction:
The path forward for improving frame extraction performance I think is to further progress @remotion/media-parser + the WebCodecs interface so that this can happen in-browser, without the overhead of Rust+serializing to images+Node at play. Plus it would make it possible to stream the video rather than downloading it entirely and if available use the hardware acceleration.

I'm working myself towards this goal, which would be a new video tag that would replace Video and OffthreadVideo in the long-term. Please understand though that this will still take some time.

Chrome resources:
It would be nice to support having multiple Chrome instances and be able to specify the concurrency like 2x2 (2 browser instances with 2 tabs each).

[tzvc]

@JonnyBurger understandable. Do let me know if there is anything we can do to offer support for this work