Feature: Synchronise playback of multiple streams

[nikz]

Checklist

• Confirmed that there isn't already an existing related feature request.
  • (NOTE: If there is a related feature request, we encourage you to add to the discussion to add additional context or simply to upvote it)

Which Mux Elements/Packages would you like a new feature for?

• mux-video

🎥 Describe the feature you'd like to request

I would like to specify multiple Mux streams that would play back correctly synchronised.

💻 Describe the use cases the feature will solve

I have two separate Mux assets - one for the user's camera and one for the user's shared screen. I would like to be able to play them both back in the Mux player and have the timestamps line up correctly (even with buffering etc). My current solution involves processing the assets on the server with ffmpeg and reuploading, but I'd like to allow for e.g dynamic layouts.

⭐ Describe the solution you'd like to see

I'm open to suggestions! I may be simply doing it wrong™ :)

• Perhaps specifying multiple asset IDs, and a layout?
• Exposing the underlying hls.js events so I can do something like was suggested in hls.js

🔄 Describe alternatives or workarounds you've considered

• "baking in" the changes on the server via ffmpeg (slow, not dynamic etc)
• Multiple video players on the page and listening to the playback events (is jumpy with buffering etc)

[luwes]

This would be a really cool feature! I like it.

Browsers don't support something like this out of the box, they're essentially video tracks and audio tracks that could be enabled and positioned / muted but stay in sync time wise.
(Not supported on Chrome and Firefox, Safari supports this for native HLS)

This would require multiple <video> or <mux-video> elements to get working I believe, at least if you like to show 2 videos at the same time.

Multiple video players on the page and listening to the playback events (is jumpy with buffering etc)

This is pretty much the only thing that can be done front-end, the trick is indeed the syncing part.

---

The server side way might be more flexible than you think by using the canvas API.

Have a look at @mmcc's project
https://github.com/MuxLabs/video-sprites
https://video-sprites.mux.dev/

The gist is that you can crop out any part of the original playing video and render it on a HTML <canvas>, this cropped out video can be placed anywhere on the canvas by using 2d transformations and the rest of the video is just rendered below on the whole canvas.

[nikz]

Thanks @luwes, this is definitely an interesting one to try.... :D

[luwes]

I started looking into the full browser implementation, I think there's something we can do there to make this easier w/ mux-video and mux-player.

There used to be a spec for exactly this kind of behavior called the MediaController.
https://html.spec.whatwg.org/commit-snapshots/a4c46fc65e8ba138baeadf0bd1c2055b7bb43863/#dom-mediacontroller

Will keep you posted.

[nikz]

Ah, interesting - so one possibility would be to abstract away the playing streams using that interface? Does add complexity though :)

[luwes]

I have a early version ready at https://github.com/luwes/media-group, it still needs some improvements and tests but basic functionality is there. let me know if this works, cheers