MediaStreamTrackProcessor -> WebCodecs

[chcunningham]

In the recent editors call, @aboba mentioned receiving questions on using MediaStreamTrackProcessor w/ WebCodecs. We have some code demonstrating this in the explainer. Maybe its just a matter of sharing that more broadly? Or is there some aspect of the integration we need to highlight?

The explainer is somewhat psuedo-code, so I've made a similar demo here that folks can actually run in Chrome Canary (92).

The code looks like this

// Setup trackProcessor to grab every VideoFrame from the camera
let stream = await navigator.mediaDevices.getUserMedia({
  audio: false,
  video: true
});
let trackProcessor = new MediaStreamTrackProcessor(
  stream.getVideoTracks()[0]
);

// Log encoder outputs. A real app would do something like send to a remote peer.
function handleChunk(chunk, metadata) {
  console.log("handleChunk( timestamp=" + chunk.timestamp + " )");
}

// Setup VideoEncoder
let videoEncoder = new VideoEncoder({
  output: handleChunk,
  error: e => {
    console.log(e.message);
  }
});

// Read frames and encode()!
const reader = trackProcessor.readable.getReader();
while (true) {
  const { done, value } = await reader.read();

  if (done) return;

  if (videoEncoder.state == "unconfigured") {
    videoEncoder.configure({
      codec: "vp8",
      width: value.cropWidth,
      height: value.cropHeight
    });
  }

  videoEncoder.encode(value);
  value.close();
}

[aboba]

@chcunningham A frequent question relates to writableControl. From the sample, it would appear that this doesn't need to be hooked up. Is there any situation in which it would be useful to do something with it?

[chcunningham]

WebCodecs users are free to entirely ignore writableControl. They may opt-in to sending control signals if they find them useful.

Currently the only control signal is "request-frame". I'm guessing this is used when you need a new frame on-demand from a low fps stream (@guidou @alvestrand, do I have that right?). If so, WebCodecs users could send that signal if needed a frame. The camera's I've tested with push frames at a steady 20-30fps, so I've not needed this.

@alvestrand mentioned that they may eventually define a signal that allows request a different frame size from the camera. This could be useful if apps are adapting to congestion or viewport size changes.

[guidou]

You are totally right, both about control signals being an opt-in mechanism and about "request-frame" being used to request a new frame on demand from a low-fps stream. For example, a screen capturer capturing a presentation may not produce any frames while a presentation is not changing. In these cases, peer connections sometimes need to send extra frames as part of their protocol and send this signal to the screen capturer.

…

On Sat, Apr 24, 2021 at 5:19 AM chcunningham ***@***.***> wrote: WebCodecs users are free to entirely ignore writableControl. They may opt-in to sending control signals if they find them useful. Currently the only control signal is "request-frame". I'm guessing this is used when you need a new frame on-demand from a low fps stream ***@***.*** <https://github.com/guidou> @alvestrand <https://github.com/alvestrand>, do I have that right?). If so, WebCodecs users could send that signal if needed a frame. The camera's I've tested with push frames at a steady 20-30fps, so I've not needed this. @alvestrand <https://github.com/alvestrand> mentioned that they may eventually define a signal that allows request a different frame size from the camera. This could be useful if apps are adapting to congestion or viewport size changes. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#191 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADDRIL4INZVEKVACFFBZ5PLTKI2EFANCNFSM43PVCUXA> .

[aboba]

One other question that came up today in the editor's meeting. Is there a way to reuse buffers, rather than continually allocating/freeing them?

[padenot]

One other question that came up today in the editor's meeting. Is there a way to reuse buffers, rather than continually allocating/freeing them?

This is now being discussed in #212

[chcunningham]

I think we can track the remaining question in #212. Feel free to re-open if more to discuss.

[vitaly-castLabs]

That piece of sample code Chris posted in the beginning of this thread was very helpful to me, however I ran into a problem with Firefox - it does not support MediaStreamTrackProcessor. I wonder if there's a way to tie MediaStreamTrack from getUserMedia to a VideoEncoder, well, "manually"? I can't find any API to extract a frame from a track and pass it to the encoder or anything similar I can use in the absence of MediaStreamTrackProcessor

[padenot]

https://jan-ivar.github.io/polyfills/mediastreamtrackprocessor.js is a polyfill written by us that you can use. We're planning to implement MediaStreamTrackProcessor, video-only, worker-only (as it's the only thing that has been agreed upon).

[vitaly-castLabs]

@padenot Brilliant! Hugely appreciated! I had to remove the curly brackets from constructor({track}) to make it work, just FYI.

I suspected I had to something like that - not pretty performance-wise, but does the job for now for Safari (works pretty well actually) and Firefox (probably not fast enough and causes the encoder to compress the heck out of it, combined with very low frame rate):
https://vitaly-castlabs.github.io/webcodecs-mse-player/

[jan-ivar]

Chrome works with or without the curly brackets but they're needed per spec.

Glad you got it working! Not sure what's up with the encoder. This unencoded demo seems to run well for me in Firefox.

[vitaly-castLabs]

You're right, it was the problem of the original code calling the MSTP constructor wrong: vitaly-castLabs/webcodecs-mse-player@183504a

[vitaly-castLabs]

One more thing. @jan-ivar shouldn't controller.enqueue(new VideoFrame(canvas, {timestamp: t1})) use t1 * 1000 instead? performance.now() returns milliseconds which need to be converted to microseconds. I'm definitely getting wrong timestamps in Firefox...