Define codec registry for EncodedAudioChunk and EncodedVideoChunk byte representations

[manifest]

I'd like to encode audio and video frames in the browser, then process the chunks in an external application (e.g. a backend) and finally return them back to the browser. I assume that EncodedAudioChunk and VideoAudioChunk byte representations have to be stable because of the requirement of browser interoperability.

Where can I find information about EncodedAudioChunk and VideoAudioChunk byte representations? In particular, I'm interested in opus and vp8 codecs at the moment.

[sandersdan]

We expect this mapping to be described in a registry, the start of which is in #149.

More directly, each chunk is a complete sample and you can refer to the individual codec specifications for more details. Sample boundaries should match what you would expect in an MP4 containerization (eg. reference frames should be bundled into superframes). There is some ambiguity when SVC is involved that we are still analyzing.

[manifest]

Ok, I think that answers my question. Thank you.

[sandersdan]

@chcunningham I don't see any reference in #149 to packetization rules. There is a bit of text in the bitstream format part of the AVC entry, but I don't think it covers all of the questions we have received. (eg. "can I put SPS/PPS in an AVC format stream anyway?" and "should I send a buffer with just SPS/PPS for Annex B?")

[chcunningham]

I agree, we should add more clarity. Do you think codec registry is the way? It implies we have to create a registration for every codec (not otherwise our plan). It would simplify if we could find some way to describe this more generically in the main document.

[sandersdan]

So far I we've always chosen the rule of 'do it the way that it is spec'd for MP4 containerization', but we may want to relax those requirements in some cases. Every codec already has to be in the registry to give it a codec string, so it seems the logical place.

Note that even if we just reference the MP4 containerization specs, there are inevitable points of divergence. For example:

• MP4 has (optional) parameter streams. I suspect we'll eventually allow configuration chunks (just and SPS+PPS, no slice data) to fill that role, at least for H.264.
• We support codecs that don't have an MP4 containerization, eg. Theora. (There is probably an OGV spec we can point to for that.)

FYI the AVC-in-MP4 spec (ISO/IEC 14496-15) devotes 4 pages to defining "sample" (WebCodecs chunk) and "configuration" (WebCodecs description).

[padenot]

(There is probably an OGV spec we can point to for that.)

https://www.theora.org/doc/Theora.pdf#page=159, Appendix A, it's not very long as well.

[chcunningham]

Triage note: marked 'editorial', as I believe we have (codec-specific) consensus on what a chunk is, but the registry should be updated to describe that.

[padenot]

Per #155 (comment)

[chcunningham]

Registry now defined. All codecs registered.