diff --git a/media-source-respec.html b/media-source-respec.html index 61046d8..d3d907b 100644 --- a/media-source-respec.html +++ b/media-source-respec.html @@ -78,7 +78,7 @@ - +
This specification extends {{HTMLMediaElement}} [[HTML]] to allow JavaScript to generate media streams for playback. @@ -87,10 +87,13 @@
-

On top of editorial updates, substantives changes since publication as a W3C Recommendation in November 2016 are the addition of a {{SourceBuffer/changeType()}} method to switch codecs, the possibility to create and use {{MediaSource}} objects off the main thread in dedicated workers, and the removal of the createObjectURL() extension to the {{URL}} object following its integration in the File API [[FILEAPI]]. For a full list of changes done since the previous version, see the commits.

+

On top of editorial updates, substantives changes since publication as a W3C Recommendation in November 2016 are the addition of a {{SourceBuffer/changeType()}} method to switch codecs, the possibility to create and use {{MediaSource}} objects off the main thread in dedicated workers, the removal of the createObjectURL() extension to the {{URL}} object following its integration in the File API [[FILEAPI]], and the addition of {{ManagedMediaSource}}, ManagedSourceBuffer}}, and {{BufferedChangeEvent}} interfaces supporting power-efficient streaming and active buffered media cleanup by the user agent. For a full list of changes done since the previous version, see the commits.

The working group maintains a list of all bug reports that the editors have not yet tried to address.

Implementors should be aware that this specification is not stable. Implementors who are not taking part in the discussions are likely to find the specification changing out from under them in incompatible ways. Vendors interested in implementing this specification before it eventually reaches the Candidate Recommendation stage should track the GitHub repository and take part in the discussions.

+

Changes since last publication

+

On top of editorial updates, substantives changes since publication as a W3C Recommendation in + November 2016 are the addition of a {{SourceBuffer/changeType()}} method to switch codecs, the possibility to create and use {{MediaSource}} objects off the main thread in dedicated workers, the removal of the createObjectURL() extension to the {{URL}} object following its integration in the File API [[FILEAPI]], and the addition of {{ManagedMediaSource}}, ManagedSourceBuffer}}, and {{BufferedChangeEvent}} interfaces supporting power-efficient streaming and active buffered media cleanup by the user agent. For a full list of changes done since the previous version, see the commits.

@@ -249,7 +252,6 @@

MediaSource Object

a media element=] algorithm. The extended [=resource fetch algorithm=] uses this internal slot to conditionally fail attachment of a {{MediaSource}} using a {{MediaSourceHandle}} set on a {{HTMLMediaElement}}'s {{HTMLMediaElement/srcObject}} attribute.

-
         enum ReadyState {
           "closed",
@@ -268,7 +270,6 @@ 

MediaSource Object

The source is still attached to a media element, but {{MediaSource}}'s {{MediaSource/endOfStream()}} has been called.
- @@ -401,28 +402,21 @@

Attributes

  • If the {{MediaSource/readyState}} attribute is not in the {{ReadyState/""open""}} state then throw an {{InvalidStateError}} exception and abort these steps.
  • -
  • Create a new SourceBuffer object and associated resources.
  • -
  • Set the {{SourceBuffer/[[generate timestamps flag]]}} on the new object to the value in the "Generate - Timestamps Flag" column of the byte stream format registry [[MSE-REGISTRY]] entry that is associated with +
  • + Let |buffer| be a new instance of a {{ManagedSourceBuffer}} if [=this=] is a {{ManagedMediaSource}}, or a {{SourceBuffer}} otherwise, with their respective associated resources.
  • +
  • + Set |buffer|'s {{SourceBuffer/[[generate timestamps flag]]}} to the value in the "Generate + Timestamps Flag" column of the [[[MSE-REGISTRY]]] entry that is associated with |type|. -
  • -
    -
    If the {{SourceBuffer/[[generate timestamps flag]]}} equals true:
    -
    - Set the {{SourceBuffer/mode}} attribute on the new object to - {{AppendMode/""sequence""}}. -
    -
    Otherwise:
    -
    - Set the {{SourceBuffer/mode}} attribute on the new object to - {{AppendMode/""segments""}}. -
    -
  • -
  • Add the new object to {{MediaSource/sourceBuffers}} and [=queue a task=] to [=fire an event=] named {{addsourcebuffer}} at {{MediaSource/sourceBuffers}}.
  • -
  • Return the new object.
  • +
  • + If |buffer|'s {{SourceBuffer/[[generate timestamps flag]]}} is true, set |buffer|'s {{SourceBuffer/mode}} to {{AppendMode/"sequence"}}. + Otherwise, set |buffer|'s {{SourceBuffer/mode}} to {{AppendMode/"segments"}}. +
  • +
  • [=List/Append=] |buffer| to [=this=]'s {{MediaSource/sourceBuffers}}.
  • +
  • [=Queue a task=] to [=fire an event=] named {{addsourcebuffer}} at [=this=]'s {{MediaSource/sourceBuffers}}.
  • +
  • Return |buffer|.
  • -
    ParameterTypeNullableOptionalDescription
    |type|{{DOMString}}
    Return type: SourceBuffer
    removeSourceBuffer

    Removes a {{SourceBuffer}} from {{MediaSource/sourceBuffers}}.

    @@ -898,6 +892,7 @@

    Detaching from a media element

  • Set {{MediaSource/[[port to main]]}} null.
  • Set the {{MediaSource/readyState}} attribute to {{ReadyState/""closed""}}.
  • +
  • If [=this=] is a {{ManagedMediaSource}}, then set {{ManagedMediaSource/streaming}} attribute to `false`.
  • Update {{MediaSource/duration}} to NaN.
  • Remove all the SourceBuffer objects from {{MediaSource/activeSourceBuffers}}.
  • @@ -2696,6 +2691,285 @@

    Event Summary

  • +
    +

    ManagedMediaSource interface

    +

    + A {{ManagedMediaSource}} is a {{MediaSource}} that actively manages its + memory content. Unlike a {{MediaSource}}, the [=user agent=] can evict content + through the [=ManagedMediaSource/memory + cleanup=] algorithm from its {{MediaSource/sourceBuffers}} (populated with {{ManagedSourceBuffer}}) + for any reason. +

    + +
    +      [Exposed=(Window,DedicatedWorker)]
    +      interface ManagedMediaSource : MediaSource {
    +        constructor();
    +        readonly attribute boolean streaming;
    +        attribute EventHandler onstartstreaming;
    +        attribute EventHandler onendstreaming;
    +      };
    +      
    + +

    Attributes

    +
    +
    + streaming +
    +
    +

    On getting:

    +
      +
    1. Return the current value of the attribute.
    2. +
    +
    +
    + +

    Event Summary

    + + + + + + + + + + + + + + + + + + + + +
    Event nameInterfaceDispatched when...
    + startstreaming + {{Event}} + A {{ManagedMediaSource}}'s {{ManagedMediaSource/streaming}} attribute changed from `false` to `true`. +
    + endstreaming + {{Event}} + A {{ManagedMediaSource}}'s {{ManagedMediaSource/streaming}} attribute changed from `true` to `false`. +
    + +

    Algorithms

    +

    `ManagedSourceBuffer` Monitoring

    +

    + The following steps are run periodically, whenever the [=MediaSource/SourceBuffer Monitoring=] algorithm is scheduled to run. +

    +

    + Having enough managed data to ensure uninterrupted playback + is an implementation defined condition where the user agent determines that + it currently has enough data to play the presentation without stalling for a + meaningful period of time. This condition is constantly evaluated to determine + when to transition the value of {{ManagedMediaSource/streaming}}. These transitions + indicate when the user agent believes it has enough data buffered or it + needs more data respectively. +

    +

    + Being able to retrieve and buffer data in an efficient way + is an implementation defined condition where the user agent determines that + it can fetch new data in an energy efficient manner while able to achieve the desired memory usage. +

    +
      +
    1. Run the {{MediaSource}} [=MediaSource/SourceBuffer Monitoring=] algorithm.
    2. +
    3. Let |can play uninterrupted and efficiently| be a flag that is true if the {{HTMLMediaElement/buffered}} attribute contains a {{TimeRanges}} that includes the current playback position and [=enough managed data to ensure uninterrupted playback=] + and is [=able to retrieve and buffer data in an efficient way=] +
      +
      If |can play uninterrupted and efficiently| is not equal to {{ManagedMediaSource/streaming}}, [=queue an element task=] on the [=media element=] + that runs the following steps: +
      +
      +
        +
      1. Set [=this=] {{ManagedMediaSource/streaming}} attribute to |can play uninterrupted and efficiently|.
      2. +
      3. If |can play uninterrupted and efficiently| is false [=fire an event=] called {{startstreaming}} at the {{ManagedMediaSource}}.
      4. +
      5. Otherwise, [=fire an event=] called {{endstreaming}} at the {{ManagedMediaSource}}.
      6. +
      +
      +
      +
    4. +
    + +

    Memory Cleanup

    +
      +
    1. +
      +
      + For each |buffer:ManagedSourceBuffer| in [=this=]'s + {{MediaSource/sourceBuffers}}: +
      +
      +
        +
      1. Run the |buffer|'s [=ManagedSourceBuffer/memory cleanup=] algorithm.
      2. +
      +
      +
      +
    2. +
    +
    + +
    +

    BufferedChangeEvent interface

    +
    +        [Exposed=(Window,DedicatedWorker)]
    +        interface BufferedChangeEvent : Event {
    +          constructor(DOMString type, BufferedChangeEventInit eventInitDict);
    +
    +          [SameObject] readonly attribute TimeRanges addedRanges;
    +          [SameObject] readonly attribute TimeRanges removedRanges;
    +        };
    +
    +        dictionary BufferedChangeEventInit : EventInit {
    +          TimeRanges addedRanges;
    +          TimeRanges removedRanges;
    +        };
    +      
    +

    Attributes

    +
    +
    + addedRanges +
    +
    + The time ranges added between the last {{updatestart}} and {{updateend}} events (which would have occurred during the last run of the [=coded frame processing=] algorithm). +
    +
    + removedRanges +
    +
    + The time ranges removed between the last `updatestart` and `updateend` events (which would have occurred during the last run of the [=coded frame removal=] or [=coded frame eviction=] algorithm or if the + user agent evicted content in response to a [=ManagedSourceBuffer/memory cleanup=]). +
    +
    +
    + +
    +

    ManagedSourceBuffer interface

    +
    +        [Exposed=(Window,DedicatedWorker)]
    +        interface ManagedSourceBuffer : SourceBuffer {
    +          attribute EventHandler onbufferedchange;
    +        };
    +      
    +

    Attributes

    +
    +
    + onbufferedchange +
    +
    +

    An [=event handler IDL attribute=] whose [=event handler event type=] is {{bufferedchange}}.

    +
    +
    + +

    Event Summary

    + + + + + + + + + + + + + + + +
    Event nameInterfaceDispatched when...
    + bufferedchange + {{Event}} + The {{ManagedSourceBuffer}}'s buffered range changed following a + call to {{SourceBuffer/appendBuffer()}}, {{SourceBuffer/remove()}}, + {{MediaSource/endOfStream()}}, + or as a consequence of the user agent running the + [=ManagedSourceBuffer/memory cleanup=] algorithm. +
    + +

    Algorithms

    +

    Buffered Change

    +

    + The following steps are run at the completion of all operations to the {{ManagedSourceBuffer}} + |buffer:ManagedSourceBuffer| that would cause a |buffer|'s {{SourceBuffer/buffered}} to change. + That is once {{SourceBuffer/appendBuffer()}}, {{SourceBuffer/remove()}} or + [=ManagedSourceBuffer/memory cleanup=] algorithm have completed. +

    +
      +
    1. + Let |previous buffered ranges:normalized TimeRanges| equal the + {{SourceBuffer/buffered}} attribute before the changes occurred. +
    2. +
    3. + Let |new buffered ranges:normalized TimeRanges| equal the new + {{SourceBuffer/buffered}} {{TimeRanges}}. +
    4. +
    5. + Let |added:normalized TimeRanges| equal the |previous buffered + ranges:normalized TimeRanges| subtracted from |new buffered + ranges:normalized TimeRanges|. +
    6. +
    7. + Let |removed:normalized TimeRanges| equal the |new buffered + ranges:normalized TimeRanges| subtracted from |previous buffered + ranges:normalized TimeRanges|. +
    8. +
    9. + Let |eventInitDict| be a new {{BufferedChangeEventInit}} dictionary + initialized with |added| as its {{BufferedChangeEventInit/addedRanges}} and + |removed| as its {{BufferedChangeEventInit/removedRanges}} +
    10. +
    11. + [=Queue a task=] to [=fire an event=] named {{bufferedchange}} at |buffer| + using the {{BufferedChangeEvent}} interface, initialized with + |eventInitDict|. +
    12. +
    +

    + Memory cleanup +

    +
      +
    1. +
      +
      + If [=this=] is not in [=this=]'s {{ManagedMediaSource}} parent + {{MediaSource/activeSourceBuffers}}: +
      +
      +
        +
      1. + Run the [=coded frame removal=] algorithm with start set to 0, end set to positive infinity, and abort these steps. +
      2. +
      +
      +
      +
    2. +
    3. + Let |removal ranges:normalized TimeRanges| equal a list of presentation + time ranges that can be evicted from the presentation to ensure + uninterrupted playback from {{HTMLMediaElement/currentTime}} until such + presentation could be retrieved again. +

      + Implementations can use different strategies for selecting |removal ranges| + so web applications shouldn't depend on a specific behavior. The web + application would listen to the {{bufferedchange}} event to observe + whether portions of the buffered data have been evicted. +

      +
    4. +
    5. + For each range in |removal ranges|, run the [=coded frame removal=] + algorithm with |start:double| and |end:unrestricted double| equal to the + removal range start and end timestamp respectively. +
    6. +
    +
    +

    HTMLMediaElement Extensions

    This section specifies what existing {{HTMLMediaElement}}.{{HTMLMediaElement/seekable}} and @@ -3162,7 +3436,65 @@

    Examples

    </script>
    +

    Use of the Managed Media Source Extensions

    +
    +      <script>
    +      async function setUpVideoStream() {
    +        // Specific video format and codec
    +        const mediaType = 'video/mp4; codecs="mp4a.40.2,avc1.4d4015"';
    +
    +        // Check if the type of video format / codec is supported.
    +        if (!window.ManagedMediaSource?.isTypeSupported(mediaType)) {
    +          return; // Not supported, do something else.
    +        }
    +
    +        // Set up video and its managed source.
    +        const video = document.createElement("video");
    +        const source = new ManagedMediaSource();
    +
    +        video.controls = true;
    +
    +        await new Promise((resolve) => {
    +          video.src = URL.createObjectURL(source);
    +          source.addEventListener("sourceopen", resolve, { once: true });
    +          document.body.appendChild(video);
    +        });
    +
    +        const sourceBuffer = source.addSourceBuffer(mediaType);
    +
    +        // Set up the event handlers
    +        sourceBuffer.onbufferedchange = (e) => {
    +          console.log("onbufferedchange event fired.");
    +          console.log(`Added Ranges: ${timeRangesToString(e.addedRanges)}`);
    +          console.log(`Removed Ranges: ${timeRangesToString(e.removedRanges)}`);
    +        };
    +
    +        source.onstartstreaming = async () => {
    +          const response = await fetch("./videos/bipbop.mp4");
    +          const buffer = await response.arrayBuffer();
    +          await new Promise((resolve) => {
    +            sourceBuffer.addEventListener("updateend", resolve, { once: true });
    +            sourceBuffer.appendBuffer(buffer);
    +          });
    +        };
     
    +        source.onendstreaming = async () => {
    +          // Stop fetching new segments here
    +        };
    +      }
    +
    +      // Helper function...
    +      function timeRangesToString(timeRanges) {
    +        const ranges = [];
    +        for (let i = 0; i < timeRanges.length; i++) {
    +          ranges.push([timeRanges.start(i), timeRanges.end(i)]);
    +        }
    +        return "[" + ranges.map(([start, end]) => `[${start}, ${end})` ) + "]";     
    +      }
    +      </script>
    +      <body onload="setUpVideoStream()"></body>
    +    
    +

    Acknowledgments