ExoPlayer signals Player.STATE_ENDED with MPD transition from dynamic to static

[stevemayhew]

Version

Media3 main branch

More version details

Main@1ffeafecc374ed82d94ce16fb57c23f99cb78765

Devices that reproduce the issue

Any

Devices that do not reproduce the issue

None

Reproducible in the demo app?

Yes

Use Case Background

Start Over / Catch Up (SoCu) is a use case where live playback is stored by the origin in a rolling buffer that the clients can address within a time window in order to perform a "Start Over", that is re-start live playback of an in-progress linear at the start of the program bounded to the end. This is typically done with a "start-over URL" is a SoCu URL where start_time is in the past and end_time is in the future. The origin will play this as live (dynamic MPD) until the end_time is reached then transition to VOD (static MPD). The VOD is accessible (the "Catch Up" case) for some time after the end_time, always starting as a simple static MPD.

Issue Steps

The steps to recreate the use-case are:

1. Create a "start-over URL", start playing it. Player starts live as expected
2. Seek back in the window to build some buffer back from live.
3. Wait until program ends (end_time is in the past)

It is expected that once end_time is in the past the MPD becomes static (see background notes below on "Transitioning Live to Static"). Playback should continue and play all the way to the end of the show, with ExoPlayer it transitons to Player.STATE_ENDED as soon as the current buffered media is exausted.

Transitioning Live to Static (On-Demand)

The DASH-IF IOP Working Group on updating the description of how to do this properly in a working group that is updating the IOP "Section 4 Liver Services". This is in the DASH-IF Interoperability WG, I am a reviewer for this and am looking at how ExoPlayer handles it.

The origin that reproduces this bug is following these guidelines (as currently written). The basic transiton to static does the following

1. Period@duration to the on-demand content duration
2. Period@startTime to 0.
3. removes all dynamic attributes from the MPD
4. adjusts SegmentTemplate@presentationTimeOffsetto map the existing segments time to the new period start at 0.

Analysis

Note that the working group has put a "fix" for this in dash.js, see issue Dash.js issue #3311, this "fix" is far from perfect as the fix does not appear to update the HTMLMediaElement's view of time correctly so seeks are not possible. For ExoPlayer a much more robust fix is possible as the mapping from Timeline to Period position is completely in control of ExoPlayer code.

My analysis of the ExoPlayer code shows it is handling the update of the period to a fixed duration correctly (setting the duration) but not reseting the mapping to render time correctly (so the current Period Position does not update)

Background

This background material is for readers that are not familure with how ExoPlayer handles position internally.

Positions In ExoPlayer

ExoPlayer maintains position in three coordinate spaces:

• Timeline — The Timeline abstracts one or more media items, the current playing Timeline is available to the client to determine a User Facing Position
• Window Position — The position relative to the current playing Window, this is the position returned by getCurrentPosition(), as long as an ad is not playing.
• Period Position — Stored internally asPlaybackInfo.positionUs, this is position relative to the current playing Period@startTime for DASH (note for HLS this is 0 as HLS is "single period". Most of ExoPlayer internally maintains position in the Period Time coordinate space. Window Position maps to Period Position via the Period.positionInWindowUs.

As ExoPlayer loads segments and extracts samples from them the actual queued timestamps are adjusted to Period Position as they are, for DASH the position on the sample timeline, actual EPT of the samples is adjusted to Period Position using the presentationTimeOffset when the MediaChunk is queued for loading. For HLS there is a single Period that starts at time 0, and sample PTS/EPT time is normalized to this time by the timestamp master (selected by the TimestampAdjuster)

User Facing Position

ExoPlayer exposes getCurrentPosition() to the client (UI). For non-ad playback this is the position relative to the start of the current playing Window. If the window is "live" (Window.isLive()), then the Window.windowStartTimeMs will be the time since the epoch of the start of the window. For HLS windowStartTimeMs comes from the EXT-X-PROGRAM-DATE-TIME, and for DASH it is the availability window

Period Position

ExoPlayer maintains multiple Periods in a set of MediaPeriodHolder objects, these match up with DASH Periods. The methods toPeriodTime() and fromPeriodTime() map between render position (the Sample Timeline) and Period Position

Suggested Fix

Looking at the code in ExoPlayerImplInternal.handleMediaSourceListInfoRefreshed() IMO the solution is to recoginze the Period Position has changed and update PlaybackInfo.positionUs to match this, note the renderPositionUs would not change and it would be best not to stop and/or flush the renderers as only the mapping has changed not the actual samples and segments.

Logging shows the update is seem (the change to positionInWindow after the new DashTimeline is copied) but this is after the position discontinuity check, here is the change to dynamic is false:

06-10 14:31:01.818 20849 21140 D ExoPlayerImplInternal: window became static
06-10 14:31:01.822 20849 21140 D ExoPlayerImplInternal: call handlePositionDiscontinuity() - periodPositionChanged: false requestedPositionChanged: true
06-10 14:31:01.825 20849 21140 D ExoPlayerImplInternal: before playbackInfo: PlaybackInfo - bufferedPosition: 537398862 position: 537363466 period: [ id: id_PT1717517582S duration: -9223372036854775807 positionInWindow: -537074566] window: [ dynamic: true placehold: false duration: 324296 positionInFirstPeriod: 537074566 windowStartTime: 1718054657500]
06-10 14:31:01.827 20849 21140 D ExoPlayerImplInternal: after playbackInfo: PlaybackInfo - bufferedPosition: 537398862 position: 537363466 period: [ id: id_PT1717517582S duration: -9223372036854775807 positionInWindow: -537074566] window: [ dynamic: true placehold: false duration: 324296 positionInFirstPeriod: 537074566 windowStartTime: 1718054657500]
06-10 14:31:01.827 20849 21140 D ExoPlayerImplInternal: renderPosition: 537363466 renderPosition.toPeriodTime: 537363466
06-10 14:31:01.828 20849 21140 D ExoPlayerImplInternal: before copyWithTimeline: PlaybackInfo - bufferedPosition: 537398862 position: 537363466 period: [ id: id_PT1717517582S duration: -9223372036854775807 positionInWindow: -537074566] window: [ dynamic: true placehold: false duration: 324296 positionInFirstPeriod: 537074566 windowStartTime: 1718054657500]
06-10 14:31:01.828 20849 21140 D ExoPlayerImplInternal: after copyWithTimeline: PlaybackInfo - bufferedPosition: 537398862 position: 537363466 period: [ id: id_PT1717517582S duration: 361000 positionInWindow: -28] window: [ dynamic: false placehold: false duration: 354326 positionInFirstPeriod: 28 windowStartTime: -9223372036854775807]
06-10 14:31:01.829 20849 21140 D ExoPlayerImplInternal: renderPosition: 537363466 renderPosition.toPeriodTime: 537363466
06-10 14:31:02.599 20849 20849 D EventLogger: timeline [eventTime=105.82, mediaPos=537363.44, buffered=0.00, window=0, period=0, periodCount=1, windowCount=1, reason=SOURCE_UPDATE
06-10 14:31:02.600 20849 20849 D EventLogger:   period [361.00]
06-10 14:31:02.600 20849 20849 D EventLogger:   window [354.33, seekable=true, dynamic=false]
06-10 14:31:02.600 20849 20849 D EventLogger: ]

Note this is proved out pretty simply by issuing a seek right after the source update,

06-10 14:52:10.242 20849 20849 D EventLogger: positionDiscontinuity [eventTime=121.31, mediaPos=254.00, buffered=0.00, window=0, period=0, reason=SEEK, PositionInfo:old [window=0, period=0, pos=538644269], PositionInfo:new [window=0, period=0, pos=254000]]
06-10 14:52:10.245 20849 20849 I EventLogger: state [eventTime=121.31, mediaPos=254.00, buffered=0.00, window=0, period=0, BUFFERING]
06-10 14:52:10.274 20849 20849 D EventLogger: isPlaying [eventTime=121.34, mediaPos=254.00, buffered=0.00, window=0, period=0, false]

The pull request pull #1451 basically does this.

Expected result

Media plays to the end.

Actual result

It is expected that once end_time is in the past the MPD becomes static (see background notes below on "Transitioning Live to Static"). Playback should continue and play all the way to the end of the show, with ExoPlayer it transitons to Player.STATE_ENDED as soon as the current buffered media is exausted.

Media

I can send a sample URL with instructions how to set start/end time.

Bug Report

• You will email the zip file produced by adb bugreport to [email protected] after filing this issue.

[tonihei]

Thanks for the detailed explanation :) I'm still not fully sure I understand everything correctly. Let me rephrase what I understand and maybe you can point out where I got this wrong if needed:

• The live MPD keeps updating as usual. It has a period starting at a fixed point in time, but the live window only allows to play the last N seconds or keeps growing to allow 'catch-up'. Notably, the available live window has an offset in the period (which I think is often corresponding to UTC timestamps, but can be arbitrary).
• The switch to a static MPD keeps the same period(s), but changes the availability window to cover all the content. Now for the interesting part:
  • Generally, the assumption is that timestamps in the periods always match to the same content. This I think is the reason the IOP lists "SegmentTemplate@presentationTimeOffset does not change" as a requirement for MPD updates, which is also mentioned in the main ISO 23009-1 (although there is just says "any Representation shall provide functionally identical Segments with the same indices in the corresponding Representation", no mention of presentationTimeOffset).
  • Contrary to that, you described above that SegmentTemplate@presentationTimeOffset does change and this is also what I can see in your logs (changing positionInWindow despite the same window.durationUs). So I think this also means the same period timestamp no longer corresponds to the same content?
  • The IOP rules also say that "In a static MPD, the first period starts at the time zero of the MPD timeline.". This is interesting because as soon as you change the type to 'static' it becomes impossible to both start the period at time zero and keep the SegmentTemplate@presentationTimeOffset unchanged.
  • The new Workgroup proposal tries to fix this conundrum by allowing an update to SegmentTemplate@presentationTimeOffset to let the period start at zero (I think, don't have access to the linked WG document).
• ExoPlayer's playback is likely broken by the fact that it keeps its assumption of "same timestamp in period" == "same content". And the proposed fix PR tries to change that by updating the playback position at the point where this specific transition is detected.

If all of that makes sense, I think the fix needs to be in the DashManifestParser or maybe rather in the logic that handles the update in DashMediaSource.

• We should definitely not violate the assumption of "same timestamp in period" == "same content". Doing that opens a whole can of worms of potential pitfalls and special cases we might need to add in the future. Your PR for example may cause problems for other non-DASH streams that see a similar dynamic->static update.
• Instead, we should be able to keep the old timing model intact and consistent with what ExoPlayer already published. That is, keep the Period.positionInWindow as it was before and only update the duration of the period/window as needed. This still needs some thought to cover cases like re-preparing the player after an error (which likely needs to keep this offset that is no longer in the MPD at all).

Could you confirm this matches your understanding? And if you agree on the proposed fix, we are happy to look into another PR to implement that.

[google-oss-bot]

Hey @stevemayhew. We need more information to resolve this issue but there hasn't been an update in 14 weekdays. I'm marking the issue as stale and if there are no new updates in the next 7 days I will close it automatically.

If you have more information that will help us get to the bottom of this, just add a comment!

[stevemayhew]

Sorry @tonihei I missed seeing your comment (I think you need to @stevemayhew to make GitHub notify me)

Let me work on getting you access to the DASH-IF working group on this. Perhaps we can direct the solution not to alter presentationTimeOffset (which is the source of the issue).

We have this code in our ExoPlayer version to deal with origin vendors that match the original DASH-IF guidelines (which encourage zeroing the Period start)

It is less than ideal as it forces a vacuous seek, as only the period position changes. This causes needless rebuffing

[tonihei]

Thanks, please let me know if you discover anything new.

[stevemayhew]

@tonihei Here's the document that DASH-IF is working on: https://github.com/Dash-Industry-Forum/DASH-IF-IOP/blob/master/specs/live2vod/01-live2vod.inc.md

We are not voting members anymore, but this is completely the wrong approach if we want to avoid what this pull request "fixes". I've started working on a pull request to re-write this document, but not sure I will have much weight.

Here's the link to my changes (WIP): https://github.com/stevemayhew/DASH-IF-IOP/tree/update-transition-options

Feel free to add whatever you can.

[stevemayhew]

I would say, for now, keep this pull request "DRAFT". We have this work-around in our code base, but it would be far better if origin vendors would get in-line behind the better approach to the live2vod transiton

[tonihei]

Okay, thanks! Please let us know if you find out something new and we can see how to incorporate that into our code.