overlapping fragments code and test-config.json needs careful review

[jpiesing]

The code and configuration data for overlapping_fragments needs a careful review.

1. test-config.json appears to have duplicate identical playout[] arrays for each framerate family (i.e. 6 instead of 3).
2. The spec in 8.10.4 says;

For a track buffer that supports a media profile, Overlapped Fragment Playback of a CMAF Switching Set, consisting of K tracks refers to the following actions, starting with k=1 for the track with lowest quality to k=K with the highest quality:

The playout array uses ss1 and steps from [1,1,1] to [1.1.4] then [1,2,5] to [1,2,8] and so on.
However the Representations in ss1 are not in quality order. Looking at the MPD, I might expect the following to give increasing quality order ...

• 8; 480x270
• 7; 512x288
• 6; 768x432
• 4; 1024x576 @ 1.2 MBit
• 5; 1024x576 @ 1.5 Mbit
• 1; 1920x1080x25
• 3; 1920x1080x50 @ 4.9 Mbit
• 2; 1920x1080x50 @ 6.0 Mbit

3. When I look at a recording of this running, I see the following representations;

• L1 for video frames up to 200 (L1 and L2 are both 1080p)
• L2 for video frames from 401 to 800 (L2 is 50Hz)
  I think this is 401 to 600 at one bitrate and then 601 to 800 at a second bitrate but this change of representation isn't being flagged to the OF.
• J1 for video frames from 800 to 1200 (J1 is also 50Hz)
• L2 for video frames from 1200 to the end at 3000
  (1024x576 would be I and is completely missing, most likely I1 for the lower bitrate and I2)

[FritzHeiden]

Thanks for pointing that out, I will fix the playout parameters accordingly.

[jpiesing]

Thanks for pointing that out, I will fix the playout parameters accordingly.

The results from running this test in July were worse than many of the other tests. I really recommend not just running it but very carefully checking that it plays the media according to playout[] in the correct order and for the correct time.

[gitwjr]

@FritzHeiden noted a change to the playout parameters were needed. He will add additional comments on what he has done and its status.

[FritzHeiden]

As @jpiesing already suggested and what can be seen in the SS1 MPD, representations are not ordered by quality, so the playout parameter needs to be adjusted accordingly. Doing so results in the correct behaviour of playback of representations in order of increasing quality.

As the specification suggest, there should be multiple tracks which should be played in increasing order, however, the sparse matrix for video content specifies the overlapping fragments test to use t1 content, which only has one track. Should I change the test to use SS1 content?

[gitwjr]

Fraunhofer will test again on 2 TVs in the lab. If the results are OK, close this issue.

[jpiesing]

Fraunhofer will test again on 2 TVs in the lab. If the results are OK, close this issue.

@louaybassbouss @FritzHeiden Did you do these tests as mentioned before Christmas?

[louaybassbouss]

As @jpiesing already suggested and what can be seen in the SS1 MPD, representations are not ordered by quality, so the playout parameter needs to be adjusted accordingly. Doing so results in the correct behaviour of playback of representations in order of increasing quality.

As the specification suggest, there should be multiple tracks which should be played in increasing order, however, the sparse matrix for video content specifies the overlapping fragments test to use t1 content, which only has one track. Should I change the test to use SS1 content?

@jpiesing as @FritzHeiden mentioned, t1 content (see sparse matrix) is not suitable for overlapping fragments since it contains only a single representation. Even the test passes, this does not mean it is logically correct since the current playout using only a single representation (because t1 has only one presentation). We need content with multiple presentations and @FritzHeiden proposed ss1. Do you agree?

[jpiesing]

As @jpiesing already suggested and what can be seen in the SS1 MPD, representations are not ordered by quality, so the playout parameter needs to be adjusted accordingly. Doing so results in the correct behaviour of playback of representations in order of increasing quality.
As the specification suggest, there should be multiple tracks which should be played in increasing order, however, the sparse matrix for video content specifies the overlapping fragments test to use t1 content, which only has one track. Should I change the test to use SS1 content?

@jpiesing as @FritzHeiden mentioned, t1 content (see sparse matrix) is not suitable for overlapping fragments since it contains only a single representation. Even the test passes, this does not mean it is logically correct since the current playout using only a single representation (because t1 has only one presentation). We need content with multiple presentations and @FritzHeiden proposed ss1. Do you agree?

Yes I agree.
@louaybassbouss @FritzHeiden do you want me to make a proposal in the sparse matrix that you will use or would you prefer to make something useful and me to update the sparse matrix to match?

[jpiesing]

February 20th meeting: Fraunhofer to go ahead and use ss1. Documentation can then be adjusted afterwards.

[jpiesing]

March 5th
Everything behaves correctly and looks good with SS1 but didn't do any in depth investigation.
This was run in the plugfest last week but recordings are yet to be analysed.
Leave it open until those recordings have been run with the OF and close it if things still look good.

[jpiesing]

In the London plugfest;

• the "[OF] Video: The presented sample shall match the one reported by the currentTime value within the tolerance of +/-(2/framerate + 20ms)" observation failed on all 8 TV sets.
• the "[OF] Video: The playback duration shall match the duration of the CMAF Track" observation failed on 6 of 8 TVs

[jpiesing]

In the London plugfest;

* the "[OF] Video: The presented sample shall match the one reported by the currentTime value within the tolerance of +/-(2/framerate + 20ms)" observation failed on all 8 TV sets.

This is addressed by the solution to cta-wave/device-playback-task-force#123 .

* the "[OF] Video: The playback duration shall match the duration of the CMAF Track" observation failed on 6 of 8 TVs

On the 3 better TVs from the London Plugfest, the duration failures are due to missing frames, which are below the failure threshold in 2 of 3 cases.

• TV 2 – 2 missing frames aren't enough to fail the 'every frame shall be rendered' observation but result in the duration observation failing by 80ms​
• TV 4 – 27 missing frames, duration error is 50.39ms when tolerance is 50ms​
• TV 7 – 7 missing frames, duration short by 250ms so about the same

I recommend we close this when cta-wave/device-playback-task-force#123 is closed.