Seeking in MediaPlayerElement

[mc0re]

I assign a home-made stream to MediaPlayerElement.Source. To be more exact, the stream ID is assigned to a MediaBinder, which is wrapped into a MediaSource, which is wrapped into a MediaPlaybackItem, and a bunch of those items are added to a MediaPlaybackList, which is finally assigned to MediaPlayerElement.Source.

When I start the playback, I can trace that the player seeks close to the end of the file, so I assume it reads the MP4 look-up table - and knows it can seek in the stream. But when I click on the time bar in order to seek forward during the playback, the player starts reading the data sequentially towards the destination, instead of seeking directly to the destination offset - which it should know from the look-up table. This takes really long time (seeking 1 hour video file in ~256KB blocks takes a noticeable amount of seconds).

I assume I am missing something in the setup. Could you tell me, under which conditions the player reads sequentially instead of seeking, or what I can do to make it use seek?

[jonwis]

@aklemets - any idea?

[mc0re]

I wasn't sure where to write best, so there is a duplicate in WinUI repository - microsoft/microsoft-ui-xaml#2964

[mc0re]

@aklemets - Could you please have a look?

[aklemets]

In general, seeking in MP4 files should work fine. It should be easy to demonstrate by just opening an MP4 file in Movies & TV.
If you cannot find a suitable MP4 file for testing, you could go to peach.blender.org and download a large size version of the Big Buck Bunny MP4 file.

You said you have a problem with a "home-made stream". Have you tried looking for some MP4 file validation tool that can check the file for correctness? Or compare the layout of your file using any other "normal" MP4 file, such as the Big Buck Bunny MP4 file? It might be that your home-made file is missing some information needed for random access seeking.

[mc0re]

Is there a tool to validate CENC-encoded MP4 file? I know I can decrypt it, but that might change the structure...

[mc0re]

I tried to narrow the problem down, and got two issues.

1. If the video file is in some way corrupt (in my case, the video track was much shorter than the audio), the player exits abruptly without throwing any exception I could catch - both Application.UnhandledException and AppDomain.CurrentDomain.UnhandledException handlers were skipped.

2. I created a simple player for trying out the playback, and fed a standard MP4 file into it. Opening as file is okay, opening as a stream is not. I'm probably doing something wrong there.

Opening as file

Media.Source = new MediaPlaybackItem(MediaSource.CreateFromStorageFile(file));

Opening as a stream

var inp = new HomeMadeStream(file, this);
await inp.Init();
Media.Source = new MediaPlaybackItem(MediaSource.CreateFromStream(inp, "video/mp4"));

In both cases, Media is MediaPlayerElement.

[mc0re]

Here is the trivial implementation of the stream. I can send the whole source code, if you tell me where to.

using System;
using System.Threading.Tasks;
using Windows.Foundation;
using Windows.Storage;
using Windows.Storage.Streams;


namespace CencPlayer
{
    public sealed class HomeMadeStream : IRandomAccessStream
    {
        #region Fields

        private readonly StorageFile mFile;

        private readonly IUiLogger mLogger;

        private ulong mPosition;

        private ulong mLength;

        private IRandomAccessStreamWithContentType mStream;

        #endregion


        #region Properties

        bool IRandomAccessStream.CanRead => true;

        bool IRandomAccessStream.CanWrite => false;

        ulong IRandomAccessStream.Position => mPosition;

        ulong IRandomAccessStream.Size { get => mLength; set => throw new NotImplementedException(); }

        #endregion


        #region Init and clean-up

        public HomeMadeStream(StorageFile file, IUiLogger logger)
        {
            mFile = file;
            mLogger = logger;
            mLogger.Log($"Opening {file.DisplayName}.");
        }


        private HomeMadeStream(StorageFile file, ulong position, ulong length, IUiLogger logger)
        {
            mFile = file;
            mPosition = position;
            mLength = length;
            mLogger = logger;
        }


        /// <summary>
        /// Needs to be called right after the constructor.
        /// Separated method due to async.
        /// </summary>
        public async Task Init()
        {
            var prop = await mFile.GetBasicPropertiesAsync();
            mLength = prop.Size;
            mLogger.Log($"Length {mLength}.");
        }


        void IDisposable.Dispose()
        {
            mStream.Dispose();
        }

        #endregion


        #region API

        IInputStream IRandomAccessStream.GetInputStreamAt(ulong position)
        {
            mLogger.Log($"Clone stream from position {position}.");
            return new HomeMadeStream(mFile, position, mLength, mLogger);
        }


        IOutputStream IRandomAccessStream.GetOutputStreamAt(ulong position)
        {
            throw new NotImplementedException();
        }


        void IRandomAccessStream.Seek(ulong position)
        {
            if (mPosition == position)
                return;

            mLogger.Log($"Seek from {mPosition} to {position}.");
            mPosition = position;
        }


        IRandomAccessStream IRandomAccessStream.CloneStream()
        {
            mLogger.Log("Clone stream.");
            return new HomeMadeStream(mFile, mPosition, mLength, mLogger);
        }


        IAsyncOperationWithProgress<IBuffer, uint> IInputStream.ReadAsync(IBuffer buffer, uint count, InputStreamOptions options)
        {
            return new AsyncOperationWithProgress<IBuffer>(async () =>
            {
                if (mStream is null)
                {
                    mStream = await mFile.OpenReadAsync();
                }

                mStream.Seek(mPosition);
                mPosition += count;
                return await mStream.ReadAsync(buffer, count, options);
            });
        }


        IAsyncOperationWithProgress<uint, uint> IOutputStream.WriteAsync(IBuffer buffer)
        {
            throw new NotImplementedException();
        }


        IAsyncOperation<bool> IOutputStream.FlushAsync()
        {
            throw new NotImplementedException();
        }

        #endregion
    }
}

[aklemets]

This worked for me. But I could not compile your program as-is, because you have a custom implementation of IAsyncOperationWithProgress, and I could not find a suitable replacement for it. But three was no need for this class by moving the code around a little bit:
Instead of doing "await mFile.OpenReadAsync()" inside the implementation of IInputStream.ReadAsync, I moved that call to the HomeMadeStream.Init() function. That allowed IInputStream.ReadAsync() to simply return the result from mStream.ReadAsync, without needing to wrap that result in a custom class.
I also modified your second constructor, the one that takes four input parameters, to take the stream as an input parameter, and invoke CloneStream() so that it gets it own private copy of the stream. This ensures that HomeMadeStream.CloneStream() works properly.

With those changes, I used your class to open a file. I used the implementation of IRandomAccessStreamWithContentType to set it as the source of a MediaPlayer, like this:
mediaPlayerElement.MediaPlayer.Source = MediaSource.CreateFromStream(stream, "video/mp4");

I tried playing some very large files and was able to seek very quickly to a position near the end of the file. I also set a breakpoint in the implementation of IRandomAccessStream.Seek in your class and it got hit in the debugger.

[mc0re]

Thanks for rearranging. I wasn't happy about the need of a custom IAsyncOperationWithProgress implementation, and although it's not too complicated, better without :-) I'll try that out.
On a related note - why does Microsoft not provide a standard wrapper for this interface?.. Or does it, and I just didn't find it?

[mc0re]

@aklemets Have you tried a corrupt video file? It seems that some exceptions are not caught properly in the MP4 decoder. And do you by any chance know which MP4 profiles are supported by the protected path player?

[mc0re]

Getting back to this issue, here is a playback log, where the player just hangs when playing a video file called "WPA`". I can reproduce it in my development environment (sometimes), but I don't know what to check for further.

Opening file WPA1.
Seek from 16 to 0.
Seek from 1024 to 0.
Seek from 262144 to 786432.
Seek from 1048576 to 773849088.
Seek from 774022194 to 773849088.
Seek from 774022194 to 786432.
Seek from 1048576 to 0.
Changing WPA1 state to "Buffering".
Changing WPA1 state to "Paused".
Changing WPA1 state to "Paused".
Changing WPA1 state to "Playing".
Seek from 129499136 to 128974848.
Seek from 163577856 to 163053568.
Seek from 243793920 to 243007488.
Seek from 248774656 to 247988224.
Seek from 276561920 to 275513344.
Seek from 341311488 to 340262912.
Seek from 401080320 to 399769600.
Seek from 426508288 to 425197568.
Seek from 481296384 to 479723520.
Seek from 533725184 to 531890176.
Seek from 538705920 to 534249472.
Changing WPA1 state to "Buffering".

And then it froze.