Timekeeping broken/Dropped frames not reported

[BlueprintBen]

Bug Description
GetLastFrame/GetLastFrameUploaded doesn't increase at the rate predicted by CurrentFramerate, and GetCaptureDroppedFrames doesn't report any dropped frames during testing despite CaptureFPS dropping below CurrentFramerate. This significantly limits our ability to accurately perform timekeeping on inputs.

My Setup:

• Unity version: 2020.3.12 (LTS)
• AVPro Live Camera version: 2.9.1
• Operating system version: Windows 10 21H1
• Hardware/webcam model: Logitech C920 (and others)
• Video mode (resolution, frame-rate, etc): 1920x1080@30fps (and others)

To Reproduce

1. Create a Unity scene with an AVProLiveCamera component configured to start using a test webcam on awake.
2. Add the attached TimingTest.cs component to the same GameObject (ensure correct file extension)
3. Run the scene and wait 10 seconds for TimingTest to finish (it will stop the application itself)
4. Review the console/log to see the unaccounted-for discrepancy between time elapsed and (frames seen / framerate).

Example Results

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[BlueprintBen]

Any progress on this issue? Not being able to keep track of the 'real' index of a frame drastically reduces the utility of this plugin for us unfortunately.

[kahnivore]

Hi @BlueprintBen

Sorry for the slow response on this.

Basically we need to investigate this further to see if this is an internal DirectShow issue or something to do with our plugin implementation, the latter which we could then look into improving.

We will look into it ASAP just haven't had anyone available to do so.

Thank you for your patience.

[BlueprintBen]

I've tested as far back as 2.7.5 without any change, so now imagining that this is more structural. Is there a device timestamp for each frame that could at least be acquired to calculate an approximate 'realtime' frame index? Otherwise we're stuck using when we see the new frame is available, which introduces a bunch of variability/latency.

[kahnivore]

We could probably expose a device timestamp quite easily, if this would work for you?

The underlying issue being that even if we expose the timestamp, it's still tricky to know if the frame you have is the frame you have the timestamp for - addressing this issue requires a more fundamental fix that we need to spend some proper time on.

Let us know if simply exposing the timestamp could help you.

[BlueprintBen]

Sorry, just a call like Plugin.GetDeviceTimestamp() isn't going to help, since it is really more data about the specific frame being accessed that I need. If the timestamp was for the active frame, at least I could calculate if it's 33ms higher than the last one (= no drops) or more (= drops), for example and make up an index from that.

Can you explain what the GetCaptureDroppedFrames() function does report since I might still be missing some aspect of the timing.

[kahnivore]

The timestamp would be for the active frame. We can add this in for you.

[BlueprintBen]

OK thanks! Definitely a leg up on the timing in cases where we're trying to match things up precisely. Let know know if you want me to try out a build.

[kahnivore]

We had a look into this and it doesn't seem as straightforward as anticipated.

We'll have to see if we have any luck with this in the future, otherwise it might be something for our next major version.

[BlueprintBen]

OK well thanks for investigating. Glad to have at least surfaced the value in having this type of information available.

[BlueprintBen]

Have come back to this and am at a bit of a loss as to why it's not feasible to implement passing along the timestamp that each DirectShow MediaSample (as long as not preview pin) has per the DirectShow docs. Feels like a core component of DirectShow and functionality that will really allow this plugin to sync with Unity better.

That page also mentions accessing Media Times which seem even more appropriate for my purposes. Anyway would love any more insight into the blockers if possible.

Thanks!