Frames dropping with action

[xplorer256]

There is a strange behaviour with resizing windows on visionOS.

A new window starts with a high frame rate as expected. Even videos and TV are transmitted with high frame rate at initialisation. As soon you try to resize a window on visionOS, the frame rate suddenly drops down to 1 per second and this problem remains whatever you try. New (possibly additional) windows are created with normal frame rate again. Until you resize it.

The 1 second frame update makes it somehow useless. But moving the window without resizing does work. Only resizing does cause the issue.

So it seems there is a problem with some of the code at TCP transport level or encoder level or the screen capture. But I think it is something on the visionOS side, that causes a drop of frames and then refreshes with merely 1 frame per second. Resizing a window probably interrupts something inside the signal flow permanently.

This is quite an important issue and I do not think, it is on the macOS side, as newly created windows always are running fine again.

If you are interested, I could make a video.

[saagarjha]

A video would be quite helpful :)

[xplorer256]

Can you not replicate this issue? Is it specific to me?
I do not think that it is specific, as i downloaded several versions (including the latest release) and it always has the same behaviour.

But I will try to capture a video.

[xplorer256]

Resizing.Dropout.Issue.mp4

Here's the video, capturing a TV app stream on the mac. Despite that there is a temporary audio dropout om the source at the beginning, there is this permanent issue after resizing windows. It is easily reconstructable each time. I used the latest commit and compiled it for this demonstration.

This happens with any kind of window, also with usual (more static) app windows without requiring such high video frame update rates. But such app windows are useless with a frame rate of 1 second after resizing nevertheless.

So this issue is not quite ignorable.

ps: My WLAN connection is fast enough to stream 4K videos and doing HQ local screen sharing between macs without any issue. I have a setup of 2 external monitors connected to my MacBook Pro with M2 Pro chip. Ensemble is able to capture the windows on all monitors.

[saagarjha]

It doesn't do that for me, no. I do know of some cases where the video degrades and I see symptoms like the ones you are describing, but I haven't been able to them to window resizing, so I am very curious to see if I am missing something.

[xplorer256]

ps: I think it is an out of sync issue, that is not corrected as soon as possible. Therefore it consists permanently . Sometimes it gets in sync after long running, but not always.

[saagarjha]

Sorry, it seems like you posted that while I was replying! Your video seems like it is of the visionOS simulator. Can you reproduce it on actual hardware?

[xplorer256]

Unfortunately no. Maybe there are some users with a real device here?

[saagarjha]

If you are on the simulator, you are almost certainly running into bugs in the Metal rendering path that I implemented as a stopgap there for testing. You can see the implementation of this here: https://github.com/saagarjha/Ensemble/blob/main/visionOS/ImageBufferView.swift. In the simulator the CoreAnimation path does not work at all, and software rendering is far too slow (set mask = true to try this if you want). The Metal renderer does but I haven't had the time to polish it so it has bugs like these, and I have it disabled on real hardware. I do at some point see switching entirely to Metal but that would be after I fix this and I just haven't gotten around to it.

[xplorer256]

I do not quite understand, why this is exactly ONE SECOND frame rate. So I thought there must be anywhere something that reduces the frame rate to exactly 1 second. Could it be the TCP transport ? Or the video decoder?

[saagarjha]

https://developer.apple.com/documentation/quartzcore/cametallayer/2887086-allowsnextdrawabletimeout

[xplorer256]

I will look somewhat closer at the ImageBufferView now...

[xplorer256]

Made the test!!!

Did set :
var mask: Bool = true
static var useMetal: Bool = false

And what??? IT WORKS !
So there must be a bug in the Metal renderer.
metalLayer.allowsNextDrawableTimeout does not solve the problem.

[xplorer256]

.. another possible solution: prevent the redrawin g, if a resizing action is active.

[saagarjha]

I'm going to lump this into #15