surfaceSwitching seems under-specified

[jan-ivar]

surfaceSwitching "signals whether the application would like the user agent to offer the user an option to dynamically switch the captured display surface."

This is alluding to a feature that isn't specified anywhere, and it's unclear how it would manifest and be JS observable.

In particular, all MediaStreamTrack are created with a [[Source]] internal slot that never changes, which (even though it's not exposed directly), governs invariants like: all clones have the same source. It is unclear whether UAs who implement "switching" have done so by "switching the source of the source", so to speak, to maintain this invariant.

We should probably whether there are observable aspects: does the track.label change? There might be capture-controller and capturee-side impacts as well (to remote actions, identity, even cropTo).

Lastly, we might also want to limit switching between different displaySurface types, unless we have compelling use cases for that. Switching between sources with different levels of elevated permissions, to mitigate users sharing a safer source, and be switched to a less-safe source.

[youennf]

it's unclear how it would manifest and be JS observable.

I proposed some time ago to fire https://w3c.github.io/mediacapture-extensions/#exposing-change-of-mediastreamtrack-configuration in case of surface switching. Specifying the behavior would indeed be good.
Note that the switching is not necessarily done from UA but can be done at OS level.
It seems best to add a hook in the spec that the UA can call when switching happens, similarly to https://w3c.github.io/mediacapture-main/#ends-nostop for instance.

we might also want to limit switching between different displaySurface types, unless we have compelling use cases for that.

This is UA/OS territory. I don't think the spec can mandate anything except calling again that some source types have more elevated permissions and these permissions should be enforced when doing switching.

[eladalon1983]

we might also want to limit switching between different displaySurface types, unless we have compelling use cases for that.

Users want the flexibility to quickly change what they're sharing. This is often why they share their entire screen. The more flexibility we give them to change between windows-tabs, the less often they'll score own goals by sharing the entire screen. I think that's very compelling.

Also, privacy-wise, switching between two different native apps does not seem worse than switching between a native app and a tab inside a browser, which is itself a native app. If we disallow that, users will switch to sharing the entire browser window, which is much worse.

Switching between sources with different levels of elevated permissions, to mitigate users sharing a safer source, and be switched to a less-safe source.

Should it be argued that we should only allow switching to a more private source, I'd argue that users will then start by sharing their entire screen and then quickly clamping down, so that they may retain the flexibility to share anything.

I proposed some time ago to fire https://w3c.github.io/mediacapture-extensions/#exposing-change-of-mediastreamtrack-configuration in case of surface switching. Specifying the behavior would indeed be good.

That seems reasonable.

We should probably whether there are observable aspects: does the track.label change?

Empirically, I see that Firefox currently sets as the label the title of the native window at the time the picker was displayed. That is, even if the window title changes before the user completes their selection, the old title will be set as the label. I don't think this benefits anyone.

But the above is probably a bug. More pertinent is that the label does not change if the title changes after capture starts. I don't see how that benefits users/devs. To be useful, the label must adapt.

There might be capture-controller and capturee-side impacts as well

Capture Handle already handles this by firing an event in the capturing app.