Background Blur: Unprocessed video should be mandatory to support

[alvestrand]

In the context of https://w3c.github.io/mediacapture-extensions/#exposing-mediastreamtrack-source-background-blur-support and similar mechanisms for platform effects on video tracks, concern has been raised that this type of functionality can cause deep confusion for users when the same class of effect can be applied both in an application (conferencing systems, for instance) and in the platform - users won't know where to turn it off, and double blur may cause effects that were not as intended.

For other types of effects (face touchups, background replacement, emoji reactions), the cost of being unable to prevent confusing interactions between effects is even bigger.

This concern would be alleviated if all such capabilities were defined to be "MUST support constraint=false" - that is, if the application could always turn them off and get an unprocessed video feed.

[huibk]

Since the introduction of platform effects, video conferencing applications on the web have faced increased negative user feedback such as:

• Users think the web app is broken because they are confused where and how background blur is turned on and off
• Users are noticing distorted video, such as bad cropping/segmentation or added delay (due to double processing)
• Users are noticing an increase in battery consumption (due to double processing)
• Video effects being used inappropriately or users are confused why certain effects are present

Since it's not really feasible to align what the app does and the platform does, there will always be inconsistency, conflicts and user confusion between the two. Web apps should be able to build experiences that go beyond what specific platforms are offering and without having to burden users with navigating two sets of controls.

[dontcallmedom-bot]

This issue was discussed in WebRTC November 2023 meeting – 21 November 2023 (Issue #121: Background Blur: Unprocessed video should be mandatory 🎞︎)

[alvestrand]

Discussion about this item (which branched into the general "three thumbs up" problem) seemed to be generally positive towards "it should be possible to get unprocessed", with the main issue being raised being that on some OSes, there are currently no controls exposed to that would allow this.

A suggested resolution was to make it a SHOULD, with a note saying that only in the case of UAs operating on platforms where those controls were not available, the lack of ability to get unprocessed data was acceptable.

[youennf]

only in the case of UAs operating on platforms where those controls were not available, the lack of ability to get unprocessed data was acceptable.

I am not sure we should separate OS and UA here.
Some OSes may decide that, if user selected background blur in OS UX, no application will be allowed to opt-out without user interacting with OS UX.
A UA may decide to apply the same constraint to any web page it runs, even if OS is not imposing such restriction.

The spec offers the flexibility to all those models, which is good, especially since this is quite new territory that needs to mature.

[alvestrand]

UA is the one we're writing rules for, and where we assume that the UA implementors will be listening to what we write.
If we think that it's OK for an UA to refuse to allow an application to prevent background fireworks, floating thumbs-up symbols or background blur, we should say that. But that was not the sense I got from the meeting.

[alvestrand]

an interesting question here is if we should mention (or permit) the possibility of applyConstraints() causing user interaction.

[youennf]

If we think that it's OK for an UA to refuse to allow an application to prevent background fireworks

Such a UA should have good reasons for not allowing this.
One reason that was mentioned during the call was privacy. This is a valid concern, at least for some of these effects.
User may for instance have agreed to permit camera access with the assumption that only their face or part of the captured data would be shared to the web page.

an interesting question here is if we should mention (or permit) the possibility of applyConstraints() causing user interaction.

I think it is allowed, I have no objection mentioning it.

[eladalon1983]

I wonder if we branch of the discussion of effects which clearly serve no privacy purpose, and find some way to mandate that the user agent SHOULD/MUST at least allow Web applications to avoid those...?

Or at least avoid such reactions being inserted heuristically - through analysis of the user's video - while still allowing users to force them in through explicit interaction with the OS or hardware.