Programatically disabling hardware encoding/decoding in WebRTC

[fippo]

Request for Mozilla Position on an Emerging Web Specification

• Specification Title: (part of webrtc-extensions)
• Specification or proposal URL: https://w3c.github.io/webrtc-extensions/#disable-hardware
• Caniuse.com URL (optional): n/a
• Bugzilla URL (optional): https://bugzilla.mozilla.org/show_bug.cgi?id=1766311 mentions the spec issue so I believe HW encoder being off is the default right now
• Mozillians who can provide input (optional): @jan-ivar

Other information

See also w3c/webrtc-extensions#98, spec PR was in w3c/webrtc-extensions#128

[ekr]

I certainly understand why one would want to disable hardware encoding, but I'm skeptical that the right API answer is to have the JS do it rather than the browser. This seems like it is sure to create situations where the JS is turning off the HW encoder long after it would be fine to have it on because whatever bug it is trying to work around has been fixed.

[fippo]

That is indeed a risk and the usage of such an API should be closely monitored since prolonged use suggests there is an underlying bug that needs to be fixed. See slide 68+ from TPAC.

I do hope that in general hardware acceleration is desirable enough that folks won't blindly disable it.

Browser release cycles make it hard to react quickly. If you look at this Chrome issue you can imagine what two weeks of basically complete breakage mean for a product.

[ekr]

It seems to me the fix there is for browsers not to introduce that kind of bustage.

[fippo]

See my list-of-doom which shows we are not winning that battle. Sadly CI doesn't help much since this often boils down to a combination of hardware and driver versions.

[ekr]

Browsers are perfectly free to disable hardware encoding entirely.

[fippo]

True but that will make users use a different browser. Also hey, this used to be one of @jesup's favorite topics!

[fabricedesre]

As someone working with various chipset vendors, I agree with @ekr that this kind of api should not be exposed to web content. It is usually set on a per-chipset basis (either through configuration or some autodetection mechanism) and there is not really a meaningful choice to make at the content or end user level.

[nils-ohlmeier]

I believe the topic of hardware acceleration boils down to maintaining big white and black lists of chipset, firmware, driver combinations.
Neither have the services access to all of this through JS to maintain such lists. Nor do the browser vendors, to the best of my knowledge, have the resources in their CI systems to regularly check all the possible combinations and maintain good lists.

I can fully understand that a service would want to have the option to turn off all hardware acceleration to avoid having their support spending endless time trying to figure out what is causing issues.
But I'm not convinced that doing this on a per sender/receiver level is right level of granularity for that.

[HuaChunbo]

My concern is that this kind of high level JS API will hide the benefits and also potential issues of hw codecs deep inside WebRTC. Currently browsers have workaround list or blacklist to do the hardware/drivers info based detection and avoidance.

[slightlyoff]

Presumably there will still be enough traffic to HW-accelerated content to find hw/driver issues; a toggle like this makes good sense to me in the interest of keeping services working while those sorts of concerns can be worked out, particularly given the long lead times for some vendors in delivering driver fixes.

[fippo]

@nils-ohlmeier this is a static method so not per-sender (mostly in order not to create another bit of information by comparing the available codecs with/without HW directly) but equivalent in scope to the currently available chrome://flags for disabling HW which do allow the user to unbust their device (@fabricedesre this is the kind of choice a user can make, possibly guided by a support team)
Using this API in a service-wide manner "disable HW on all cards from this particular vendor" is possible (with methods for detecting the vendor outside the scope of WebRTC) but much less fine-grained than solving the issue(s) with the more detailed information that is not available to JS.

@ekr WebCodecs allows a much more fine-grained control already here. WebRTC has traditionally been much more "highlevel" as an API (as you know).
The status quo in Chromium is CBP not being accelerated and other hw-accelerated profiles emitting a ton of keyframes and/or tens of keyframes during the ramp-up (and I am quite surprised that no bugs have been filed about that).
That is (just like disabling HW by default as done in Firefox) not an ideal state I would like to get out of but it needs to be done in a manner that does allow a service to run A/B experiments guiding the changes in the wild. Browser-wide metrics have proven unsuitable for this kind of things.

[ekr]

@fippo I understand what you are trying to achieve, but I believe the ecosystem effects of allowing sites to turn off acceleration will be quite bad and lead to excessive disablement. In particular, I think it's quite likely that they will create a differentially negative impact on smaller browsers because sites will have less incentive to re-test and re-enable hardware. Therefore, I am not in favor of this.

[fippo]

@ekr: will note Mozilla as "not in favor".

Please note that other ways to resolve this such as origin trials (or reverse origin trials) are even more unfair towards smaller browsers.

[jan-ivar]

The Media team might be OK to move forward with a slightly different surface that links to https://w3c.github.io/webcodecs/#hardware-acceleration (which has a page-long note about when the UA might override webpage preference).

partial interface RTCRtpReceiver {
  HardwareAcceleration hardwareAcceleration = "no-preference";  
}

We think this would resolve some of the concerns raised above, specifically the UAs ability to override website preference in situations like after a hardware codec fix.

Would this work for everyone? With such an API I think we would be able to propose a positive position.

[jan-ivar]

I've filed w3c/webrtc-extensions#229 to propose the API above.