Exposing decode errors / SW fallback as an event

[henbos]

In WebCodecs it appears an event may be added, which may be considered less privacy harmful than powerEfficient[Encoder/Decoder] in stats.

Can we add similar events to RTCRtpSender/RTCRtpReceiver? @aboba

[youennf]

My reading of the conclusion is that for no-preference there will be no surfacing.
I do not see where adding an event is mentioned in the conclusion.

[henbos]

Hmh we don't have a preference API (yet), there is an assumption that HW is preferred but we may want a prefer-software/prefer-hardware option as well

[aboba]

See: w3c/webcodecs#669

hardwareAcceleration = no-preference hides errors that result in software fallback because it is interpreted to mean "I don't care if you use hardware or software". However, if hardwareAcceleration is set to prefer-hardware an error can be surfaced.

As noted in the MEDIA WG minutes, the suggestion is for EncodingError to indicate an issue with the data provided to the Encoder/Decoder, and something OperationError to indicate a resource issue (e.g. hardware encoder or decoder resource longer available).

[aboba]

Related: #127

[tidoust]

It was noted during the call that there may be value in looking up at how WebGPU reports device errors. The WebGPU explainer has a section on errors that describes the different types of errors that WebGPU may report, including fatal ones and fallible ones, the latter being captured in a stack. For fatal errors, GPU devices come with a lost promise, which gets resolved when the device is lost.

[henbos]

@xingri FYI other work in this area

[henbos]

Can we include the following errors?

• HW decoder could not be instantiated because the config is not supported (example: too many temporal or spatial layers or too big resolution).
• HW decoder is used but fails during decode.

As long as we don't fire any events due to "HW is already in use"? In that case you may still get SW but you can't correlate "HW availability" with cross-site usage?

[henbos]

Sounds like we'll need an event sooner or later, both fatal and non-fatal errors will be a thing... @aboba @Orphis

[xingri]

@henbos, @aboba, @Orphis Is there a WIP PR? Otherwise, can I start working on it?

[henbos]

@xingri You are very welcome to provide a PR. I suggest a custom event exposing what type of error happened as an enum (RTCTrackEvent is an example of an event interface adding custom attributes to the Event).

I suggest adding the event handler to RTCRtpReceiver (and if we want to expose encode errors we would add a similar event to RTCRtpSender in a follow up). The question is how to expose the necessary information for the app to take action.

[henbos]

I would prefer ondecodererror over onsoftwarefallback because then we could use the same event handler for both fatal and non fatal errors, WDYT? For example if no backup decoder is available and the error is permanent that would fire even if SW fallback was not possible

[aboba]

Latest discussion in WebCodecs has been about the distinction between types of decoder errors. Hardware decoders are more strict, so they can error due to inputs that software decoders would accept. It is useful to be able to distinguish data errors from situations in which hardware resources become available (e.g. because another application is using them or because the GPU crashed).

See: https://www.w3.org/2023/04/11-mediawg-minutes.html#t01

[dontcallmedom-bot]

This issue had an associated resolution in WebRTC WG 2023-04-18 – (Issue 146: Exposing decode errors / SW fallback as an Event (Bernard)):

RESOLUTION: Waiting for Media Working Group.

MEDIA WG meeting minutes: https://www.w3.org/2023/04/11-mediawg-minutes.html#t01

[xingri]

@aboba Could you please share working group discussion if there is an update?
Also can I start working on creating PR for this?

[aboba]

Related: w3c/webcodecs#669

Eugene Zemstov is currently looking at how to expose encode/decode errors in WebCodecs. This will include differentiating between types of errors (e.g. data errors versus hardware resource availability).

[xingri]

This issue had an associated resolution in WebRTC WG 2023-04-18 – (Issue 146: Exposing decode errors / SW fallback as an Event (Bernard)):

RESOLUTION: Waiting for Media Working Group.

MEDIA WG meeting minutes: https://www.w3.org/2023/04/11-mediawg-minutes.html#t01

@aboba thank you for the update. Seems like the link is about the meeting before WebRTC montly WG meeting. I thought there might be additional discussion after WebRTC WG meeting(04-18)

[xingri]

Related: w3c/webcodecs#669

Eugene Zemstov is currently looking at how to expose encode/decode errors in WebCodecs. This will include differentiating between types of errors (e.g. data errors versus hardware resource availability).

Thank you for the update once again. Will Eugene's work be applicable for the general WebRTC cases beyond WebCodec?

[aboba]

We are hoping that the approach can be used for both WebRTC and WebCodecs. The big question is whether we can differentiate errors to provide applications with the information to figure out what to do next.

[xingri]

We are hoping that the approach can be used for both WebRTC and WebCodecs. The big question is whether we can differentiate errors to provide applications with the information to figure out what to do next.

@aboba Thank you again for the prompt response. Will keep watching on it. Please let me know if there are any thing to contribute from my end.

[dontcallmedom-bot]

This issue was mentioned in WEBRTCWG-2023-07-18 (Page 17)

[fippo]

An additional consideration: sometimes the UA may switch from hardware encoders to software encoders intentionally, e.g. because the software encoder provides better quality for small resolutions (see e.g. here)
Should this be exposed via the event as well?

[aboba]

Would be useful to cover this in the joint meeting slides.

In WebRTC, it might make sense to indicate that software failover occurred, but not as the result of an error (not a decode error or a resource exhaustion problem).

In WebCodecs it is not clear if this should be exposed. If hwAcceleration = "no-preference", is set, software failover occurs without an error or other notification. If hwAcceleration = "prefer-hardware" then there is no automatic software failover, just an error to tell you why (e.g. resource unavailability vs. decode error). Not sure if an error should be surfaced just because software quality is better.

[dontcallmedom-bot]

This issue was mentioned in WEBRTC-MEDIA-2023-09-15 (Page 25)

[fippo]

From my recollection the path forward here is
1/ do it on the sender/receiver (no opinions expressed during the meeting but it seems more consistent with discussions earlier in the week about events on individual objects)
2/ preference for proposal B but with event names a bit less "RTC*" focused (same for rid for which spatialLayer might be the better term) with an enum for errorCode.

I can take a stab for October 17th

[xingri]

Adding previous discussion security concern regarding on SW Fallback. I got the positive feedback on event style approach.

• The HW exposure check does not solve Cloud Gaming use cases webrtc-stats#730 (comment)

[dontcallmedom-bot]

This issue was discussed in WebRTC November 2023 meeting – 21 November 2023 (Issue 146 Exposing decode errors/SW fallback as an event 🎞︎)