Mark sampleRate, sampleSize, latency as features at risk

[dontcallmedom]

Looking at the recently updated implementation report based on Web Platform Tests, the following constraints have only one implementation (in Chromium):

• resizeMode ((FF #1433480
• sampleRate (FF #1388586
• sampleSize
• latency

The feedback I've gotten from potential implementors don't seem to indicate implementing these would be high-priority. As a result, I would suggest making them at risk.

Presumably, if/when the spec goes to Rec, we could keep them as "candidate additions" so that they can get easily integrated once other implementations surface.

[jan-ivar]

We intend to implement resizeMode FWIW. @padenot what do you think about the others on that list?

[padenot]

• sampleRate can make some sense if you want to filter out low-quality devices (e.g. bluetooth handsfree), or understand some things about the local devices (e.g. if you have very high sample-rate), but it's pretty niche. It can be used for fingerprinting.
• sampleSize can be useful if you want to know very advanced properties of the device, but generally it's not particularly useful. It can also be used for fingerprinting. It's useless without knowing how the bits are stored: it's possible to have a 32-bits DAC, but what's important to know is if those samples are float32 (because the OS is exposing f32 audio samples) or int32 (because you have a real 32-bits dac and this is getting through somehow to script -- only webcodecs supports dealing with s32 audio samples afaik), and this is not possible. I'd say it's not very useful in the current state of the web platform. Similarly, this can be used for fingerprinting.
• latency can typically and generally only be trusted after measuring it while the device is being used. We're adding good APIs to get this number. Again, this can be used for fingerprinting.

[bradisbell]

sampleRate is required in a lot of my usage of the API, where collaborators on a common broadcast stream need to match sample rate. For example, one radio show host may be streaming at 48 kHz. The next radio show host for hour #2 may have a default sample rate of 44.1 kHz, but we can get a 48 kHz stream for them by requesting it with constraints. (Of course this is sometimes transparently resampled, but without it we're forced to resample in JS which isn't efficient and adds extra latency.)

And, a link for reference... fingerprinting concerns around sample rate were previously discussed over on the Web Audio API: WebAudio/web-audio-api#1457

Regarding sampleSize, I'd agree completely with @padenot about problems with the current API. We have actually been trying to move to float32 audio interfaces, but there are compatibility issues from the OS, to the audio API the user agent uses, all the way through the user agent. Even once we get those issues ironed out, there's no way to know if it was 32-bit integer samples, or one of the float32 implementations.

[padenot]

There are no guarantees that using an input-device with a sample-rate of x will result in an audio stream of x. Browsers typically resample all audio input to the sample-rate of the object that does the audio routing, or directly open at a different sample-rate. The resampling can be done by the OS or the implementation.

What's important here is e.g. to set the sample-rate of the AudioContext to the sample-rate you want, and this is useful and implemented.

[jan-ivar]

Discussed at last interim where we agreed to mark sampleRate, sampleSize, latency as features at risk.

[youennf]

I thought the idea was to move these to the extension spec?
resizeMode is apparently difficult to move to the extension spec but quickly looking at it, I do not think it is more difficult than other constraints.

[guidou]

Yes, the decision was to move them to the extension spec.
The argument for resizeMode staying in the main spec is that it is much more used than the others, it has a clear and well understood use case, it influences the settings selection algorithm significantly since it is correlated with width, height, and aspectRatio; and Firefox was planning to add support for it (https://bugzilla.mozilla.org/show_bug.cgi?id=1433480).

[youennf]

The argument for resizeMode staying in the main spec is that it is much more used than the others

The main criteria is whether there is a concrete short-enough-term plan for another UA to implement this support.
It would make no sense to move it to extension spec since we would move it back to main spec a few months later.

From what I understood, there is no such plan though.
Keeping resizeMode in the main spec seems to only have downsides then: delay moving the whole spec to CR, no impact on implementation catch up.

[guidou]

I'm fine with moving resizeMode to the extensions spec and moving it back to main once Mozilla implements https://bugzilla.mozilla.org/show_bug.cgi?id=1433480.