What to do with per-key Baseline statuses

[LeoMcA]

tl;dr I don't think per-key Baseline statuses are yet correct enough to use to power the Baseline banner at the top of MDN pages, and as a first step will merely use them as a replacement for the blocklist we currently manually maintain

Background

A short bit of background for those not familiar with the general issue (which can be skipped by those that are):
Baseline statuses on MDN are currently calculated on a feature level (where something like "Async clipboard" is a feature), but this leads to some oddities where particular parts of a feature which are Baseline are displayed as not (or not widely available) because the feature as a whole isn't (despite it being entirely possible to e.g. use the writeText method of async clipboard without using the rest of the feature). So instead of displaying the overall status of the feature a BCD-key/page belongs to, we've received guidance from upstream to display per-BCD-key statuses, calculated with compute-baseline, at the top of pages.

The experiment

Before implementing this, I had some suspicions that there would still be some edge cases, so I wanted to investigate those. To do this, for every page on MDN which has a BCD key which is part of a web-features feature, I compared the compute-baseline BCD key status to the web-features feature status. Here's the results:

Pages with multiple BCD keys: 89 (I'll be auditing these soon and trying to map them directly to features, so let's ignore these for now)

Pages where the feature and key status are the same: 2307

This is good, and shows that for the vast majority of pages, whether we use the feature status or the per-BCD key status, we're ok.

Demotions

Pages where the feature is a higher status than the key (i.e. the page has been "demoted"): 32

• High to low: 8
• High to "not": 21
• Low to "not": 3

These demotions are a bit weird, I'm not sure why these keys are part of a feature when they don't contribute to the status. @ddbeck or @foolip perhaps you can shed some light? These 32 pages present a bit of a bug, as currently we show a Baseline banner on them when they're either only-recently usable, or not usable at all. So we need to do something here.

Promotions

Pages where the feature is a lower status than the key (i.e. the page has been "promoted"): 65

• Low to high: 42
• "Not" to high: 13
• "Not" to low: 10

Auditing these by page type, the vast majority seemed ok, and were "leaf" pages (e.g. interface properties or methods) which is exactly what we wanted to promote with per-key statuses (like the earlier example of writeText).

However, there were some page types which made me cautious, namely web-api-interface pages, and subpages of Navigator, since they tend to document the methods and properties on the interface as much as the interface itself - and those methods and properties may have a lower status. Let's look at specifically what these pages are:

• Interface pages
  • Low to high
    • /en-US/docs/Web/API/ByteLengthQueuingStrategy
      • seems fine, none of its methods/properties make the feature low
    • /en-US/docs/Web/API/Clipboard
      • not fine, some of its methods/properties are what make the feature low
    • /en-US/docs/Web/API/CountQueuingStrategy
      • seems fine, none of its methods/properties make the feature low
    • /en-US/docs/Web/API/InputEvent
      • maybe ok? one of its methods/properties make the feature low
    • /en-US/docs/Web/API/ReadableStream
      • maybe ok? two of its methods/properties make the feature low
  • Not to high
    • /en-US/docs/Web/API/HTMLDataListElement
      • not sure, is there value in knowing the js interface for an element works cross-browser if the element itself doesn't
  • Not to low
    • /en-US/docs/Web/API/RTCEncodedAudioFrame
      • seems fine, none of its methods/properties make the feature "not", though its constructor does
    • /en-US/docs/Web/API/RTCEncodedVideoFrame
      • seems fine, none of its methods/properties make the feature "not", though its constructor does
• Navigator pages
  • Low to high:
    • /en-US/docs/Web/API/Navigator/clipboard
      • Not ok, documents a method readText() which is low
  • Not to high:
    • /en-US/docs/Web/API/Navigator/mediaSession
      • Potentially ok

There were also some pages outside these pages types where the promotion seemed a little wrong, for instance the overflow-related pages. Here, obviously, overflow itself is Baseline high, but the clip functionality takes the feature into Baseline low, and it's difficult to decide what to display at the top of page: without otherwise warning that the clip functionality is Baseline low (with an inline status, for instance), I think it's misleading to overall call the page Baseline high.

The data

I've dug into the data above, but you might want to yourself. I present some that and some incredibly messy and inefficient code to regenerate the files if you so desire here: LeoMcA@74587d0

If you just want the data I used:

• https://github.com/LeoMcA/yari/blob/74587d07f516153701952e6a0bcf42cc38e534f2/baseline.count.json
• https://github.com/LeoMcA/yari/blob/74587d07f516153701952e6a0bcf42cc38e534f2/baseline.list.json

If you want to generate it yourself:

• Check out the branch
• yarn && yarn build --nohtml --locale en-us
• Wait about 5 minutes

To explain the files:

• Both baseline.count.json and baseline.list.json have roughly the same layout, until their leaves
• In terms of the keys:
  • multiple_keys refers to pages with multiple bcd keys, we don't do any further calculation on these, and simply break them down by page type
  • previously_filtered refers to pages in the current manually-maintained blocklist, we break these down by per-key status, and also include them again in the keys below
  • feature_{high|low|not} groups the pages by their web-features feature Baseline status, we break these down on their per-key status
  • {high|low|not} groups the pages by their compute-baseline per-key Baseline status, we break these down by page-type
  • all groups all the page types within a status, for easy counting
  • * (any other key) is the page type
• baseline.count.json then simply gives counts for the pages
• baseline.list.json lists our their urls

Conclusion

For now, I'm not comfortable simply showing the compute-baseline status for a key at the top of pages.

I want to experiment with traversing down the BCD tree for a key, and taking the lowest status for the result of compute-baseline for each of those - I reckon it may produce what we want.

But for now, given we have some clearly-wrong statuses (especially the "demotions" above), I'll implement conditional hiding of the banner if the web-feature status and compute-baseline status disagree, so we no longer need to manually curate the blocklist.