VTT Metadata Cue format is ambiguous; some metadata may be unintentionally presented to the user in a context outside HTML

[cookiecrook]

VTT Metadata Cue format is ambiguous; some metadata may be unintentionally presented to the user in a context outside HTML.

Consider clarifying that metadata cues SHOULD or MUST be formatted as one or more unambiguous patterns. JSON is the obvious one, to retain backwards compatibility with the JSON usage documented in the VTT spec, but there may be others.

WEBVTT

1
00:00:10.123 --> 00:00:15.432
{
  key: "value"
}

Background

§ 4.2.1. WebVTT metadata text (Normative) defines metadata text as:

WebVTT metadata text consists of any sequence of zero or more characters other than U+000A LINE FEED (LF) characters and U+000D CARRIAGE RETURN (CR) characters, each optionally separated from the next by a WebVTT line terminator. (In other words, any text that does not have two consecutive WebVTT line terminators and does not start or end with a WebVTT line terminator.)

WebVTT metadata text cues are only useful for scripted applications (e.g. using the metadata text track kind in a HTML text track).

§ 1.7. Metadata example (Informative) clarifies:

A WebVTT file can consist of time-aligned metadata.

Metadata can be any string and is often provided as a JSON construct.

Problem

"Metadata can be any string" results in a format that is ambiguous, and therefore may be presented to the user unintentionally.

In an HTML <video> element, this ambiguity is resolved by the author providing a kind="metadata" attribute on the text track.

But there isn't a logical place to duplicate this disambiguation in some other VTT contexts, including when they are embedded in some media container formats.

Proposed Solution

Consider clarifying that metadata cues SHOULD or MUST be formatted as one or more unambiguous patterns. JSON is the obvious one, to retain backwards compatibility with the JSON usage documented in the VTT spec, but there may be others.

Additional context for this change in the following issue.

• VTT Type Proposal for time-coded general-flash metadata  #512

[cookiecrook]

@eric-carlson @chrisn @gkatsev

[bdougherty]

FWIW, there is a somewhat common metadata format that uses URLs and not JSON for describing preview thumbnails when hovering over the progress bar:

WEBVTT

00:00:00.000 --> 00:00:00.979
https://cdn.example.com/thumbnails.jpg#xywh=0,0,284,160

00:00:00.979 --> 00:00:01.959
https://cdn.example.com/thumbnails.jpg#xywh=284,0,284,160

00:00:01.959 --> 00:00:02.938
https://cdn.example.com/thumbnails.jpg#xywh=568,0,284,160

[cookiecrook]

Any leading URI protocol regex seems easy enough to incorporate in addition to JSON.

[chrisn]

It seems difficult to introduce stricter constraints on the format without risking breaking someone's application. I note that the HTML spec includes a VTT example with a non-JSON custom metadata format. I have no idea how commonly used such formats may be used in practice, though. So if we do make a change to the VTT spec, that example would also need to be updated.

Have you considered other options for how schema information could be signaled? One idea: the VTT parsing algorithm requires implementations to ignore everything following WEBTT on the first line of the file. Conceivably, a backwards-compatible change could be to extended this line to include an optional schema identifier which would be ignored by today's implementations.

[gkatsev]

As @chrisn mentioned, introducing stricter constraints won't work, as it'll break folks. As you mentioned, the kind is available if the VTT file is included an HTML file which describes its kind, it's been briefly talked about having this be in the WebVTT file itself but so far nothing has materialized.

Some related discussions: #259, #346 and #485.

Those discussions involve WebVTT re-adding headers as an extension point to facilitate HLS's X-TIMESTAMP-MAP and webm's metadata. This header could be used to display the type of VTT file this is.

@chrisn's suggest of using the WEBVTT line is similar to using headers. Personally, I don't think it's good enough because we want to definitively remove ambiguity, so, using an optional extension for a core part of the spec seems fraught.

I think we probably want to have some other formal backwards-compatible signal that's in-file to indicate the kind of VTT file is being represented. Perhaps a new METADATA block, a la REGION and STYLE? It could start with a single property kind, that is equivalent to HTML Text Track kind.

My only concern with such an addition is the potential appetite for implementing WebVTT additions by folks.

[cookiecrook]

Mentioned in TTWG April 27 Minutes.

Resolved to keep this open and explore @gkatsev's idea:

Perhaps a new METADATA block, a la REGION and STYLE? It could start with a single property kind, that is equivalent to HTML Text Track kind.

[cookiecrook]

Resolved to keep this open and explore @gkatsev's idea:

Perhaps a new METADATA block, a la REGION and STYLE? It could start with a single property kind, that is equivalent to HTML Text Track kind.

@eric-carlson and I would like to make some progress on this idea, defining potential properties with #512 as the initial use case. Is anyone else interested in meeting about this in a breakout session at the Sept 2023 TPAC in Sevilla?

• VTT Type Proposal for time-coded general-flash metadata  #512

[cookiecrook]

Here's what we're thinking: A new ATTRIBUTES block. Renamed from @gkatsev's original proposal to avoid the redundancy in the case of METADATA kind: metadata.

Alongside Existing Usage (Subtitles, Captions, Descriptions)

Dialog-Only Subtitles

WEBVTT

ATTRIBUTES
kind: subtitles
srclang: es-mx
label: Español

NOTE
Standard subtitles (unlike CC or SDH captions) typically 
translate spoken dialog or signage, but not audible sounds 
effects like "dogs barking."

1
00:00:10.123 --> 00:00:15.432
¡Hola! ¿Qué tál?

Captions aka Subtitles for the Deaf and Hard-of-Hearing (including non-dialog sounds)

WEBVTT

ATTRIBUTES
kind: captions
srclang: es-mx
label: Español (SDH)

NOTE
Captions (SDH aka Subtitles for the Deaf and Hard-of-Hearing) 
typically include spoken dialog as well as important audible 
sounds such as "floor boards creak", "dogs barking", or in 
this case, "music".

1
00:00:10.123 --> 00:00:15.432
¡Hola! ¿Qué tál?

2
00:00:47.462 --> 00:01:04.028
[♫ música ♫]

Descriptions (e.g. audio descriptions and/or braille for the Blind/LowViz/DeafBlind communities)

WEBVTT

ATTRIBUTES
kind: descriptions
srclang: en-us
label: English (AD)

NOTE
VTT-based descriptions are meant to render as text-to-speech audio or braille,
for blind or deafblind audiences, not usually as visual captions on screen. 
As such, the option/label might be displayed in an audio menu or elsewhere. 

1
00:00:10.123 --> 00:00:15.432
A young girl tiptoes down a dark hallway.

Metadata Examples (with a new type attribute)

A potential accessible update to the Thumbnails usage @bdougherty mentioned above.

WEBVTT

ATTRIBUTES
kind: metadata
type: video-thumbnails

NOTE
In order to support accessibilty, the simple URL-only thumbnail 
format mentioned above should be updated to include "alt" text for 
each. In the potential format below, I've written that as a JSON 
block containing alt strings for multiple supported languages.

00:00:01.959 --> 00:00:02.938
{
	"src": "https://cdn.example.com/thumbnails.jpg#xywh=0,0,284,160",
	"alt": {
		"en-us": "Miguel crosses the marigold bridge to the land of the dead.",
		"es-mx": "Miguel cruza el puente marigold hacia la tierra de los muertos."
	}
}

A Proposal for the Flashing Lights Avoidance Use Case from Issue #512

WEBVTT

ATTRIBUTES
kind: metadata
type: video-flash-avoidance

NOTE
Spec for "video-flash-avoidance" (or "video-flash", "strobing", etc.) type would define 
usage as a JSON block with one required and two optional key/value pairs:
- integer "intensity": 0-100
- opt token "flash-type": ["general-flash" (default) | "red-flash" | "spatial-pattern"]
- opt token "algorithm": ["undefined" (default) | "harding" | "apple-vfr" (bikeshed, algo needs name)]

NOTE
The v1 Apple open-sourced algorithm (bikeshed name "apple-vfr" for "video 
flashing reduction") only detects "general-flash" patterns (not yet 
"red-flash" or "spacial-pattern"), but we think it performs better than the 
de facto Harding test in those instances of "general-flash". See below for 
example where "harding" would still need to be used to denote the 
"spatial-pattern" cue that the open-sourced algorithm doesn't yet account for.
Cite: https://developer.apple.com/accessibility/#dim-flashing-lights

1
00:00:10.123 --> 00:00:15.432
{
  "intensity": "75",
  "flash-type": "general-flash",
  "algorithm": "apple-vfr"
}

2
00:00:47.462 --> 00:01:04.028
{
  "intensity": "100",
  "flash-type": "spatial-pattern",
  "algorithm": "harding"
}

[cookiecrook]

Is anyone else interested in meeting about this in a breakout session at the Sept 2023 TPAC in Sevilla?

Better yet, time on the standard TTWG meeting schedule for TPAC.

Note: I looked for a F2FCandidate or TPACCandidate keyword. Not sure how you're tracking that list.

[chrisn]

Is anyone else interested in meeting about this in a breakout session at the Sept 2023 TPAC in Sevilla?

I'm interested, yes.

[cookiecrook]

@nigelmegitt et al, can we get time on the TTWG schedule at TPAC? I think that's a better forum than a breakout session. Also would be good to coordinate with @jasonjgw and others interested in MAUR Issue 2