Explicitly define supported MIME types in Async Clipboard Spec

[gked]

@snianu has started working on pickling formats implementation and realized that there is no defined mime types for async clipboard apis.
This has also been mentioned in w3c/clipboard-apis#135
Should we define the mime types explicitly in the clipboard spec?

[dway123]

Yes, I think https://www.w3.org/TR/clipboard-apis/#mandatory-data-types-x should also mention types required for the async clipboard apis.

We could mention that these formats are also mandatory for the async clipboard api. (Alternatively, as most browsers only implement a subset of those formats, we could also have a separate section for formats required by the async clipboard api. A separate section would be less consistent though...)

[rniwa]

Do we have any data on interoperability of the set of "mandatory" MIME types there? FWIW, WebKit/Safari doesn't support have at least have of those MIME types and we have no plan to add the support.

[dway123]

I'm not sure about overall interoperability data, but I'm aware that at least some of these formats also aren't supported / planned to be supported by Chromium as well. My understanding is that the spec intends to provide recommendations for formats browsers should support (and hence may not reflect actual supported formats on each platform).

For Chromium, a quick search suggests that there is at least some implemented/planned clipboard support for text/plain, text/html, image/svg+html, text/uri-list, image/png, and application/octet-stream, and no support for all the other types (besides as the undocumented/unspecified pickled formats that Blink/Webkit both support). For async clipboard, I'm aware that Chromium currently supports text/plain, text/html, and image/png.

What MIME types do WebKit/Safari and Gecko/Firefox currently support on the general clipboard or async clipboard API? We can build this interoperability data here, if we can't find it elsewhere. I'm aware that Webkit/Safari's Async Clipboard API supports the same formats as Chromium, plus text/uri-list, per this blog post, but I'm not sure what formats are supported in the overall clipboard for Webkit/Safari, or overall for Firefox. (later, this information should likely also go in the MDN article or similar)

[rniwa]

I'm not sure about overall interoperability data, but I'm aware that at least some of these formats also aren't supported / planned to be supported by Chromium as well. My understanding is that the spec intends to provide recommendations for formats browsers should support (and hence may not reflect actual supported formats on each platform).

That seems problematic. We need to write spec's based on the reality, not desires of various people involved in writing them. We should probably remove all those types not planned to be supported by Blink/Gecko/WebKit.

What MIME types do WebKit/Safari and Gecko/Firefox currently support on the general clipboard or async clipboard API? We can build this interoperability data here, if we can't find it elsewhere.

Yeah, we should probably do that.

FWIW, I believe WebKit supports text/plain, text/html, text/uri-list, and image/png but certain native formats like web archive, tiff, etc... are transformed as text/html, image/png in our end.

@whsieh

[whsieh]

Yeah. My understanding is that if we want to explicitly define supported MIME types at all, we should just define the set of MIME types that Blink, Gecko, and WebKit already support.

From what I understand, this only includes text/plain, text/html and image/png. (It wasn't clear to me whether Firefox supports text/uri-list — I tried copying a URL from Safari and pasting into Firefox, but only observed text/plain).

Hypothetically, if we were to add to this set in the future, it seems that doing so should require both (1) implementation from all major browser engines, and (2) that it's possible to sanitize this data, if it has the potential to leak PII or otherwise trigger serious issues when pasted in native applications.

[dway123]

My understanding was that specs were supposed to be written to ensure APIs can be flexible and future-proof, potentially including things that aren't yet implemented in all browsers, but I understand that this can make them confusing for interoperability.

@evilpie to comment on Gecko's supported formats. Per @whsieh 's analysis, perhaps Gecko supports just text/plain, text/html, and image/png.

[rniwa]

My understanding was that specs were supposed to be written to ensure APIs can be flexible and future-proof, potentially including things that aren't yet implemented in all browsers, but I understand that this can make them confusing for interoperability.

I think that could be achieved by that browsers MAY support those types. But we really need to say MUST for the types that all browsers support.

[dway123]

Thanks. Agreed that "mandatory data types" is fairly strong language, and that we should trim the unused formats in that list. We could also leave formats implemented by at least one browser (like text/uri-list) in a separate "optional data types" section.

(For history, this type list used to be named "mandatory and optional data types", and was updated to remove the "optional" name in 2011 in this commit)

[evilpie]

So looking at the Gecko we seem to support a lot of MIME types when using the clipboarData API like

window.addEventListener('copy', event => {
    event.clipboardData.setData('text/plain', "Hello World!");
    event.clipboardData.setData('text/x-moz-url', "http://example.org\nTitle")
    event.clipboardData.setData('text/rtf', "RTF mime");
    event.preventDefault();
});

The actual list is here: https://searchfox.org/mozilla-central/rev/2f1a015b004b79f1145c81cdf86b15481a5630e2/dom/events/DataTransfer.cpp#941. I hadn't actually looked at this before, I am not convinced we should support all of them. @annevk

For clipboard.write, which we aren't shipping anywhere, we support text/plain, text/html and image/png. I suspect we would align with whatever is specified before shipping.

[annevk]

I think ideally we have a fixed list of MIME types that we all agree on and all other MIME types get pickled. It seems that once we start with pickling we could not later change this without potentially breaking applications? Although I suppose you could have duplicate entries, but I'm not sure what the point would be.

[mkruisselbrink]

I don't think changing the list of non-pickled mime types after we start with pickling would break anything (or at least I think we can design the pickling API to make sure of that). This is one reason why currently the explainer mentions that if a website specified that a particular mime type should be pickled, we'd still also write the unpickled version of the same type, if the type is supported that way. The pickled format won't undergo sanitization, while the unpickled one will. As such there is value in including both on the system clipboard.

Having said that, I do agree that it we should document what types are supported without pickling across browsers. In chrome that list currently is indeed also text/plain, text/html and image/png, although we're hoping/planning to add image/svg+xml to that list shortly.

I also agree we should invest more in making the async clipboard API spec more precise, for example I don't think it currently specified at all what should happen when a type isn't supported by the async clipboard API. Hopefully as part of the work on pickling we can make sure that all this gets better specified. We definitely should have done better here.

[snianu]

I created a PR clarifying the mandatory types that the UAs may or must support in the clipboard read/write APIs. Please let me know if anyone has any concerns.