Add Custom Clipboard format support to async clipboard.

[snianu]

For normative changes, the following tasks have been completed:

• Modified Web platform tests (https://wpt.fyi/results/clipboard-apis?label=master&label=experimental&aligned&q=async%20clipboard)

Implementation commitment:

• WebKit (https://bugs.webkit.org/show_bug.cgi?id=)
• Chromium (https://bugs.chromium.org/p/chromium/issues/detail?id=106449)
• Gecko (https://bugzilla.mozilla.org/show_bug.cgi?id=)

---

Preview | Diff

[tomayac]

At the risk of sounding incredibly stupid but with the excuse of being an English as a second language person: what is the background of the word pickling 🥒? Or is it about the tool ⛏ to pickle the strong edges off of dangerous clipboard items?

[snianu]

@tomayac Here is some additional background :) https://github.com/w3c/editing/blob/gh-pages/docs/clipboard-pickling/explainer.md#additional-background

I think I should just change the name to clipboard custom formats as it could be confusing to many. Thanks again for your feedback!

[tomayac]

@tomayac Here is some additional background :) https://github.com/w3c/editing/blob/gh-pages/docs/clipboard-pickling/explainer.md#additional-background

#TIL! Oh my God… This is super geeky, but completely inaccessible to the uninitiated. Now I know.

I think I should just change the name to clipboard custom formats as it could be confusing to many. Thanks again for your feedback!

Definitely consider renaming. Danke!

[annevk]

This depends on #158, right? And we also still have a discussion going on over in #150 as to whether these sanitization options make sense. It's not clear to me how this is ready for evaluation (or shipping).

[snianu]

@annevk

This depends on #158, right?

Yes, I think the spec is now in a good state after the recent changes. Will address Domenic's recent comments and then this will be ready for review.

And we also still have a discussion going on over in #150 as to whether these sanitization options make sense. It's not clear to me how this is ready for evaluation (or shipping).

#150 is related to writing unsanitized HTML content under the standard HTML format, not the custom format that this feature is about. I think we should evaluate this separately as custom formats are unsanitized content by definition. Also we had this discussion in multiple meetings with Apple and Google folks, and we all agreed to add this feature to the spec in a non-normative note. Here are the meeting minutes: w3c/editing#334 (comment)

[rniwa]

And we also still have a discussion going on over in #150 as to whether these sanitization options make sense. It's not clear to me how this is ready for evaluation (or shipping).

#150 is related to writing unsanitized HTML content under the standard HTML format, not the custom format that this feature is about. I think we should evaluate this separately as custom formats are unsanitized content by definition. Also we had this discussion in multiple meetings with Apple and Google folks, and we all agreed to add this feature to the spec in a non-normative note. Here are the meeting minutes: w3c/editing#334 (comment)

I don't see how? This PR adds unsanitized as a dictionary option to ClipboardItem. I see no reason that's necessarily if we don't add unsanitized in the first place.

[snianu]

I don't see how? This PR adds unsanitized as a dictionary option to ClipboardItem. I see no reason that's necessarily if we don't add unsanitized in the first place.

We added unsanitized to ClipboardItemOptions as it is optional and we don't want to create yet another member to track the unsanitized list. We don't need the unsanitized list during write(), we only need it for read(). I guess we could also make the unsanitized member optional in the dictionary. That way we could reuse this type without breaking anything? Let me know what you think.. These will be added as a non-normative note to the spec so browsers may or may not support this type.

[rniwa]

I don't see how? This PR adds unsanitized as a dictionary option to ClipboardItem. I see no reason that's necessarily if we don't add unsanitized in the first place.

We added unsanitized to ClipboardItemOptions as it is optional and we don't want to create yet another member to track the unsanitized list. We don't need the unsanitized list during write(), we only need it for read(). I guess we could also make the unsanitized member optional in the dictionary. That way we could reuse this type without breaking anything? Let me know what you think.. These will be added as a non-normative note to the spec so browsers may or may not support this type.

I don't see how that makes sense for a Web API. It means if someone writes code using Safari, it may not work in Chrome / Edge. Generally, we'd like to avoid that sort of thing if possible. My perspective on this PR is that we need to resolve the issue 150 first.

[snianu]

I don't see how that makes sense for a Web API.

We have several concepts in the ClipboardItem object that don't make sense on other platforms except iOS/iPadOS/MacOS. e.g.

1. We have a presentationStyle member in ClipboardItemOptions that is only supported in iOS/iPadOS. Chromium doesn't support this attribute and has no plans to support this type in the future.
2. We have multiple ClipboardItem support that doesn't make sense on platforms like Windows, Android, ChromeOS etc.

Yet we chose to add it to the async clipboard API anyways, so I think we can add the unsanitized option as an optional type in ClipboardItemOptions as we already have a precedent to add options that may never be implemented/supported on other platforms.

[rniwa]

Yet we chose to add it to the async clipboard API anyways, so I think we can add the unsanitized option as an optional type in ClipboardItemOptions as we already have a precedent to add options that may never be implemented/supported on other platforms.

Okay, we can agree to disagree on this. We'd submit an objection when the time comes.

[slightlyoff]

It's pretty unhelpful for us to be debating the shape of the API without example code and an explainer; it makes it somewhat difficult to judge how the proposed changes would be used.

@snianu: are you able to write up an explainer using the TAG's template? Example code in particular would be helpful:

https://github.com/w3ctag/tag.w3.org/blob/main/explainers/index.md

[snianu]

@slightlyoff

@snianu: are you able to write up an explainer using the TAG's template?

Sorry, forgot to mention in the PR. Here is the explainer: https://github.com/w3c/editing/blob/gh-pages/docs/clipboard-pickling/explainer.md

[mbrodesser]

@annevk

This depends on #158, right?

Yes, I think the spec is now in a good state after the recent changes. Will address Domenic's recent comments and then this will be ready for review.

And we also still have a discussion going on over in #150 as to whether these sanitization options make sense. It's not clear to me how this is ready for evaluation (or shipping).

Let's please finish #158 first and not rush these changes in.

[snianu]

Since we cannot come to an agreement, we have decided to abandon this PR. We will write about the new custom format map and the unsanitized option in a separate document which wouldn't be part of the clipboard API spec.

[annevk]

Hey @snianu, this seems really weird. Overall this is a feature that has buy-in from everyone, but there's a small aspect that doesn't. Is the idea now to not standardize it and just ship something in Chromium? Doesn't that kind of defeat the point of having a standards venue?

(I'm not sure why I deleted the branch, that was not intentional.)

[snianu]

@annevk I think the main point of contention was the unsanitized option which we would like to keep since we want to support sharing of the custom format content across origins(e.g. from Excel online to PPT to Word online to Outlook etc) and able to write unsanitized HTML using the standard HTML format. For Safari it is moot because they don't want to support that scenario. In our meetings with Apple folks, they raised the concern that a web spec shouldn't define platform specific clipboard format naming convention/format structure -- They wanted the OS to specify the custom formats. We think that these custom formats are special as they are generated by the web browsers, and the OS (have only checked Windows, CrOS, Android, Linux and Mac) already has support for custom formats, so standardizing at least the custom format map shouldn't be an issue. I wouldn't say that they were completely opposed to the custom format concept in general (and neither are you :)) but if we don't want to standardize the custom format map and unsanitized option, then I'm not sure what changes need to be made in the spec?

[annevk]

We don't have to define the exact format the OS clipboard uses, but we can say that these entries are special in that native apps don't recognize them. So it appears as application/example to websites, but not native apps. Defining for which formats that happens seems important for interoperability.

I do wonder how @rniwa et al envision the macOS clipboard to work in a way that native apps can work with Chrome/Firefox clipboard entries just as well as Safari entries. But if the macOS clipboard is extended for these kind of web entries it seems Chrome/Firefox could support that as well, in theory.

[snianu]

@annevk

Defining for which formats that happens seems important for interoperability.

Agreed. We do want to define the custom format map naming and its structure in the spec for interop reasons. Is it possible to discuss in the next EditingWG meeting? Would you be able to attend? Apple made it clear that they were not interested in implementing this new custom format feature, but with at least Chromium and Firefox support, I think we can add it to the official clipboard API spec using the MAY language instead of MUST? @BoCupp-Microsoft FYI

[annevk]

I've added https://www.w3.org/events/meetings/a97eea9a-0249-4da8-888a-95edc68aa65f/20220114T090000 to my calendar. Let me know if we cannot discuss it as it's clashing a bit with my normal evening.

[snianu]

@annevk thank you! I've added the agenda label but tagging few folks to this issue as FYI. @BoCupp-Microsoft @travisleithead @johanneswilm @whsieh

[css-meeting-bot]

The Web Editing Working Group just discussed Add Custom Clipboard format.

The full IRC log of that discussion

<Travis> Topic: Add Custom Clipboard format
<Travis> github: https://github.com//pull/162
<Travis> BoCupp: Context: Folks weren't interested in unsanitized option (that would require a sanitization library).
<Travis> .. Chromium-based browsers still wanted to move forward... one proposal was that a "browser may do X or Y". To specify whether they want unsanitized content or not.
<Travis> .. another options was not to add anything to the spec; but to fork the spec into another CG or separate doc. (The latter was what we did.)
<Travis> .. If we look at Anupam's PR, we see it will be a big maintenance headache ;(
<Travis> .. Therefore, I would like us to reconsider.
<Travis> .. I believe AnneVK chimed in showing their interest in this feature?
<Travis> .. Finally, why would we not add language in the primary spec? Might create pressure on Apple ("not spec compliant").
<Travis> .. I think we'd still have web.dev posts (and other blog posts when it launches).
<Travis> .. So... the absense of this (in the spec) may not justify not adding it to the spec.
<Travis> .. So, will listen to others' reactions?
<Travis> Anne: I'm not sure about unsantizied either.
<Travis> .. mainly was expressing support for custom formats (in the spec).
<Travis> .. I think that is something worth having in the spec (incl. Apple)
<Travis> .. Want to preserve that we would have a specification for custom formats.
<Travis> .. Separately, we need to consider the unsanitized option. I'm *less* favorable on that one.
<Travis> .. It's weird that the legacy API and new API do different things...I think we should try to unify them.
<Travis> BoCupp: For custom formats, I think we agree that there isn't a sanitization algorithm that we'd apply to custom formats (unknown structure)
<Travis> .. Wanted to be sure that together with unsanitized keyword, that the format would be listed.
<Travis> .. Do you think it's useful in that context?
<Travis> Anne: Interesting. I might be ammenable to that.
<Travis> BoCupp: To be fair, we doubled-up the purpose here... we wanted to provide this to the HTML format.
<whsieh_> sorry, not sure why my mic isn't working :( what I recall agreeing on last was that sanitization should be a feature of the browser, not something that sites should be able to opt out of. so blink would always unsanitize, and WebKit can continue sanitizing
<Travis> .. (details for legacy API vs new API)
<Travis> .. Don't really want to touch the legacy API behavior.
<Travis> BoCupp: Yes, that matches what I recall.
<whsieh_> I think there are ways to ensure backwards compatibility even without the sanitize option
<Travis> .. I think that's Apple's position.
<Travis> .. I think Chromium's position is different... want to focus on the mechanics (assuming we won't come to an agreement).
<Travis> .. And what's the most appropriate way to document it.
<whsieh_> right, but this is about whether or not we should build the concept of 'sanitize' into the spec right?
<whsieh_> (the sanitize option, that is)
<Travis> snianu: Domenic's last comment on my PR was that we don't have to define all the details, but add ref to the sanitizer API (and how its configured).
<Travis> .. per Domenic, we should just call algorithms that have undefined (up to the UA) behavior.
<Travis> .. We should be sure it's specified and deterministic.
<Travis> .. This is now blocking the PR.
<Travis> Anne: Historically, I would see failure of standardization process if we can't get agreement.
<Travis> .. We don't have many API in this sort of deadlock...
<Travis> BoCupp: Why I think it may be OK in this circumstance.
<Travis> .. The clipboard shape is unknown--someone could have written santizied content to the clipboard originally--how would one know?
<Travis> .. Today, you can get a loss of fidelity when clipboard transfer occurs.
<whsieh> we're okay with offering platform API to support unsanitized transport of information from native apps to the web
<Travis> .. I think it's OK if a UA wants to be more cautious (or not) in sanitizing at their discretion.
<Travis> .. So that web devs don't have to write unique code.
<BoCupp> q?
<Travis> johanneswilm: Devs do have to write different code if they get sanitized from one browser and unsanitized from another.
<Travis> .. might have provide an upload mechanism to help understand the data.
<Travis> .. (as an alternative)
<Travis> snianu: Most websites that parse HTML string.. they use getData/setData.
<Travis> .. in Firefox and all Chromium browser we get unsanitized content.
<whsieh> we only sanitize cross-origin
<Travis> .. this is not so for Safari (not sure if there's special code paths for Safari)
<Travis> Anne: Why even have a sanitize (if most are not sanitizing today)?
<Travis> snianu: I believe whsieh mentioned they only wanted to provide unsanitized content between same origin...
<whsieh> right. the only reason is compat
<Travis> Anne: I don't think we need the sanitized option then... if this is already something that is there? Why provide it? Wondering if there's a compat thing?
<whsieh> (but there are ways to fix compatibility issues without baking it into the spec)
<Travis> BoCupp: If you Ctrl+V and don't prevent the default, the browser needs it's own santiized logic (style adjustment, JS stripping, etc.) prep logic for "merge ready" HTML.
<Travis> .. There's a way for authors to do this too.
<whsieh> would it be useful to provide API to sanitize markup in the style of the clipboard?
<Travis> .. It would be more ergonomic to allow authors to leverage this "merge-ready" logic... thought having an option to choose would be good (for well-known formats).
<Travis> johanneswilm: On "is it a failure of the standardization process" is there another way we can move forward?
<BoCupp> q?
<Travis> .. On Bo's last comment: this can be problematic for editor creators--if they rely on the browser sanitization, it can lead to incompatible paste content in their editor--so they have to sanitize anyway.
<Travis> BoCupp: For moving forward, I'd like to see language that is a delta to the existing spec (rather than an entire clone of the spec).
<Travis> .. (assuming can't get to an agreement)
<Travis> .. but would like to have something like this (either or option) in the spec.
<snianu> https://github.com/w3c/editing/pull/383
<Travis> 👆 the forked PR
<Travis> snianu: The presentation style attribute is not supported in Chrome/Firefox. Also multiple clipboard formats are only supported on Apple platforms. So there is some precident already that is Safari-specific.
<Travis> .. (can this be another instance of that?)
<Travis> Anne: Specific to Safari for Apple platforms!
<Travis> whsieh: but does it need to be in the spec?
<Travis> Anne: only argument that's persuasive to me is Ctrl+V, and I'm not sure its that specific...
<Travis> BoCupp: Hmm, there are some potentially risky unsanitized content on the pickled formats.
<Travis> Anne: potentially different accessor for pickled formats versus built-in formats?
<Travis> .. Wouldn't say "unsanitized"... might have "unsafe" prefix or something similar. (Have used 'unsafe' in other APIs in the past.
<Travis> .. Could work for HTML and/or image formats?
<Travis> BoCupp: If we look at separate accessors... it's possible that an unknown format (today) could become a built-in format (future). Would it migrate?
<Travis> .. (thinking on the lfy here...)
<Travis> Anne: Could access some format through a pickled format and also not (through separate APIs)
<Travis> .. In that case security is up to website/native apps.
<Travis> whsieh: If the website writes pickled formats... we wouldn't want to expose that raw content to native app (they' need an opt-in)
<Travis> Anne: The native app would need to opt-in also, correct?
<Travis> .. native app can get to normal html/png directly, but if you wrote pickled html/png, they wouldn't get it.
<Travis> whsieh: and if they asked for regular html they'd get the sanitized version...
<Travis> .. could lead to issues if the website authors use pickled versions for the clipboard.
<Travis> .. today the browser writes a safe AND unsafe version to the clipboard...
<Travis> .. so that Apps can decide which to use. (If they don't opt-in they get sanitized version.)
<Travis> .. Important point: we should push websites to write both versions (or ensure the browser has a way to do that).
<Travis> BoCupp: Taking a moment to point out our implementation status...
<Travis> .. Don't think we'll get agreement on how all our implementations will align.
<Travis> .. Happy to change chromium implementation if we can get agreement.
<Travis> .. But assuming (based on history) that we won't.
<Travis> .. In Edge, we have shipped something to allow native apps to exchange content...
<Travis> .. In the chromium process, the hold-up is where we are going to write the behavior down.
<Travis> .. There are many options here. Looking for what the right approach is that minimized disruption/impact on the WG?
<Travis> Anne: felt like we were pretty close to an agreement!
<Travis> whsieh: We agree that the behaviors can be different.
<Travis> .. We don't agree that writing the unsanitized option into the spec is OK.
<Travis> .. last time, we talked about ways of configuring this (that are different from the unsanitized flag)
<Travis> .. mentioned a <meta> tag for this purpose...
<Travis> .. could solve the compat issue.
<Travis> BoCupp: Wouldn't be my preference...
<Travis> johanneswilm: Wondering if we could move this out... or have another meeting?
<Travis> .. moving to the time-zones meeting?
<Travis> BoCupp: To conclude... I'll try to setup a meeting over email. By default, Anne, whsieh, me... smaug, johanneswilm....
<Travis> .. +megan
<Travis> .. I will follow-up on the public-editing-tf list.

[annevk]

I've put forward an alternative proposal in #165.

[pwnall]

I think this PR had some non-controversial parts that seem valuable. Would it make sense to try to get those merged separately?

For example, I think that the added lines 610-616, 623-648 (minus 636) document Async Clipboard API behavior that is either shipped or non-controversial (not related to sanitization, custom types, or permissions).

I'm hoping this would make it easier to compare proposals for the controversial pieces.

[snianu]

@pwnall Yes, I created a separate PR to merge those changes into the clipboard API spec: #158