Why does headers-normalize-response.htm expect null bytes to be allowed?

[wisniewskit]

Per spec (first line is from the XHR spec; others are from Fetch spec which it links to):

A request has an associated header list (a header list). Unless stated otherwise it is empty.
A header list consists of zero or more headers.
A header consists of a name and value.
A value is a byte sequence that matches the following conditions:

• Has no leading or trailing HTTP whitespace bytes.
• Contains no 0x00, 0x0A or 0x0D bytes.

Apparently only Firefox goes against the spec and supports null bytes for XMLHttpRequests (but not Fetch in general): https://wptdashboard.appspot.com/XMLHttpRequest/headers-normalize-response.htm

Should the tests be changed to reflect the spec, or the spec (and other browsers) updated to match Firefox?

[annevk]

Hmm, that's for setting header values. I'm not sure response header values were fully considered. Sigh.

Dropping the header completely makes some sense (the null results), but would require changes to Chrome as well.

cc @mnot @tyoshino @yutakahirano

[mnot]

FWIW, corresponding HTTP spec is field-value:
http://httpwg.org/specs/rfc7230.html#header.fields

VCHAR gets you 0x21-7E, field-content adds in 0x20 and 0x09, and obs-text gets you 0x80-FF.

obs-fold gets you 0x0A and 0x0D in certain patterns (but you can also add that they're automagically folded in, which probably makes the best sense).

That means HTTP doesn't like 0x00-09 and 0x0A-1F.

Anybody have data about how often they're seen in the wild?

[annevk]

@mnot see httpwg/http-core#19 and whatwg/fetch#332 for how the new field-value is broken. I suspect the same might go for other places where the HTTP revision started to ban C0 controls.

[yutakahirano]

@davidben Probably you know more why Chrome behaves in the current way.

[davidben]

I don't know anything about this stuff, sorry. (Also not sure what the current behavior is to begin with... ask git log or mmenke? Dunno what his Github is.)

[wisniewskit]

@RByers, is there anyone we could ping to help figure this Blink XHR quirk out?

[annevk]

@MattMenke2 @bencebeky any thoughts on this?

[bencebeky]

My gut feeling is that (1) if no other browsers support null bytes, then it is probably safe for Firefox to reject them too, and (2) it might be less error-prone if the requirements for headers for XMLHttpRequests were consistent with the requirements for other requests. Other that this, I do not have strong feelings, and am happy to change Chrome behavior if there is a compelling use case.

[annevk]

@bencebeky thanks for the quick reply! I think the main problem with Chrome's behavior at the moment is that it strips 0x00 from header values. It doesn't reject anything.

http://w3c-test.org/xhr/headers-normalize-response.htm was added in https://bugzilla.mozilla.org/show_bug.cgi?id=1277019 and apparently Safari also passes all tests.

From that bug it seems this was already tested in http://w3c-test.org/cors/allow-headers.htm which reveals some minor security implications. (Disregarding the response or response header would also work for that specific case.)

The simplest solution based on this is probably accepting that response header values can contain 0x00 (at least when going over the network, no need to add support for this to Response objects and such). Rejecting headers or responses that have headers whose values contain 0x00 seems more involved (as it would require changing all clients), but doable if there's enough willingness.

(I also looked at Edge and it simply seems to hang on loading the page, which doesn't seem great. @travisleithead you might want to look at that.)

[bencebeky]

I understand. Indeed stripping characters seems fragile.

What is the plan with the Fetch spec? Is it going to be changed to allow null characters in header values? Or changed to require that such header fields are ignored?

[annevk]

@bencebeky well, I tried to solicit a plan with the above comment. I think the easiest would be to allow response headers to contain 0x00, but never include them in request headers and never allow 0x00 to be set through the API. @jakearchibald or would that be problematic for service workers if you wanted to forward a response that included 0x00 in header values?

[jakearchibald]

@annevk how would a request with a header like this show up in a service worker?

Rejecting headers or responses that have headers whose values contain 0x00 seems more involved

At what point would it be rejected, and what kind of rejection would it be?

[bencebeky]

It seems inconsistent to allow null characters in responses and not in requests. Why not stick to the current specification and disallow them in both? In any case, I agree that Chrome's current behavior is not optimal.

What is currently done with requests that have null bytes? Are the header fields ignored, or the whole request rejected with an error? If they are rejected, could responses be rejected with a similar error? If they are happily accepted, maybe the spec could reflect that.

Are you aware of a use case that would justify the need to accept headers with null characters?

[annevk]

@jakearchibald it would only be a response. (If we go with rejection, we'd probably have to treat it as a network error. Only dropping the header seems too dangerous.)

@bencebeky there's no way that I know of whereby you can set a request header value that contains a 0x00 from anywhere within the web platform. API-wise there's no way for response header values either, but I had forgotten to test this scenario (whereby the server sets it).

[bencebeky]

Okay, so it seems like it makes sense to disallow null bytes in request headers. For consistency, it would make sense to do the same in response headers, both XHR and otherwise. I agree that network error would be the best choice, otherwise there would remain server deployments that send null bytes and we would be stuck with having to support them forever.

[annevk]

Okay, I definitely support trying that out. I'll create a PR for web-platform-tests changes, will file bugs against Edge, Firefox, and Safari, and report back. I take it you're willing to take care of Chrome?

[bencebeky]

Yes, I'll take care of Chrome. Thank you for driving this!

[annevk]

Updated tests: web-platform-tests/wpt#10424.

Bugs:

• https://bugzilla.mozilla.org/show_bug.cgi?id=1453318
• https://bugs.webkit.org/show_bug.cgi?id=184493
• https://developer.microsoft.com/en-us/microsoft-edge/platform/issues/16880179/

[MattMenke2]

I'm a bit late to respond, but I agree with this decision - rejecting headers with nulls seems the safest option here. We should also add code to Chrome's code, either in net/ or in services/network, to reject requests with nulls in headers, to prevent compromised renderers sending headers with nulls in them.

…

On Wed, Apr 11, 2018 at 9:15 AM, Anne van Kesteren ***@***.*** > wrote: Updated tests: web-platform-tests/wpt#10424 <web-platform-tests/wpt#10424>. Bugs: - https://bugzilla.mozilla.org/show_bug.cgi?id=1453318 - https://bugs.webkit.org/show_bug.cgi?id=184493 - https://developer.microsoft.com/en-us/microsoft-edge/ platform/issues/16880179/ — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#165 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AUBj8bnPQFa35UNA6-7gxQyljLlnfiMQks5tngIOgaJpZM4P6CbD> .

[bencebeky]

Didn't find bug against Chromium, so just filed https://crbug.com/832086.

[annevk]

The tests finally landed. It seems no implementation has been updated so far. I guess the one thing that remains here is clarify somewhere that if an HTTP response header value contains a 0x00 a network error is to be returned?