Implement spec-compliant CanonicalizeLanguageTag

[jackhorton]

V8 relies on ICU for seemingly the same things that we do, and we both fail a lot of test262 tests because of it.

[jackhorton]

@jefgen I noticed that ICU has a hardcoded list of grandfathered language tags and their replacements, and some of them are simply wrong as of https://www.iana.org/assignments/language-subtag-registry/language-subtag-registry. Do you know the rationale there and/or ICU's general thoughts towards RFC5646/UTS35 canonicalization? Aka, is there a specific reason why ICU doesn't canonicalize (either by default or through an API) to those standards, or is it just that no one has cared enough to implement it?

[jefgen]

Sadly, I think it is more a case of nobody has has time to implement/update and/or change things.

Though recently Jungshik filed several bugs and tasks in JIRA for better handling of BCP47 tags. For grandfathered tags there is ICU-13650, which also has a number of linked tickets...

WinGlob used to rely on another component, Bcp47Langs.dll which actually had a full copy of the IANA Subtag Registry internally. However, it imported the data in a very odd way (and also encoded it in a hand-rolled encoding scheme), which made updating the IANA data very cumbersome. It also had some limitations with some of the grandfathered tags, but I think it was actually "better" generally speaking than what ICU currently supports.
Hopefully this area can be improved in ICU in the near future.

[jackhorton]

I see. If I feel like writing some python/powershell I may write a script to go and parse the registry and generate a version of the data that we can use, but its good to know that its at least possible in ICU (in that there isn't a philosophical or back-compat reason for having custom canonicalization)