Offer fallback for missing characters?

[kindaro]

A long-standing problem with Latex is that fancy Unicode characters never really work, because they are not in the font.

Normal text rendering systems (browsers, terminal emulators, and so on) would in this event look for another font that does have the character in question. For example, you would likely be able to see these symbols right now: ∷ (4 dots), ★ (a star). But rendering them in Latex is hell. For Xetex, there are awkward and poorly maintained ways to circumvent this problem by switching the font on the fly, such as the package ucharclasses. Sometimes they work, sometimes they do not. Sometimes they work half way through the document and then forget to switch back to your normal font.

How about we get some code to look for fallback fonts in the event of a missing character? We could even make a command line option for the invoker to supply a list of fonts that would be tried in order.

[pkgw]

(I've converted the original issue to a Discussion since I think that's a better form for the question raised here.)

I guess my initial feeling is that it is better to guide people towards creating documents that load fonts that have wide-ranging Unicode coverage, rather than implementing a way to bring in those fonts on-the-fly when needed.

On the one hand, this approach clearly doesn't help with the large corpus of existing documents that just use Computer Modern and don't actively do anything about their font at all ... on the other hand, by definition these documents shouldn't have any Unicode characters to handle since they wouldn't have worked!

It's a good question, though, and I'm really not sure how to approach it. If someone needs to include some Chinese characters in their document you probably don't want them randomly guessing which font is best. And there are situations where you want to be able to include arbitrary text in a document that might include Chinese characters (or Thai, or Arabic, or ...) without knowing in advance what you'll find. This seems especially relevant if you want to be able to process lightweight text+TeX markup in, say, a web form, which I definitely would like to do!

So, I dunno. Does anyone else have thoughts?

[kindaro]

I have.

I guess my initial feeling is that it is better to guide people towards creating documents that load fonts that have wide-ranging Unicode coverage, rather than implementing a way to bring in those fonts on-the-fly when needed.

This is not realistic.

1. One does not pick fonts because they have Unicode coverage. One picks fonts because they are beautiful, go well with other aspects of the design, and so on.
2. There is no single font with wide enough coverage.

You can say that I should glue my own font from several fonts available to me with some font editor, instead of asking Tectonic to do it for me. This is a general solution. However, my proposition is that rendering characters is an essential feature of a type setting system. We only excuse Latex because it is so old. If any program with even a single text area shipped without support for Unicode these days, it would be considered a glaring fault.

[pkgw]

Yes, as I think about it more, I think you're right. If I have a document in English and want to include some CJK text, I shouldn't have to wrap that text in some kind of font activation command. I should be able to just include it and get good results.

I do not know enough about ucharclasses and friends to have a sense as to whether they could deliver high-quality results here, or whether the engine actually needs to provide additional hooks to make everything work as well as it could/should. Either way, one thing I was trying to get at above is that I am OK with a world where Tectonic can deliver this functionality but one needs to do something special during setup (e.g. \usepackage{TectonicMakeMyFontsJustWork}) to activate the functionality.

[kindaro]

So, what is the way forward here?

[pkgw]

I think that basically the proverbial "someone" needs to do a lot of work 🙁

As a starting point, if you're able to document the shortcomings of the ucharclasses package, it would be helpful to have that written down. Or, putting it another way, if there's an existing framework like ucharclasses that can solve this problem, I'd much rather pursue that than try to build a new solution from scratch. I'd start by looking into whether its problems might be solveable or not.