Journal text rendering broken for non-English languages

[dgelessus]

Originally discovered here: H-uru/moul-assets#237 (review)

Several journals render badly in non-English languages. At some point after a non-ASCII character appears, the text stops rendering until the end of the current double page. The following pages sometimes work, sometimes they're partially broken too, and sometimes also completely blank. Sometimes a page renders partially at first, but then turns blank after flipping back and forth a few times.

This affects at least the credits journal, Yeesha's book for the pillar ages on the Relto bookshelf, and the KI/Nexus notebook in the hood classroom. I've reproduced this with French, German, and Russian.

How it's supposed to look in German - with Gehn 1.28, based on 2f1c126 from 2021-04-09:

How it looks right now - with 730d2a3 from 2023-06-21 and H-uru/moul-assets@08f4fdf from 2023-05-17:

I have no idea yet what causes this bug or when it was introduced (other than "sometime in the last two years").

[dgelessus]

@dpogue mentions:

IIRC this came up in testing of Kalamee's credits journal translation too (on an OU client, I think?)

[dpogue]

The discussion about the Kalamee journal is here(ish): https://discord.com/channels/282045216221822978/1053344855876714537/1114318208560807956

Since @DoobesURU was testing it, I'm going to venture that it was more likely a H'uru client.

[dgelessus]

Narrowing down the timespan a bit: 8b27f1e (2023-01-02) was still working, whereas d09814f (2023-03-04) already has the bug.

[dpogue]

oh good, that is a much more reasonable time period to try to bisect 😅

8b27f1e...d09814f is the diff

[dgelessus]

273e662 (2023-02-06) good, c49ec08 (2023-02-19) bad. Guessing I broke it in #1316. So much for "Improve Unicode support" 🙃

[dpogue]

Previously it was using wchar_t for a bunch of stepping through strings and character comparisons. In theory the change to char should be safe, but that would be my first suspicion given that this bug appears to only affect non-English text.

[dgelessus]

It was indeed #1316. Result of git bisect:

There are only 'skip'ped commits left to test.
The first bad commit could be any of:
53a706a
8efde95
4d866db
d4cff0b
42d8d15
We cannot bisect more!

The skipped commits are because I introduced a crash there and only noticed and fixed it a few commits later. That part is probably worth looking at... (I'm done for tonight though)