Display errors with certain characters

[whiteplastic]

I use a custom irssi theme that contains the UTF-8 "Fleur de Lys" symbol (U+269C - ⚜). While this character is displayed just fine when I use ssh, it just disappears in mosh. Also, there are display errors in irssi: random characters just disappear or get swapped by other characters. This only occurs when I use my custom theme so there might be a connection.

[keithw]

On Linux, this works fine for me, but on Mac OS X 10.7.3, the system does not know about this character and wcwidth() returns -1 (unprintable), so mosh does not know how many columns the character will occupy.

Assuming you are using a Mac, that unfortunately is the answer. We will report this to Apple.

[whiteplastic]

Yes, I'm on OSX 10.7.3. It seems like the system does know about this character. ssh and any other application I use knows and displays it, the only application that seems not to know it is mosh.

[kmcallister]

SSH doesn't need to know about characters; it just conveys a stream of bytes from one end to the other. Mosh has a terminal state object which is synchronized between server and client, so it needs the character metadata on both machines.

What outer terminal emulator are you using; is it OS X's standard Terminal.app? And do you have any other terminal emulators in the mix, e.g. screen or tmux?

You can compile and run this C program on both machines to check if wcwidth knows about U+269C.

#define _XOPEN_SOURCE
#include <wchar.h>
#include <locale.h>
#include <stdio.h>

int main() {
    setlocale(LC_ALL, "");
    printf("%d\n", wcwidth(0x269C));
    return 0;
}

(I didn't test this on OS X, so it's possible it will fail to compile for some reason.)

It will print a positive number iff the character is known. Make sure to run it in a Unicode locale. If you don't have one by default, you can do something like

gcc -o foo foo.c;  LANG=en_US.UTF-8 ./foo

If you get a positive number on both server and client, and yet Mosh does not work correctly, then there's a bug in Mosh and we can investigate further.

(In the long run I would like to use a dedicated Unicode library, and drop our dependence on the system locale libraries, which have caused no end of trouble. See discussion on #74.)

[keithw]

I think officially speaking, a Unicode app is supposed to use the "default" properties of the code point range (including width) if it doesn't know about the particular character. Unfortunately there doesn't seem to be a way to get these default properties in POSIX. A dedicated Unicode library would help with this.

[EdSchouten]

Hi Keith,

Just checking. I think you can't assume wchar_t is ISO 10646. It is just an implementation defined `wide character'. If you are working with ISO 10646 inside Mosh explicitly (not wide characters), then you shouldn't use wcwidth(). In the past I once needed a compact implementation of wcwidth(), explicitly for use with ISO 10646. Markus Kuhn has an implementation that seems to work quite nicely:

http://www.cl.cam.ac.uk/~mgk25/ucs/wcwidth.c

Maybe it is of any use to you? Otherwise, I'm pretty sure IBM's ICU should be of use:

http://site.icu-project.org/

Ed

[keithw]

Hi Ed,

At configure time we check for __STDC_ISO_10646__, which the C library is supposed to define if wchar_t is ISO/IEC 10646 / UTF-32. (We used to assert it, but in practice only GNU libc seems to define it, even though OS X and FreeBSD do also obey it in practice. We print a warning on configure on these systems.)

We may have to ship our own Unicode library eventually. ICU is kind of a monstrous beast though.

-Keith

[EdSchouten]

Hi Keith,

Thanks for the explanation!

[lilyball]

I just commented this on #361, but since it's about OS X it seems a bit more relevant to this ticket (although both tickets appear to be virtually the same thing):

---

I keep periodically hitting situations where various characters don't render in Mosh, because wcwidth() doesn't support them (OS X client, Ubuntu server). As documented, some characters are because OS X's wcwidth() returns -1, but I also see a bunch of characters (notably, emoji like U+1F4A9 PILE OF POO) that OS X supports but Ubuntu's doesn't (curiously, __STDC_ISO_10646__ on Ubuntu claims that Unicode 6.0 is supported, and the code chart for Unicode 6.0 does list this character, so I don't know why wcwidth() is returning -1).

At this point I'm thinking the only real solution to this problem is for Mosh to calculate character widths itself. Perhaps it could fall back to its own calculation if the platform-provided wcwidth() returns -1, thus allowing the platform's idea of width to take precedence for all characters it knows about. The only real issue with this that comes to mind is if the calculated width disagrees with how the rendering terminal thinks the character should display, but I did some research earlier today and it seems that all characters (including reserved ones) outside of the already-defined East_Asian_Width blocks are assumed to be "Neutral", which basically means they'll never have a width of 2. Assuming a width of 1 for any reserved characters seems reasonable, because if the OS disagrees it will provide an explicit 0 instead of -1 (and I'm suggesting you use this calculation only when the OS version returns -1).

---

Or as suggested in this ticket you could just ship your own unicode library entirely. My concern is that if Mosh thinks a character has a width of 1 but the terminal emulator thinks it has a width of 2, that will presumably render incorrectly. I'm assuming that the terminal emulator agrees with wcwidth() (for all characters where wcwidth() returns a non-negative value; Terminal.app on OS X renders e.g. U+26A1 HIGH VOLTAGE SIGN as one cell but wcwidth() on OS X returns -1). That assumption is why I suggested above to use the return value of wcwidth() whenever it's non-negative and fall back to a custom implementation otherwise.

[lilyball]

Addendum: Apparently glibc uses Unicode 6.0 but it's LC_CTYPE support is still stuck at Unicode 5.0 (and wcwidth() uses LC_CTYPE).

[jhrmnn]

⚡, U+26A1 seems to be problematic for example. Mosh under Terminal.app displays it is as a zero-width character in Vim. Leading to very strange behaviour in a shell...

The left terminal is mosh/tmux/fish, right ssh/tmux/fish in the same tmux session.

When the mosh terminal is smaller than ssh, mosh is off by one character on the command-line. But if the ssh terminal is bigger, mosh is by some miracle right even though skipping ⚡.

This is probably not worth any work, I guess, but it might useful to mention this problem in documentation, so one can find it upon searching for unicode or utf-8. I spent good two hours on this :)

[cgull]

My current thinking on Unicode issues:

Mosh is a virtual terminal, split across client and server, and it
uses normal terminal datastreams between client and server.
Therefore, it must be consistent between client and server, and should
be as advanced with its Unicode version as it can be. If we are up to
date on Unicode, there's no need to match the server application's
notion of Unicode: if a server application outputs a Unicode character
that it doesn't know about, then it has already lost: if it's doing
any formatting of the output, it doesn't know how wide the character
is and may be feeding us corrupt line or full screen formatting to
begin with.

This argument dictates that Mosh must have its own internal wcwidth
implementation for its virtual terminal, because client & server may
have different host wcwidth implementations. If mosh receives a
character known by the server's wcwidth but not the client's, then
its placement of subsequent characters on the line will be wrong in
our virtual terminal, and we will lose badly, because Mosh quite
efficiently avoids redisplaying characters it doesn't think have
changed.

Mosh then sends the character off to the client's terminal, where it
can be correctly formatted and displayed. Now we have the problem
that the display terminal may have a lower version of Unicode than
Mosh does, and may therefore corrupt output if its notion of character
width differs from ours.

This is in general a hard problem: Most current terminal emulators
either depend on a system's GUI environment (gnome, kde) for i18n, or
have their own implementation to escape the vagaries of host OS
implementation. So most terminal emulators actually do something
better than the host OS's wcwidth implementation, which also means
that the host wcwidth does not usefully tell us what the terminal
will actually do. Mosh cannot know what version of Unicode the
display terminal is using; the only thing it can even begin to do is
output characters and check the cursor position after output. There
is one heuristic that we can check for: most terminal emulators set
environment variables to indicate their presence and sometimes even
their version. Using this heuristics means maintaining tables of
programs/versions against Unicode versions they support, though.

But the user can legitimately ssh into a remote host and run
mosh-client there, in which case these variables have been discarded
and we have no clue. We can't handle that. At all.

My current best idea for handling this is to offer the user two
options:

• Pass through all the Unicode characters Mosh knows about,
  naively expecting the client terminal emulator to handle it all. This
  is not at all what ssh does by blindly passing bytes through, but is
  similar in spirit and will be similar in behavior.
• Offer a --restricted-unicode option that implements, say, Unicode
  5.0 or the client host's wcwidth, and translates all characters
  unknown to that wcwidth to U+FFFD REPLACEMENT CHARACTER (�), padded with
  a space if our internal wcwidth tells us it's double-wide. This
  would probably happen at the point of final output from the client's
  virtual terminal.

One unfortunate thing here is that Unicode will continue to grow
with new versions. When that happens, if we upgrade our internal
wcwidth, we are back to the current situation of differing client and
server Unicode versions-- but if we have an up-to-date wcwidth
implementation, we are doing better than using the system
implementation.

Perhaps we need to design a scheme where the client gets a character
width table from the server. I think this idea has been mentioned before.

About the Markus Kuhn wcwidth implementation: It's
been brought up several times in Mosh discussion. It's an excellent
easy-to-understand sample implementation, But it has a number of
unpredictable branches, and then an expensive binary search through
its tables. The commonly-available copies of it available around the
net are now out of date, and it is slow. It has significant
performance impact when coupled with my performance code; I have
benchmarked it against the FreeBSD wcwidth and the musl wcwidth,
both are much better (but a lot less readable). Also, I offer you
this tidbit:

http://osdir.com/ml/internationalization.linux/2001-01/msg00191.html

Mosh is an application that uses wcwidth heavily, and can spend
significant time in slower wcwidth implementations, slowing down
character handling noticeably.

Separately, Google shows me a discussion on GNU libc that its wcwidth
calls an expensive linear search to determine which locale it's in.
That will no doubt get fixed, but.

I have not looked at it as closely, or in a while, but if I remember right ICU does not directly offer a wcwidth function, and in general it's a heavyweight featureful implementation not suited to be called for individual characters as often as we do.

[zuzak]

This doesn't appear to be a mac-specific issue: I have this problem in gnome-terminal on Ubuntu. Emoji don't render in an irssi screen session over mosh 1.2.4a, but do on the same screen session over SSH.

[rapha8l]

Hi,
Also on Linux ⮂ and ⮀ do not display at all with mosh 1.2.4a with any terminal and utf-8 set on both sides
Thanks