Support Unicode Keyboard events in xrdp

[MagicStarTrace]

xrdp: A Remote Desktop Protocol server.
Copyright (C) Jay Sorg 2004-2014
See http://www.xrdp.org for more information.
Version 0.9.5

In the same environment of Chinese and related dependencies, vnc is a device that can bring external Chinese into the remote ubuntu 18.04 system

I currently use the input method in ubuntu, but I hope that the Chinese from the external client can be brought into the ubuntu system

---

apt install -y language-pack-zh-hans fonts-droid-fallback ttf-wqy-zenhei ttf-wqy-microhei fonts-arphic-ukai fonts-arphic-ukai fonts-arphic-uming

[matt335672]

Hi @huyifanstar

I don't understand what you are trying to do.

Are you trying to get a Chinese keyboard mapping working on the xrdp session? Or are you trying to get the whole locale transferred from the client to the server?

[MagicStarTrace]

Hi @huyifanstar

I don't understand what you are trying to do.

Are you trying to get a Chinese keyboard mapping working on the xrdp session? Or are you trying to get the whole locale transferred from the client to the server?

I hope that the Chinese input from the mac client (rdp client) can be transparently transmitted to the remote instance on the mac through the client

[metalefty]

@matt335672 The feature he wants is called "Unicode input" / "Unicode Keyboard Events" that the client sends inputs as Unicode characters rather than scan codes. See also #441. xrdp partially supports it. It is possible to type ASCII characters in Unicode mode but further uses are not supported because the inputs are transmitted from xrdp to Xorg as keystrokes via virtual keyboard. We need to invent direct passing of Unicode characters to Xorg/Xvnc to implemtnt it.

Android/iOS RDP clients support it. Also, macOS RDP client support it since last year.

It will work like this.

RPReplay_Final1631250814.mov

[MagicStarTrace]

@matt335672 The feature he wants is called "Unicode input" / "Unicode Keyboard Events" that the client sends inputs as Unicode characters rather than scan codes. See also #441. xrdp partially supports it. It is possible to type ASCII characters in Unicode mode but further uses are not supported because the inputs are transmitted from xrdp to Xorg as keystrokes via virtual keyboard. We need to invent direct passing of Unicode characters to Xorg/Xvnc to implemtnt it.

Android/iOS RDP clients support it. Also, macOS RDP client support it since last year.

It will work like this.

RPReplay_Final1631250814.mov

Convert Unicode characters to Xorg/Xvnc , Is there any way I can configure it to enable support?
Configuration file ?

Thank You

[metalefty]

t is possible to type ASCII characters in Unicode mode but further uses are not supported because the inputs are transmitted from xrdp to Xorg as keystrokes via virtual keyboard. We need to invent direct passing of Unicode characters to Xorg/Xvnc to implemtnt it.

You didn't read here.

[matt335672]

Thanks for the explanation.

@huyifanstar - I'm going to change the title of this issue to reflect @metalefty's comments above.

[MagicStarTrace]

@matt335672 The feature he wants is called "Unicode input" / "Unicode Keyboard Events" that the client sends inputs as Unicode characters rather than scan codes. See also #441. xrdp partially supports it. It is possible to type ASCII characters in Unicode mode but further uses are not supported because the inputs are transmitted from xrdp to Xorg as keystrokes via virtual keyboard. We need to invent direct passing of Unicode characters to Xorg/Xvnc to implemtnt it.

Android/iOS RDP clients support it. Also, macOS RDP client support it since last year.

It will work like this.

RPReplay_Final1631250814.mov

Adjusted to Unicode on mac, just tried it, it did not meet expectations
Is there any progress in this area?

Eventually the xrdp server will support Chinese Unicode keyboard events?

Thank you!

rdp.mov

[zsj1029]

Run these commands to generate the keyboard mapping for your chosen input.
00000804 is chinese keyboard Mapping Code
xrdp-genkeymap km-00000804.ini
To copy it in the xrdp folder

sudo mv km-00000804.ini /etc/xrdp
Remember that you must change the permissions of the file, so that it can be used

sudo chown root:root /etc/xrdp/km-00000804.ini
Restart the service and it should work

sudo service xrdp restart

That'all . Enjoy it

[zsj1029]

Maybe Xrdp could set km-00000804.ini as default installation

[MagicStarTrace]

Maybe Xrdp could set km-00000804.ini as default installation
Ok, thanks a lot!

Where can I get this (km-00000804.ini)?

Do I need to make special settings on the client side?

Thank You!

[zsj1029]

Run these commands to generate the keyboard mapping for your chosen input. 00000804 is chinese keyboard Mapping Code xrdp-genkeymap km-00000804.ini To copy it in the xrdp folder

sudo mv km-00000804.ini /etc/xrdp Remember that you must change the permissions of the file, so that it can be used

sudo chown root:root /etc/xrdp/km-00000804.ini Restart the service and it should work

sudo service xrdp restart

That'all . Enjoy it

Just run command upon
Client no need make special settings
Change to chinese input method, it works

[seflerZ]

Run these commands to generate the keyboard mapping for your chosen input. 00000804 is chinese keyboard Mapping Code xrdp-genkeymap km-00000804.ini To copy it in the xrdp folder
sudo mv km-00000804.ini /etc/xrdp Remember that you must change the permissions of the file, so that it can be used
sudo chown root:root /etc/xrdp/km-00000804.ini Restart the service and it should work
sudo service xrdp restart
That'all . Enjoy it

Just run command upon Client no need make special settings Change to chinese input method, it works

Tried but doesn't work. @zsj1029 Did I miss something?

[zsj1029]

Run these commands to generate the keyboard mapping for your chosen input. 00000804 is chinese keyboard Mapping Code xrdp-genkeymap km-00000804.ini To copy it in the xrdp folder
sudo mv km-00000804.ini /etc/xrdp Remember that you must change the permissions of the file, so that it can be used
sudo chown root:root /etc/xrdp/km-00000804.ini Restart the service and it should work
sudo service xrdp restart
That'all . Enjoy it

Just run command upon Client no need make special settings Change to chinese input method, it works

Tried but doesn't work. @zsj1029 Did I miss something?

make sure u have installed Chinese input method Fcitx in your OS first

[seflerZ]

@zsj1029 Here is the log. As you can see all keyboard maps has been found. I'm using Fcitx to use WuBi-86 input method too.

Would you mind to provide the logs you have successfully enabled?

[MagicStarTrace]

Run these commands to generate the keyboard mapping for your chosen input. 00000804 is chinese keyboard Mapping Code xrdp-genkeymap km-00000804.ini To copy it in the xrdp folder
sudo mv km-00000804.ini /etc/xrdp Remember that you must change the permissions of the file, so that it can be used
sudo chown root:root /etc/xrdp/km-00000804.ini Restart the service and it should work
sudo service xrdp restart
That'all . Enjoy it

Just run command upon Client no need make special settings Change to chinese input method, it works

root@S0-GPU:/etc/xrdp#xrdp-genkeymap km-00000804.ini
xrdp-genkeymap: unable to open display ''

How should this be resolved?

[seflerZ]

Well, I digged into the code recently. You can't send unicode to remote directly at present because xrdp simply doesn't support it. It does accept unicode input at method "xrdp_wm_key_unicode()" in file "xrdp_wm.c". But it converts it into scan code by comparing the glyphs and invokes "xrdp_wm_key()".

The reason xrdp module send scan code is because in the backend xorgxrdp, it simulated a virtual keyboard and uses "xf86PostKeyboardEvent()" to send keys. This function supports scan codes only.

I think what @zsj1029 said it can support other keyboard layouts other than en-US. For example, you can type in Greece glyphs or Japanese hiragana with unicode input enabled. It might work because it can be mapped to keys on the board. But you can not input other unicode which is not mapped to keys. For example, the Chinese has more than 10k characters, it's impossible to map them to the keyboard directly. Therfore, a way must be found to support input unicode directly, other than simulating keypresses.

As what I said, the "xf86PostKeyboardEvent()" supports scan code only. I don't know which X Window API supports it so I can upgrade the xorgxrdp input driver. @metalefty do you have any idea?

[matt335672]

I'm not as familiar with X11 internals as some of the other developers here, but I suspect the keycode model is too deeply embedded in X11 to implement this, at least with X11 backends.

The RFB protocol used by the VNC back end is documented here:-

https://github.com/rfbproto/rfbproto/blob/master/rfbproto.rst

There's no provision for providing generic Unicode input as keypresses.

X11 key events (which are visible to the application) are here. They are keycode dependent:-

https://tronche.com/gui/x/xlib/events/keyboard-pointer/keyboard-pointer.html

Despite the 'unsigned int' in the above, on the wire, X11 keycodes are only 8 bits, which gives us very limited possibilities:-

https://x.org/releases/X11R7.7/doc/xproto/x11protocol.html

I'll be very happy to be proved wrong here though, if anyone has better info than I have.

[seflerZ]

@matt335672 Okay, thank you. It seems X Server API takes over of hardware abstration only. We can make another way, to take advantage of iBus or Fcitx input method APIs to generate characters. It is centain that we can implement our iBus or Fcitx engine and accepts requests from xorgxrdp to input unicode characters through IPC calls. But this approch is difficult, costly and hard to maintain. I hope there is a way to send requests to iBus directly to generate unicode characters. I'm working on it.

[matt335672]

That sounds doable, @seflerZ but I agree with all of "difficult, costly and hard to maintain".

At the moment, the RDP input PDUs are send directly from xrdp to xorgxrdp or VNC. It might be possible to send the Unicode PDUs to chansrv instead and get that to interface to the session iBus. That's fine providing chansrv is connected to xrdp. There are still a few bugs in this area as I'm sure you are aware.

[seflerZ]

Hi, @matt335672 . I found it's much easier than I expected. We can implement a tiny iBus input engine within 100 code lines and use these functions to send unicode characters to iBus.

// Unicode identifier for Chinese character "你"
gunichar chr = 20320;
ibus_engine_commit_text(engine, ibus_text_new_from_unichar(chr));

As for the engine, we can use "ibus_engine_desc_new()" and "ibus_component_add_engine()" APIs to create a temporary one.

So the question becomes, where shall we put this implementation, in "xrdp" or "xorgxrdp" module? I think in "xrdp" is more approachable, since "xorgxrdp" is designed for dealing with X Window system.

[matt335672]

It really depends on how the IPC works for ibus.

ibus runs on top of glib2, so it probably uses D-Bus, although I'm not sure about this.

Neither xrdp or xorgxrdp have a connection to the session D-Bus. I think you'll need to get the data into xrdp-chansrv which has full access to the session.

That's going to be quite fiddly. By all means have a go, but if you'd like a hand let me know what branch of xrdp you're working on.

[jsorg71]

I think @matt335672 is right, if you want to talk to ibus, chansrv is the place to do it. There is already some non channel message in chansrv. I think the rail code can send RDP orders for new window or update window for example.

I don't think ibus is good for general input. I'll suggest another possibility. If we can map unicode to keysyms we could do something like xdotool. It looks if the keysym is in the current map, if it is, it uses it, if not, it looks for the first free scancode and maps the new keysym to the scancode, uses it, then removes it.
You can use xev and xdotool to see how it works.
We could do this in xorgxrdp where we already have a communication path.

[seflerZ]

I agree that iBus is not a general input solution. But if we implement our own input system, we'll need to deal with various window frameworks like Qt\Gtk2.0-4.0\x11\Wayland etcetera. That will be a big project...

As for "keysyms", It does map unicode characters to keys on the keyboard. But as I memtioned, some language like Chinese has more than 10k characters (Japanese also use characters origin from Chinese), we must simulate a gaint keyboard with thousands of keys to hold all the unicode characters. It is doable but some what weird though. (Besides, xrdp limited the size of keys to 256 at present.)

Of course, it will be great if we can find any simple and elegant approach.

[akallabeth]

would be really great to have this implemented, modern clients more and more start to use this as they do not have a keyboard to forward but some touch interface. (using UnicodeKeyboard is a huge help there as the guessing game which keyboard layout is active can be avoided)

[matt335672]

I can't see a simple and elegant approach myself I'm afraid:-

1. ibus isn't by any means globally used.
2. The reconnection to chansrv is quite flaky at times for current xrdp versions. We'd have to solve this for a robust solution based on chansrv.
3. Jay's method is (I think) going to generate a lot of MappingNotify events for all X applications. We also can't use it for the few remaining use-cases where a VNC backend is necessary.

If we did implement something in chansrv, a production solution would need a way for the user to configure the desktop input method - a global method wouldn't be adequate.

At the same time, if @seflerZ want to try modifying chansrv for prototyping purposes I'm happy to give him a hand. We'll learn more that way about whether a robust solution can be found.

[matt335672]

@seflerZ

I had a think about this over the weekend, and had a look at the code myself. It's pretty hard to find your way around it, so I thought I'd put something together quickly to get you started. That way you can focus on the immediate problem rather than worrying about how xrdp is put together.

I've implemented a basic method to send unicode input from xrdp to chansrv. Code is here, based on the devel stream:-

https://github.com/matt335672/xrdp/tree/initial_ibus_testing

This code adds an --enable-ibus switch to configure. If this is specified, Unicode keyboard events are passed to chansrv (if it's up) and simply logged in chansrv in process_message_unicode_key_press().

I've added the relevant libraries to chansrv, so you should be able to expand the code in there and call ibus methods.

This is pretty basic and would need more work before it went into production. After you've played with it, we can make it more robust if you think between us we can make something useful.

Also, being based in the UK, I can't find a way to test this here. If you have any suggestions, please let me know.

[seflerZ]

@matt335672 Thanks. I'm dealing with the iBus now. Got some progresses but also problems. But I think it won't take too much long.

[seflerZ]

Hello! Good news. It is basically working now!

HnVideoEditor_2024_03_29_001554912.mp4

[matt335672]

That's great @seflerZ

When you've had a bit more time to make sure you're happy with it feel free to raise a PR on my repo. I'll have a look and get my stuff better engineered and then we can put it out for wider review.

[seflerZ]

@matt335672 Hi, this feature is almost done. However according to MS-RDPBCGR 2.2.7.1.6 Input Capability Set, there is a flag sent from client named TS_INPUT_CAPABILITYSET to indicate it supports unicode input service. I want to use it as a flag to decide whether the iBus component to be initialized (otherwise resource wasted).

I managed to figured out I can add a new attribute named "support_unicode_input" the "struct xrdp_client_info" and set it in "xrdp_caps_process_input" method of "xrdp_caps.c". However when I add the attribute the xorgxrdp seems stopped working and go into black screen.

I know the "struct xrdp_client_info" is shared by "xrdp" and "xorgxrdp", but I don't know why it can not be added easily. This may be caused by the "xrdp_client_info" structure is transferred through socks to "xorgxrdp". Would you help me to figure it out? Or we can just do the initialization no matter what the client requests. It works though.

[metalefty]

@seflerZ Did you rebuild xorgxrdp after you added a new attribute to xrdp_client_info with new xrdp_client_info.h? Usually, shared structures contain padding but we didn't add padding so I guess even the simple adding of new attributes may cause a glitch.

[matt335672]

@seflerZ - I had some thoughts on this.

I'm really short of time this week, but I saw the workflow going like this:-

1. chansrv is built with ibus/fcitx/whatever
2. chansrv sends a subscription message to xrdp when it starts to say it can handle Unicode injection
3. At that point xrdp can check the flags it was sent, and either forward Unicode to chansrv,, or not.

That puts the decision as to whether Unicode is supported into chansrv, rather than making it a bit of a mix between chansrv and xrdp.

The client info structure is another way to do it of course.

I'll happily add more detail when I can. In the meantime, feel free to come back to me about this.

[seflerZ]

Yeah, there is not need to transfer the flag to xorgrdp module. I was intend to use the flag to decide whether the iBus will be initialized. Any way, I'll prepare the PR to make my question more clear.

[akallabeth]

@matt335672 @seflerZ that might not be sufficient. RDP input capability set is sent by server to client and client to server.
so, if the server announces support the client decides if it will use it or not. (I might have missed when chanserv starts up, so please ignore if that happens before the session initialization)

[matt335672]

Thanks @akallabeth

At the moment the server always announces Unicode support to the client. If the client then sends Unicode, there's a fallback implementation that reverse-looks up the character in the existing character map. If the character is found, the relevant keypress event is generated. We need this for the login screen, as Unicode support is negotiated before the session is started.

chansrv is started as part of the user's session, so it's relatively late in the process.

@seflerZ's function is replacing this fallback in the event that chansrv decides it can provide the Unicode character injection directly.

[seflerZ]

Hello, everyone. The feature is ready. This is the branch I will raise for PR. I've tested it for two weeks and it works perfectly. For testing, just compile the "xrdp" with "./configure --enable-ibus --enable-fuse" (You need have iBus and lastest xorgxrdp devel branch installed). One of the best benefits of supporting unicode input is, you can use your phone's voice input to input long text by one time. It's just awesome (especially in small screen devices, lol)! Thanks for @matt335672 's help!

SVID_20240417_220934_1.mp4

Besides, I still have two questions left:

1. Is located in "xrdp_mm.c:3043". Where this the best place to trigger iBus component initialization? Currently I put in the "xrdp_mm_chansrv_connect".
2. Is located in "input_ibus.c:226". The iBus feature needs to connect to iBus as we know, but the iBus daemon may not be ready when the user first logged in. So here I just make process waiting for 3 seconds. I don't think it's an elegant way, but I can find a flag which indicate iBus daemon is ready at present.

[matt335672]

Hi @seflerZ

In answer to your questions:-

1. That seems like a good idea. Following on from that, it might be useful (if the initialization succeeds) for chansrv to send a message to xrdp saying that it supports Unicode input. That way, if anything goes wrong, xrdp won't send Unicode to chansrv and might be able to do something else with it. I can give you a hand to make that change if you like.
2. What happens if the daemon isn't ready? Can you pick that up, sleep for a small amount (say 500ms) and then try again with a loop count? It's not elegant either, but is more adaptive.

[seflerZ]

@matt335672

1. Yes, I was intend to add an attribute in client_info to indicate unicode input is supported and set it in method "xrdp_caps_process_input" in file xrdp_caps.c but resulted in black screen. It seems that the xorgaxrdp has compatibility issues (I'm sure I've recomipled it with latest client_info struct because I know it is shared with "xorgxrdp" module). Your help is highly appreciated.
2. If the iBus daemon is not ready, then the ibus_init() method will fail and fall back to non-unicode support input. It still work, but you know it a little bit weired since users've been satisfied with everything needs to enable unicode input.

[matt335672]

I'll take 2) first:-

If the iBus daemon is not ready, then the ibus_init() method will fail and fall back to non-unicode support input. It still work, but you know it a little bit weired since users've been satisfied with everything needs to enable unicode input.

Is it possible to detect this and retry?

Yes, I was intend to add an attribute in client_info to indicate unicode input is supported and set it in method "xrdp_caps_process_input" in file xrdp_caps.c but resulted in black screen. It seems that the xorgaxrdp has compatibility issues (I'm sure I've recomipled it with latest client_info struct because I know it is shared with "xorgxrdp" module). Your help is highly appreciated.

I think we need an additional message from chansrv to xrdp to say Unicode input (via ibus/fcitx5/whatever) is supported. When that message is received, xrdp will send any Unicode characters to chansrv.

Data flow goes as follows. The first three steps are part of the current v0.9.x implementation

1. Chansrv is started
2. Chansrv receives 'channel setup' message from xrdp
3. Chansrv sets up channels

---

4. Chansrv starts one or more input methods (currently ibus only)
5. If an input method is started successfully, chansrv sends a message to xrdp saying that Unicode input is possible
6. xrdp sends Unicode to chansrv (if it's received the message, and chansrv is still up).

We don't really need to check the client flag for Unicode support. We always set the 'Unicode supported' flag in our initial attributes. We either get Unicode from the client or we don't.

What do you think?

[seflerZ]

Well, for the first part. I tried to use a loop to retry if ibus_init() resulted not connected but it's very strange that once it returns error no matter how many times you've tried the result won't change, it's not connected.
For the second, the question is how xrdp could know ibus/fcitx is running in the host. Should xrdp module use its priviledges to read process running on the host? If that is possible, I can try to implement it.

[matt335672]

The second part looks like this:-

1. Chansrv calls ibus_init()
   a) If that call works, chansrv tells xrdp to forward Unicode characters
   b) If the call doesn't work, chansrv does nothing. xrdp does not forward Unicode characters.

So xrdp doesn't need to know anything other than whether chansrv has asked for Unicode or not.

Does that make sense?

[seflerZ]

Sorry. I made the wrong order. I'm talking about Q1. That is whether we should have a flag to init ibus input or not. Otherwise the ibus thread will be wasted. Of course, we can always consider the client support unicode input and do fall back if not. In that case, we do not need that flag and Q1 can be resolved.The Q2 is about how to remove the sleep which is necessary now. Because even ibus was installed and properly configured, if we do no sleep when the session is firstly connected, the ibus could not be connected. So if we do fall back here, it does not work as it is expected.

[matt335672]

OK - I see now. If the client doesn't support ibus, we create a thread we don't use. I agree we should try not do do that.

I'm a bit short of time right now, but I'll get a patch to you ASAP.

[matt335672]

@seflerZ (and anyone else reading) - I've added PR #2 to your repository

Happy to discuss.

[poetic-edge]

Thank you for your contributions. You have provided me with great help.ヾ(≧▽≦*)o

However, I have also discovered two bugs.

1. When I specify to launch a certain program, such as a browser, IBus doesn't work. Currently, I manually start IBus in that session to resolve the issue.
2. In my client, resizing the window repeatedly triggers the function xrdp_caps_process_input. The original UIS_ACTIVE state is reset to UIS_SUPPORTED, which prevents subsequent Unicode delivery.

[seflerZ]

Hi. Not quite sure what do you mean by start a new program but iBus doesn't work. iBus should start with the user session. @poetic-edge

[poetic-edge]

@seflerZ In a similar way to "rdesktop -s: shell / seamless application to start remotely," you can achieve the desired functionality.

[matt335672]

@poetic-edge - are you saying that running without a full desktop session is the problem?

[poetic-edge]

@poetic-edge - are you saying that running without a full desktop session is the problem?

Yes, in this case, ibus will fail.

[seflerZ]

I think it is kind of like Remote APP.

[matt335672]

This might be something we need to fix outside the scope of this PR.

When a modern desktop is started on Linux, there's a whole load of systemd --user services that get started too. One of these is the input method daemon. Starting individual applications will need some, but possibly not all of these services.

@seflerZ - sorry to introduce more delay to this, but I'll be away from my dev. environment for a week or so. When I get back (assuming no emergencies) I want to get this merged into devel.

[JulyIghor]

@poetic-edge - are you saying that running without a full desktop session is the problem?

I use this bash code, it fixes the issue without full desktop environment. I see no reason for the sleep to be in the c++ code.

IBUS_ADDRESS=""
while [ "$IBUS_ADDRESS" = "" ] || [ "$IBUS_ADDRESS" = "(null)" ]; do
    sleep 1
    IBUS_ADDRESS=$(ibus address)
done
export IBUS_ADDRESS