Skip to content

On EGFX architecture and Dynamic Monitors

Jump to bottom Edit New page
Nexarian edited this page Jun 20, 2022 · 13 revisions

NOTE

This document is a work-in-progress. Some of the information here may be incomplete or incorrect.

A note on authorship

Nearly all the code referenced here was written by @jsorg71. @Nexarian wrote this wiki to help communicate the architecture and get reviewed to enable it to be checked in. @Nexarian also added the pieces to enable EGFX to work with Microsoft clients other than MSTSC as well as dynamic resizing, as well as misc fixes to enable the branched EGFX code to work rebased off of the devel branches of xrdp, xorgxrdp, and librfxcodec. But the core algorithms and code to implement them is @jsorg71's.

Introduction

The entire point of this is to hyper-accelerate the speed at which XRDP remoting sessions can work. The original version was useful for simple IT administrative tasks and maybe light development work, but after that the latency and the experience degraded significantly. When this project is finished, we should be able to have it all: Fast sessions, dynamic resizing, hardware acceleration, and easy installation. About the only think you won't be able to do with it is 4K gaming!

Right now the target audience is those who have some familiarity with Remote Desktop technologies. Some terms are not clearly defined, and some references are not directly called out. If this document needs more investment, please mention it on the XRDP developer Gitter channel

Background and history

In the mid-late 2000 aughts, from the research I can find, it seems that there was a debate about how to develop good video compression technologies. The first popular experiment was known as M-JPEG, or essentially the JPEG 2000 codec, optimized for video. XRDP itself has a JPEG compression mode that was much faster than the original protocol's raster protocol ancestor. However, it never caught on fully.

The main revelation is that JPEG, while highly compressed, is still a full frame every time. What if, instead, we could leverage high quality compression tech AND not send unnecessary redundant data? Experiments show that the main limiting factor of remote desktop speed is bandwidth. Even if the server and clients have to do a lot of work compressing/decompressing respectively, that's still a win in overall speed and smoothness of the experience. This is even true with high latency (EGFX H264 with XRDP seems to do well up to 100 ms of latency) if you have good bandwidth, so reducing the bandwidth is critical.

Enter the MPEG compression technologies, which DVDs use, and were later adopted by remote desktop and streaming services. The current version that most Remote Desktop systems use is H264, which sends a key frame, and then a "Group of Pictures (GOP)" in series that contain diffs from the key frame. Video streaming services like Netflix will send a key frame periodically to make sure everything is in sync, but currently XRDP doesn't do this. Tuning when to send a key frame refresh is a subject of debate. The details of how these encoders work is out of scope for this document, and it has taken years of R&D to build them.

In 2011 Intel released their Quick Sync GPU H264 encoder, and in 2013 Nvidia followed suit and released NVENC, their hardware based H264 encoder for their GeForce GPUs. This tells us that the industry had doubled down on the technology, and shortly after Microsoft released research on how to add H264 to their RDP protocol, using the GFX extension (more on this can be found in the references at the end of this article). AMD has an encoder, but it's not nearly as good as these other two.

There are successors to H264, such as H265. Google has VP8 (fork of H264) and VP9 (fork of H265). H265 compression is EXTREMELY computationally intensive and requires even more tuning than H264, and therefore is not useful for Remote Desktop at this time. The next big shift looks to be AV1, which is based on H265 but more efficient (though not yet efficient enough to be used for streaming). Of the above, only H264 has widespread adoption in terms of hardware support, meaning it's going to be the gold standard for many years to come.

Architecture

There are several codecs currently in use for XRDP prototypes for GFX integration:

  • Microsoft Progressive RFX
    • Based on Microsoft's RFX protocol, this is an adaptation that incorporates frame diffing into the JPEG2000 protocol to mimic H264. It is not as fast as H264 (It is merged into the devel branch as https://github.com/neutrinolabs/librfxcodec)
  • H264: X264 CPU accelerated
    • When a supported hardware encoder is not available, this relies on the common Linux X264 library, which does encoding in CPU and system RAM.
  • H264: Nvidia-NVENC accelerated.
    • If you own a relatively recent Nvidia GPU and can enable the standard Nvidia drivers on your session, this is blazing fast. It's also one of the preferred encoders for streamers that use software like OBS.
  • H264: Intel libva accelerated
    • Anecdotally the fastest experience available with XRDP right now, but it is also the hardest to install and configure. You'll need an Intel iGPU or presumably the up-and-coming Intel ARK GPUs for this. Note that for Enterprise servers most Xeon CPUs do not have the necessary encoder, also known as QuickSync.

Note that while AMD appears to be working on an encoder for their upcoming RDNA 3 architecture, current versions of AMD's H264 encoders are considered inferior and not broadly adopted in the industry.

Dynamic session resizing (aka resize-on-the-fly)

See the diagram of how this currently works here

Initial implementation of the ability to resize the XRDP session dynamically was relatively straightforward, protocol-wise, at least. However, when combined with the GFX pipeline, things became much more complex. With non-GFX use-cases, the entire session can be resized in a single-threaded function and performed nearly instantly with blocking calls, including with RemoteFX.

This isn't possible given the way Microsoft has combined the two features in their clients. Microsoft gives us little to go on for how to orchestrate the resizing, and a lot of this information is derived from trial and error. What we do have is this quote:

"Restart the graphics pipeline using the surface management commands (specified in [MS-RDPEGFX] section 1.3) if the Remote Desktop Protocol: Graphics Pipeline Extension is being used to remote session graphics." Reference

Essentially: The GFX Extended Virtual Channel has to be shut down, and then everything has to be re-incarnated using the new monitor descriptions. Ugh! This makes the resizing workflow MUCH more complex. Now, instead of a simple function that re-sizes everything, we have to re-write it such that we close the channel, resize the session, then re-establish the virtual channel with the new size. To do this, we need a state machine. To figure out the proper sequence in that state machine, we have to experiment, as it's unclear how the Microsoft clients will respond to that.

Anecdotally, the only easy-to-access client that can handle H264/EGFX and resize-on-the-fly is the Mac OS one. Windows is a fragmented mess. MSTSC appears to only support resize-on-the-fly in limited circumstances (but it does support H264/EGFX) and the Microsoft Store client supports resize-on-the-fly, but not H264. There is another Windows Desktop Client (MSRDC) that I haven't been able to test due to the fact that it only works on Azure systems with Active Directory. So, nearly all of my investigating and testing is done with FreeRDP and the Microsoft Mac OS client.

First revelation is that the note from Ogon is correct. You MUST delete and re-create the EGFX connection, and you must MAKE SURE the connection is shut down before you re-create a new one. It currently has a bug where sometimes, if the EGFX connection is not dropped properly, the session is now dead. There is a zombie connection on the Mac OS side and if a new connection is attempted to be created, the client can go into an infinite loop and leak nearly a gigabyte of memory. Not fun.

Second is that H264 is a delicate protocol. The Microsoft Mac OS client must receive the H264 keyframe before any incremental frames. If it doesn't receive the keyframe first after the EGFX connection is re-created, the session is again dead and can't be salvaged; you have to reconnect. Thus, the resizing workflow takes great pains to make sure that Frame 0 from the xorgxrdp backend is the first frame encoded (as a keyframe), that said frame is labeled as a keyframe, and that it's the first frame that is sent from the xrdp server to the client. This doesn't always work, and is continually being optimized. Currently, the "invalidate" signal is used to trigger this.

Third is that monitor descriptions matter. The XRDP server has to be very clear what monitor descriptions and surface ids it sends to the Microsoft client. Currently, everything is done with surface 0 and no client-side composition is happening. With Wayland, that could change.

The XRDP architecture above poses a problem, then. While the EGFX connection is established, we also have an asynchronous encoder running. Worse, the encoder's settings are resolution specific. What do we do with that encoder when the resizing state machine begins?

Special Topic: Deleting and re-creating the encoder in the EGFX workflow for resizing

My proof that we need to delete the encoder, wait for xrdp and its modules to resize, and then restart the encoder is one of proof by contradiction. This only applies if we have resizing in a state machine. Without a state machine or even without an EGFX state machine yes, we can get away with doing this and it does work.

But once we introduce an EGFX state machine (see above as to why this is needed), it becomes messy.

Let's assume that we delete and re-create the encoder in a one, single-threaded step. X is single-threaded, so this should work, right? To make this easier, let's also say we're resizing from full screen to smaller screen.

We have several choices of where to place this in the state machine:

  1. Before we resize xrdp/xorgxrdp
  2. Somewhere in the middle of resizing xrdp/xorgxrdp
  3. After resizing xrdp/xorgxrdp.

Try (1): The encoder will now receive, at least temporarily, frames that do not match the size it's created for because it is now created for a screen that is small, but xorgxrdp is still sending it larger screens while waiting for the EGFX connection to the client to be reconstituted. This is wrong, and could cause the encoder to send garbled data to the client in the best case, or crash in the worst case.

Try (2): Maybe between when xorgxrdp is resized and when xrdp is resized. Now the encoder may match what is being sent from xorgxrdp, but XRDP itself hasn't been resized, so if the encoder tries to use XRDP to send a frame of a different size into the XRDP network buffer that hasn't been resized yet, we could have issues.

Try (3): xrdp/xorgxrdp are now consistent, but all the while the encoder has been running, it's now received both a full screen and a smaller screen frame, so the issue isn't that the recreating shouldn't happen here, it's that we left it exposed to shifting memory earlier. To double check, I just tried switching this in my mainline_merge branch with EGFX and my torture test, it's not stable.

So, there is no place in the state machine where we can delete and re-create the encoder in ONE STEP.

Furthermore, H264 and EGFX with NVENC and Intel QuickSync (hardware acceleration) make this more complicated.

In the current master of xrdp, xorgxrdp captures software-video-driver's the memory into a shared memory buffer, it's encoded by xrdp, and then sent over the wire with the RDP protocol. On my Ubuntu system, when I query for this I use glxinfo | grep vendor and the video driver vendor is "SGI." This amounts to a RAM->RAM memory copy from xorgxrdp -> xrdp, not a huge order of magnitude change in performance.

Ah but this doesn't work with hardware accelerated video memory, at least, not if you want performance. What needs to happen is all the encoding needs to happen in a separate video memory buffer, everything from RGB -> YUV conversion and then H264 encoding. Jay wrote a version that used NVENC where the frame was copied from video memory -> RAM, back to video memory for Nvidia NVENC encoding, and then back to RAM for sending over the network, and it was slowing than CPU accelerated X264!

State Machine/Sequence Diagram

Resizing Sequence Diagram

Special Topic: Nvidia Driver Integration with the Xorg server system

Historically, Nvidia has been a trouble spot for Linux integration because their drivers aren't open source. When attempting to load Nvidia drivers into the virtual sessions, it would crash. jsorg71@ found a way around this. In his words:

Xorg has a built-in RandR support and there is something called an 'output'
In the output, the driver can store its own private pointer for its
own specific implementation.
Normally when NVidia driver defects a monitor, they create the randr
output and set up their private pointers.  All cool, when something
gets the private pointer, it's valid and they dereference it and do
whatever.
When xorgxrdp adds an output, I don't create this private.  I don't
even know what it is.  It's some internal structure they use.  Anyway
then later something dereferences it and they don't check for nil
because it should never be nil normally.  I can not fix this behavior,
NVidia would have to.

My work around.
Start up Xorg with the RandR extension disabled.  In fact, never use
any of the built- in RandR functions in Xorg.  Then after screen init
and the X session is up, add an extension called RANDR.  There is a
function for that.  It's the exact same name.  In this state, the
NVidia driver thinks RandR is disable but X client apps see the
extension and work as normal.  I just have to complete this
alternative RANDR extension.  We don't need most of the randr
functionality, just output list and those notifications.

That alternative extension is called "lrandr" (Local RandR) and is present in Nexarian's mainline merge branch. RandR is short for "Resize and Rotate," and is documented better elsewhere. The lrandr extension allows Gnome 3 to load. To my knowledge, this is the first time this type of integration has ever been possible and is a major win for the XRDP ecosystem. Before this, Nvidia only worked with window managers such as Mate and Cinnamon, which were not nearly as optimized for 3D acceleration. Anecdotally, Gnome 3 is 2x as fast as MATE in SxS 3D rendering.

The other way around this is a system known as VirtualGL, which is what most remote desktop systems currently use, from NoMachine to Chrome Remote Desktop to X2Go. This version is superior: No emulation, the driver is DIRECTLY loaded into the Xsession.

Special Topic: Intel libva Integration

Intel wrote libva for people to use QuickSync, and then didn't document it well. However, there is another library out there that does use it called libyami. To use, here are instructions from jsorg71:

yami_inf is a wrapper around libyami which is a wrapper around libva. Not saying it's going to stay this way, maybe we can later use libyami or libva directly. I was trying to follow the LoadLibrary / GetProcAddress thing that the NVidia encoder does.
Easiest way to get yami_inf is to clone
git clone https://github.com/jsorg71/builders.git
cd into builders/yami/omatic/ and run
./buildyami.sh --prefix=/opt/yami --disable-x11
It will put everything in /opt/yami
If you run buildyami.sh as regular user you might have to create /opt/yami as root and give yourself rights.
It will build and patch libva, libyami, etc
It would be great if you can get it working so far it's just me.

Note, this Intel xorgxrdp_helper is using EGL and dmabufs to pass video to the encoder where as the NVidia xorgxrdp_helper is using GLX and NVEnc accepts an OpenGL texture. In both cases, there is zero copy to system memory.

Future work to merge these technologies into devel

  • Logging needs to be updated to the latest paradigm used by XRDP.
  • Intel's libva integration needs to be streamlined -- Right now this is very complex to install and set up.
  • Need to select better names for xorgxrdp_helper and lrandr
  • Dynamic resizing needs to be tested with the xorgxrdp_helper and Intel integration.
  • YUV444 streaming needs to be enabled for all encoders as an option for higher-bandwidth, lower-latency connections.

Appendices

RDP client testing notes

Microsoft Clients

Non-Microsoft Clients

  • FreeRDP generally handles everything written above with ease (EGFX + Resizing). It's the most flexible client, and generally can't be relied upon to be a reference implementation as it's so tolerant of what might otherwise be a protocol error that would be enforced by other Microsoft clients. It works well on Linux with ffmpeg (OpenH264 only works well some of the time). ** However, FreeRDP on Mac OS is poorly supported. It relies on XQuartz, which runs a small Xserver inside of MacOS, which does not integrate with Mac OS spaces. ** FreeRDP also doesn't have good builds on Windows. There are ways to get a Windows build, but they are slapdash. YMMV

References

Neutrinolabs (XRDP) references:

Microsoft references:

FreeRDP references:

Misc

Add a custom footer
Add a custom sidebar
Clone this wiki locally