Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Running sessions "lost" if xrdp-sesman restarts #800

Open
ben-cohen opened this issue Jul 6, 2017 · 16 comments · May be fixed by #3346
Open

Running sessions "lost" if xrdp-sesman restarts #800

ben-cohen opened this issue Jul 6, 2017 · 16 comments · May be fixed by #3346

Comments

@ben-cohen
Copy link
Contributor

ben-cohen commented Jul 6, 2017

If xrdp restarts then the new process is not aware of any sessions created by the old process, so you can't re-attach to them and they aren't listed by xrdp-sesadmin. Would it be possible for the new process to "discover" them and populate g_sessions?

Test case:

  1. Connect to a new xorgxrdp (Xorg) session.
    xfreerdp /v:localhost /u:<user> /p:<password> &

  2. List sessions using xrdp-sesadmin. The session created above will be listed.
    xrdp-sesadmin -u=<user> -p=<password> -c=list

  3. Restart xrdp. Note that the existing sessions are not disconnected.
    sudo service xrdp restart

  4. List sessions using xrdp-sesadmin.
    xrdp-sesadmin -u=<user> -p=<password> -c=list
    Expected behaviour: the session created above should be listed.
    Observed behaviour: "No sessions." (or, prior to xrdp-sesadmin: fix error when there are no sessions #797, "Error getting session list.")

  5. Disconnect the RDP client (without logging out) from the RDP session.

  6. Attempt to reconnect to the existing session.
    xfreerdp /v:localhost /u:<user> /p:<password> &

(Workaround: to reconnect to the "orphaned" sessions you can set up xvnc to view the appropriate X display.)

Note that the new xrdp would somehow have to obtain the session_item info from each session it discovers. Also the new xrdp wouldn't be the parent of the sesman for each session so it wouldn't receive SIGCHLD when the session ends (i.e. it would have to poll). I would suggest solving both of these using a shared mmap() object.

@metalefty
Copy link
Member

Actually, running session will be lost when xrdp-sesman daemon restarts. Restarting only xrdp daemon won't lost session info.

@ben-cohen ben-cohen changed the title Running sessions "lost" if xrdp restarts Running sessions "lost" if xrdp-sesadmin restarts Jul 7, 2017
@ben-cohen
Copy link
Contributor Author

Thanks, yes! I have corrected the issue title.

@metalefty metalefty changed the title Running sessions "lost" if xrdp-sesadmin restarts Running sessions "lost" if xrdp-sesman restarts Jul 8, 2017
@metalefty
Copy link
Member

I've corrected the title again :)

@ben-cohen
Copy link
Contributor Author

Oops, thanks!

To elaborate on my suggestion above, xrdp-sesman could open a file say /var/run/xrdp-sesman.sessions and mmap it into memory. It can store each struct session_item (from the g_sessions linked list) in an array in that region rather than allocated by malloc().

Now if xrdp-sesman restarts, the new process can mmap the file and discover the existing sessions. Using mmap has the advantage that the contents can be shared with other processes, so it can be used to communicate between the main xrdp-sesman instance and its session instances - for example, to update the user's idle time. (A shared pthread mutex can be used to synchronise access.)

To determine if a session instance of xrdp-sesman is still running the main instance can no longer listen for SIGCHLD; instead each session could have a "heartbeat" timestamp that it updates in its mmap struct once a minute. If that timestamp is a few minutes out of date then we can assume that the session has ended. (Of course, if the pid is gone or has the wrong name then we can also assume that the session has ended, but checking shared memory is easier than checking /proc. There are some other ways to detect other processes exiting but they aren't portable.)

@speidy
Copy link
Member

speidy commented Jul 13, 2017 via email

@metalefty
Copy link
Member

Overall, the feature is nice and I'm very interested in :)

@ben-cohen
Copy link
Contributor Author

@speidy - That doesn't happen in practice. If it was intended to do that then this issue is fairly pointless! Is there a reason why it should kill the sessions on exit or is it just that it can't yet re-discover them?

Test case:

  1. Connect and log in.
    rdesktop localhost &

  2. Stop xrdp.
    sudo service xrdp restart

Observed: the xrdp session is still connected and running. (But the new xrdp-sesadmin doesn't know about it as described above.)

The sessions are not killed because:

  1. When the daemon instance starts it sets g_pid to the current process id, and installs signal handlers including sig_sesman_shutdown() for SIGINT and SIGTERM. These are inherited by the session instances when they are forked, and not changed thereafter.

  2. service xrdp restart calls /etc/init.d/xrdp which uses start-stop-daemon to send SIGINT to the xrdp-sesadmin daemon instance. It doesn't send anything to the session instances.

  3. The daemon instance handles SIGINT using sig_sesman_shutdown(). That calls session_sigkill_all(), which sends SIGTERM to each of the session instances.

  4. The session instance handles SIGTERM using sig_sesman_shutdown(). But the process id doesn't match g_pid, which is the daemon instance's process id, so it returns without doing anything (except unhelpfully logging "shutting down sesman 1"), but because the signal was handled it continues running.

@speidy
Copy link
Member

speidy commented Jul 13, 2017 via email

@speidy
Copy link
Member

speidy commented Jul 13, 2017

@ben-cohen I think your'e right.
Thats actually only happens in foreground mode, where you press Ctrl+C so the signal SIGINT is being sent to the forks (sessions) as well.
I like that feature and idea.

+1

@ben-cohen
Copy link
Contributor Author

@speidy I've just come to the same conclusion about Ctrl+C sending SIGINT to the whole process group. Thanks, I'll try it then...

ben-cohen added a commit to ben-cohen/xrdp that referenced this issue Jul 16, 2017
Allow xrdp-sesman to discover sessions still running that were created
by a previous xrdp-sesman process.

Implement this using a file-backed shared mmap region, visible to both
the daemon and session instances of xrdp-sesman.  Add a heartbeat
timestamp updated by the session instance so that the daemon instance
can infer whether the sessions in the mmap region are current or stale.

The shared memory can also be used to pass other data between daemon and
session instances, for example the session's idle time.

Add locking around access to g_sessions and the mmap region.  This is a
PTHREAD_PROCESS_SHARED lock which protects shared memory used by
separate processes.  Defining DEBUG_SESSION_LOCK enables logging to help
debug locking bugs.

Notes:

1. The number of sessions is limited by the size of the array in shared
memory, SESMAN_SHAREDMEM_MAX_SESSIONS.  This could be made dynamic
instead by growing the file and the mmap region.

2. If sesshm_try_open_existing_shm() finds a shm file but it looks
wrong, it creates a new one.  Perhaps it should instead exit with an
error?

3. In sesshm_thread() if the session xrdp-sesman notices that it has
been removed from the daemon's list it exits.  It should kill the X
server and/or window manager first.

4. This doesn't yet update idle times.  I need to work out how to get
this information from the X server.

5. This uses an array of sessions in the mmap region so the linked list
g_sessions might be redundant now.

6. Defining DONT_USE_SHM will let xrdp-sesman run without creating or
trying to use the mmap file.  This could be removed in the future.

7. I think the locking fixes an theorised bug where a session can be
removed from the g_sessions linked list in session_kill() (called from
the signal handler) at the same time as a session is added in
session_start_fork().
ben-cohen added a commit to ben-cohen/xrdp that referenced this issue Jul 16, 2017
Allow xrdp-sesman to discover sessions still running that were created
by a previous xrdp-sesman process.

Implement this using a file-backed shared mmap region, visible to both
the daemon and session instances of xrdp-sesman.  Add a heartbeat
timestamp updated by the session instance so that the daemon instance
can infer whether the sessions in the mmap region are current or stale.

The shared memory can also be used to pass other data between daemon and
session instances, for example the session's idle time.

Add locking around access to g_sessions and the mmap region.  This is a
PTHREAD_PROCESS_SHARED lock which protects shared memory used by
separate processes.  Defining DEBUG_SESSION_LOCK enables logging to help
debug locking bugs.

Notes:

1. The number of sessions is limited by the size of the array in shared
memory, SESMAN_SHAREDMEM_MAX_SESSIONS.  This could be made dynamic
instead by growing the file and the mmap region.

2. If sesshm_try_open_existing_shm() finds a shm file but it looks
wrong, it creates a new one.  Perhaps it should instead exit with an
error?

3. In sesshm_thread() if the session xrdp-sesman notices that it has
been removed from the daemon's list it exits.  It should kill the X
server and/or window manager first.

4. This doesn't yet update idle times.  I need to work out how to get
this information from the X server.

5. This uses an array of sessions in the mmap region so the linked list
g_sessions might be redundant now.

6. Defining DONT_USE_SHM will let xrdp-sesman run without creating or
trying to use the mmap file.  This could be removed in the future.

7. I think the locking fixes an theorised bug where a session can be
removed from the g_sessions linked list in session_kill() (called from
the signal handler) at the same time as a session is added in
session_start_fork().
@ben-cohen
Copy link
Contributor Author

I have created PR #819 for this. Let me know if you find any bugs or have any comments.

ben-cohen added a commit to ben-cohen/xrdp that referenced this issue Nov 26, 2019
Allow xrdp-sesman to discover sessions still running that were created
by a previous xrdp-sesman process.

Implement this using a file-backed shared mmap region, visible to both
the daemon and session instances of xrdp-sesman.  Add a heartbeat
timestamp updated by the session instance so that the daemon instance
can infer whether the sessions in the mmap region are current or stale.

The shared memory can also be used to pass other data between daemon and
session instances, for example the session's idle time.

Add locking around access to g_sessions and the mmap region.  This is a
PTHREAD_PROCESS_SHARED lock which protects shared memory used by
separate processes.  Defining DEBUG_SESSION_LOCK enables logging to help
debug locking bugs.

Notes:

1. The number of sessions is limited by the size of the array in shared
memory, SESMAN_SHAREDMEM_MAX_SESSIONS.  This could be made dynamic
instead by growing the file and the mmap region.

2. If sesshm_try_open_existing_shm() finds a shm file but it looks
wrong, it creates a new one.  Perhaps it should instead exit with an
error?

3. In sesshm_thread() if the session xrdp-sesman notices that it has
been removed from the daemon's list it exits.  It should kill the X
server and/or window manager first.

4. This doesn't yet update idle times.  I need to work out how to get
this information from the X server.

5. This uses an array of sessions in the mmap region so the linked list
g_sessions might be redundant now.

6. Defining DONT_USE_SHM will let xrdp-sesman run without creating or
trying to use the mmap file.  This could be removed in the future.

7. I think the locking fixes an theorised bug where a session can be
removed from the g_sessions linked list in session_kill() (called from
the signal handler) at the same time as a session is added in
session_start_fork().
@arnonerba
Copy link

I discovered today that the systemd unit files for xrdp.service and xrdp-sesman.service on Ubuntu 20.04 bind the two services together. There doesn't seem to be a way to restart xrdp without also restarting xrdp-sesman on that platform, short of rewriting the unit files.

Any updates on this issue? I was going to reply to @matt335672 in #819, but wanted to avoid adding more comments to a possibly dead PR.

@matt335672
Copy link
Member

Hi @arnonerba

I'll split your question into two bits; firstly, this issue under discussion, and secondly a possible way forward for your particular Ubuntu issue.

As regards the current issue, there's quite a lot to do before we get near this.

The current plan is to revamp and simplify all the inter-process comms used by xrdp. At the moment I'm looking at the link between xrdp and xrdp-sesman. The output from this work will allow us to split sesman into two bits - a session header process for each session, and an overall controller. The headers will maintain virtually all the state for each session connection, and the controller will then be little more than a broker which can easily be restarted. It's a similar model to the one used by ssh. When the controller is restarted, any necessary state info can be passed back up from the session headers to the controller.

I'd initially started at the other end, as I was hoping for a quick fix to #1684 which is relatively important. However, this turned out to be a bad approach. Solving the harder problems first is likely to end up with a better result in the long run.

I'm happy to talk more about this. I'm aware I may not have explained the above very well. We are looking at it, but as with most projects we're having to keep a lot of plates spinning at the same time with fairly limited resources.

As regards the Ubuntu unit files, you don't have to re-write these - you can provide overrides pretty easily straightforwardly which do what you want. See systemd.unit(5) an the details on the edit sub-command in systemctl(1) for more info.

Getting the Ubuntu system files changed, requires talking to the Ubuntu team. As a downstream project, our links for communication with Ubuntu/CentOS, etc are the same as everyone else's - raising a fault on Launchpad (Ubuntu) or Bugzilla (CentOS). Don't expect a quick response to this process!

Is that useful information?

@arnonerba
Copy link

Thanks, @matt335672! I appreciate the quick and detailed response. That makes a lot of sense, and it's good to hear that this is still being worked on along with everything else.

I'll think about opening a bug on Launchpad. Do you think it makes sense to treat xrdp.service and xrdp-sesman.service completely separately?

@matt335672
Copy link
Member

Personally I think the services should be separate, as xrdp doesn't need sesman in some configurations. However, there's a complication which Ubuntu has brought on themselves for what are actually very good reasons.

The Ubuntu packaging guys have set up the xrdp service to run as a non-privileged user, which is a much more secure way of doing things. However, this means they have to jump through a couple more hoops to get the services started. There's a bit of info on this on the wiki.

From what I found when I looked into this, there's no reason why the services need to be coupled, but I could be wrong.

If you're interested in this, I suggest you try editing the service unit descriptions and see how it goes. If you get something useful, please add the changes you've made to this PR as well as raising them in Launchpad. That will be very useful for the rest of the user community.

Let me know if you need a hand with any of this.

@arnonerba
Copy link

Hi Matt,

Thanks for the extra analysis. I'm realizing that I'm not going to have the chance to experiment with the unit files anytime soon. If I can find some time to work on this, I'll be sure to report back here.

@matt335672 matt335672 linked a pull request Dec 16, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants