Virtual Desktop Service (xrdp)

Project blog: http://blog.inf.ed.ac.uk/squinney/tag/xrdp/

Intervention

The RDP servers need some routine housekeeping. Here's how to do that.

If someone reports a problem

Which server are they using?

If a user is having a problem on an xrdp server, which one are they using? If they're using their UUN.remote.inf.ed.ac.uk server then you can just ssh to it, e.g.:

ssh s2345678.remote.inf.ed.ac.uk

or you could just use host to find out which machine it is:

host s2345678.remote.inf.ed.ac.uk
s2345678.remote.inf.ed.ac.uk is an alias for blahblah.inf.ed.ac.uk
blahblah.inf.ed.ac.uk has IPv6 address ...
blahblah.inf.ed.ac.uk mail is handled by ...

They're still logged in

A lot of people just don't seem to realise that when you quit the Remote Desktop app, the DICE session doesn't logout - it stays running. This has a few consequences:

• A screensaver starts after a while. This can confuse people when they next login.
• When the session's AFS credentials expire, some processes (e.g. firefox) can get upset, and use a lot of CPU while trying and failing to get access to their files.
• Sometimes processes in abandoned sessions can get so upset that the session becomes unusable - but it's still running.

Abandoned sessions are auto-killed after two weeks.

"I logged in but it was all black."

If someone starts a session but they just see blackness instead of windows and a desktop, it's almost certainly a screensaver from their previous session. If they click on the blackness, a screensaver window should pop up and ask for their password. Hopefully they'll then be able to unlock the screensaver and resume that session. If that doesn't work, try the next suggestion.

Can't authenticate to xscreensaver

A few people have found that, when reconnecting to an existing session, their username and password are accepted by DICE, but that xscreensaver does not accept their password. Our assumption is that this is because of a mismatch between the user's keyboard mapping and the one that xscreensaver is using - so when the user types their password, the wrong characters are being sent. It helps to use a password which only has upper and lower case letters and numbers, as these don't move about so much between US and UK keyboards. Also, you can kill their xscreensaver process for them. (ksu to root, ssh to the server, then ps axuww | grep xscreensaver and pick their xscreensaver process out of all the others running on that machine; then simply kill it.)

How to kill all of a user's processes on a machine

If someone's session has become stuck or unusable, you can kill all of their processes on that machine. To do that, become root on that server then:

killall -v -u UUN

Alternatively, use

loginctl terminate-user UUN

UUN.remote.inf.ed.ac.uk

If someone is having problems on their remote server, you can just ssh to it:

ssh UUN.remote.inf.ed.ac.uk

A "remote" name is just a DNS alias, so you can use it like a hostname - login to it, start an xrdp session on it, etc. Everybody has one, but since they all point to the same few shared servers, you can login to anybody else's as well as your own. Also, you can rely on each "remote" alias staying the same - they shouldn't switch about and point to other machines.

General server health

How to tell if something's wrong

One way is to check the load average on each server. This command can help:

/afs/inf.ed.ac.uk/group/cos/utils/rdphealth

If a server's load average is way into double figures then something's probably wrong.

Look more closely

Having found a machine which is being hammered, ksu to root then ssh to the machine, then run top -c. This should tell you which processes are using the most CPU time. If you find a process using near 100% CPU time (sometimes more, if it has lots of threads) here's what to do:

• Is the machine otherwise quiet? Maybe leave it alone for now.
• Is the load average high? First try limiting the CPU which that process's session is allowed to use. Copy the PID of the process, then, still as root, leash PID . This should limit that process's entire session so that it's allowed to use a maximum of half a CPU at any one time. Go back to top -c to check, and repeat for other troublesome processes.
• leash doesn't work on some processes, because they're no longer associated with a session. In this case you might want to kill them. You can do this from inside top by pressing k. This will first prompt you for the PID of the process to kill (it defaults to the first process in the display, and this is usually the one you want). When you press return, it'll ask for the signal you want to send to the process. The default of 15 usually works, but if that doesn't make the process exit, you can try again with a signal of 9. Obviously, be careful not to kill root processes.

Educate

If someone is persistently running CPU-heavy things on a remote desktop server, you could mail them to ask them not to. It might help to explain that every taught student has a UUN.lab.inf.ed.ac.uk alias they can ssh to, and they can do their coursework there. There's a help page at http://computing.help.inf.ed.ac.uk/remote-lab.

Crashes, hangs and reboots

Even with regular attention, a popular remote desktop server can become unusable after a few weeks of enthusiastic use. We've seen two problems occur multiple times. The first has been a server crashing with an AFS-related kernel crash. This should be solved now. The second is that certain commands start to hang indefinitely. The only cure we've found for this behaviour is to reboot the server.

Server

The server-side configuration is closely based on a prototype service developed by SEE. It uses xrdp project to provide remote access to the X server. In front of that is a proxy which uses HAProxy to load balance across a cluster of machines. Up to now we have only had a single "staff" server and a single "student" server but with there being a lot more interest in supporting distance learning we clearly are going to have to expand our provision and this should provide the ideal solution.

xrdp

Based on the work of FreeRDP and rdesktop, xrdp uses the remote desktop protocol to present a GUI to the user. The goal of this project is to provide a fully functional Linux terminal server, capable of accepting connections from rdesktop, freerdp, and Microsoft's own terminal server / remote desktop clients. Xrdp uses Xvnc or X11rdp to manage the X session.

HAProxy

A free, very fast and reliable solution offering high availability, load balancing, and proxying for TCP and HTTP-based applications.

SEE headers (needs EASE authentication):

• see_policies/options/xrdp.h
• see_policies/options/haproxy.h
• see_policies/options/rdp-gateway.h

LCFG headers (closely based on SEE):

• lcfg/options/xrdp.h
• lcfg/options/haproxy.h
• lcfg/options/rdp-gateway.h

DICE headers:

• dice/options/xrdp.h
• dice/options/haproxy.h
• dice/options/rdp-gateway.h
• dice/options/external-xrdp-server.h

We have these servers:

• xrdp.inf.ed.ac.uk is the general service. It's on two machines, lute and archlute. An "haproxy" service sends users to one or other of them.
• staff.xrdp.inf.ed.ac.uk is the staff, postgrad and visitor server. It's on waterloo, an old Dell PowerEdge R720.
• From time to time there's a test server on test.xrdp.inf.ed.ac.uk.

Clients

One of our biggest issues with the previous NX service was the lack of decent client software for all platforms. The open-source versions were rarely sufficiently well maintained so we relied on the closed-source versions from NoMachine, newer versions of which no longer support the old protocol. With RDP it is all a lot simpler as it's a "standard" protocol on Windows and MacOSX.

Linux

Various clients are available, the current best appears to be Remmina which is available as a package for at least Fedora, Debian, Ubuntu.

Windows

The standard Microsoft Remote Desktop application (which is free to install) should work. See this page for details.

MacOSX

Microsoft provide a free application in the App Store

SEE have put together some basic notes on how to configure various RDP clients.

Further Ideas

• Look at fail2ban
• Incorporate into BuzzSaw and Log cabin reports
• Look at sound support - https://github.com/neutrinolabs/xrdp/wiki/Audio-Output-Virtual-Channel-support-in-xrdp
• If a user has been allocated a DICE desktop could we forward them directly to that machine?

remote.inf.ed.ac.uk, lab.inf.ed.ac.uk

lab.inf comes from a netgroup. To make changes:

• Set netgroup HOST_dicelab to include the hosts you want. To remove a host from this, give it

  !roles.netgroups mREMOVE(dicelab)

  then wait up to an hour until the host no longer shows up in the output of

  netgroup -M HOST_dicelab

  on oramo.
• Set login/lab/alias to have the set of users that you want, by adding/removing/whatever entitlements.
• As root on the DNS master oramo, run /usr/lib/lcfg/dns/generateLabInf to generate a new zonefile and swap it into use.

For remote.inf rfe dns/remoteHostList instead, then run the generateRemoteInf script.