Sane Session-Switching

In a previous article I talked about the history of VT switching. Created as a simple way to switch between text-mode sessions it has grown into a fragile API to protect one testosterone monster (also called XServer) from another. XServers used to poke in PCI bars, modified MMIO registers and messed around with DMA controllers. If they hauled out the big guns, it was almost absurd to believe a simple signal-flinging VT could ever successfully negotiate. Fortunately, today’s XServer is a repentant sinner. With common desktop hardware, all direct I/O is done in the kernel (thanks KMS!) and chances of screwing up your GPUs are rather minimal. This allows us to finally implement proper device-handover during session-switches.

If we look at sessions at a whole, the XServer isn’t special at all. A lot of session-daemons may run today that provide some service to the session as a whole. This includes pulseaudio, dbus, systemd –user, colord, polkit, ssh-keychain, and a lot more. All these daemons don’t need any special synchronization during session-switch. So why does the XServer require it?

Any graphics-server like the XServer is responsible of providing access to input and graphics devices to a session. When a session is activated, they need to re-initialize the devices. Before a session is deactivated, they need to cleanup the devices so the to-be-activated session can access them. The reason they need to do this is missing infrastructure to revoke their access. If a session would not cleanup graphics devices, the kernel would prevent any new session from accessing the graphics device. For input devices it is even worse: If a session doesn’t close the devices during deactivation, it would continue reading input events while the new session is active. So while typing in your password, the background session might send these key-strokes to your IRC client (which is exactly what XMir did). What we need is a kernel feature to forcibly revoke access to a graphics or input device. Unfortunately, it is not as easy at it sounds. We need to find some-one who is privileged and trusted enough to do this. You don’t want your background session to revoke your foreground session’s graphics access, do you? This is were systemd-logind enters the stage.

systemd-logind is already managing sessions on a system. It keeps track on which session is active and sets ACLs in /dev to give the foreground session access to device nodes. To implement device-handover, we extend the existing logind-API by a new function: RequestDevice(deviceNode). A graphics-server can pass a file-system path for a device-node in /dev to systemd-logind, which checks permissions, opens the node and returns a file-descriptor to the caller. But systemd-logind retains a copy of the file-descriptor. This allows logind to disable it as long as the session is inactive. During a session-switch, logind can now disable all devices of the old session, re-enable the devices of the new session and notify both of the session-switch. We now have a clean handover from one session to the other. With this technology in place, we can start looking at real scenarios.

1) Session-management with VTs

Session management with VTs and logind

Based on the graphs for VT-switching, I drew a new one considering logind. VTs are still used to switch between sessions, but sessions no longer open hardware devices directly. Instead, they ask logind as described above. The big advantage is that VT-switches are no longer fragile. If a VT is active, it can be sure that it has exclusive hardware-access. And if a session is dead-locked, we can force a VT-switch and revoke their device-access. This allows to recover from situations where your XServer hangs without SSH’ing from a remote machine or using SysRq.

2) Session-management without VTs

Pure logind session management

While sane VT-switching is a nice feature, the biggest win is that we can implement proper multi-session support for seats without VTs. While previously only a single session could run on such seats, with logind device-management, we can now support session-switching on any seat.

Instead of using VTs to notify sessions when they are activated or deactivated, we use the logind-dbus-API. A graphics-server can now request input and graphics devices via the logind RequestDevice API and use it while active. Once a session-switch occurs, logind will disable the device file-descriptors and switch sessions. A dbus signal is sent asynchronously to the old and new session. The old session can stop rendering while inactive to save power.

3) Asynchonous events and backwards-compatibility

One thing changes almost unnoticed when using RequestDevice. An active graphics-server might be almost about to display an image on screen while a session-switch occurs. logind revokes access to graphics devices and sends an asynchronous event that the session is now inactive. However, the graphics-server might not have received this event, yet. Instead, it tries to invoke a system-call to update the screen. But this will fail with EACCES or EPERM as it doesn’t have access to it, anymore. Currently, for most graphics servers this is a fatal error. Instead of handling EACCES and interpreting it as “this device is now paused”, they don’t care for the error code and abort. We could fix all the graphics-servers, but to simplify the transition, we introduced negotiated session-switches.

Whenever logind is asked to perform a session-switch, it first sends PauseDevice signals for every open device to the foreground graphics-server. This must respond with a PauseDeviceComplete call to logind for each device. Once all devices are paused, the session-switch is performed. If the foreground session does not respond in a timely manner, logind will forcibly revoke device access and then perform the session-switch, anyway.

Note that negotiated session-switches are only meant for compatibility. Any graphics-server is highly encouraged to handle EACCES just fine!

All my local tests ran fine so far, but all this is still under development. systemd patches can be found at github (frequently rebased!). Most tests I do rely on an experimental novt library, also available at github (I will push it during next week; this is only for testing!). Feedback is welcome! The RFC can be found on systemd-devel. Now I need a day off..

Happy Switching!

Advertisement

How VT-switching works

Having multiple sessions on your system in parallel is a quite handy feature. It allows things like Fast User Switching or running two different DEs at the same time. Especially graphics developers like it, because they can test-run their experimental XServer/weston on the same machine they develop on.

To understand how it works, we need the concept of a session. See my previous introduction into session-management if you’re not familiar with it. I expect the reader to be familiar with basic session-management concepts (i.e., seats, systemd-logind, login-sessions, user-sessions, processes and daemons in a session).

1) Traditional text-mode VT-switching

Virtual terminals were introduced with linux-0.12. It’s the origin of multi-session support on linux. Before this, linux only supported a single TTY session (which was even available in the first tarball of linux-0.01). With virtual terminals, we have /dev/tty<num> devices, where <num> is between 1 and 63. They are always bound to seat0 and every session on seat0 is bound to a single VT. That means, there are at most 63 live sessions on seat0. Only one of them is active at a time (which can be read from /sys/class/tty/tty0/active).

Session management in text-mode

The kernel listens for keyboard events and switches between VTs on ctrl+alt+Fx shortcuts. Alternatively, you can issue a VT_ACTIVATE ioctl to politely ask the kernel to switch sessions.

In the old days, all sessions ran in text-mode. In text-mode, a process can get keyboard input by reading from a VT and can write onto the screen by writing to a VT. Some rather ugly control-sequences are supported to allow colors or other advanced features. The kernel interprets these and instructs the graphics hardware to print the given text.

Important to note is that in text-mode sessions don’t have direct hardware access. The kernel merges all keyboard events into one stream and a session cannot tell which device it came from. In fact, it cannot even tell how many devices there are. Same for graphics devices. As a VT has only a single output stream, there is only a single screen to write to. Multi-head support is not available. Neither are any advanced graphics-operations. But this allows the kernel to serialize access to hardware devices. Only the active VT gets input events and only the buffer of the active VT is displayed on the screen. No resource-conflicts can occur.

While there are ways to detect when a session is activated/deactivated, in text-mode a session normally doesn’t care. It just stops receiving keyboard input. If the session writes to the VT while deactivated, it will affect the internal buffer of the VT, but not the screen. Only the buffer of the active VT is shown on screen.

2) Session-switching in graphics-mode

Session management in gfx-mode

Very soon it became clear that text-mode is not enough. We wanted more! That’s when the VT graphics mode was introduced. Graphics mode doesn’t change the setup, we still have 63 VTs and each session is bound to a VT. But a VT can now be switched from text-mode KD_TEXT into graphics-mode KD_GRAPHICS (via KDSETMODE ioctl). This doesn’t do anything spectacular. Really! The only effect is it disables the kernel-internal graphics routines. As long as a VT is in graphics-mode, the kernel will not instruct the graphics hardware to display the VT on screen. Once it is reset to text-mode and the VT is active, the kernel will display it again.

So the graphics-mode itself is useless. But at the same time, the kernel started providing separate interfaces to input and graphics devices. So while a VT is in graphics-mode, a session can access the graphics devices directly and render whatever they want. It can also ignore input from the VT and instead read input-events directly from the input-event interfaces. This is how the XServer works today.

But it is pretty obvious that this becomes problematic during session switches. If the kernel switches away from a graphics VT, the session needs to release the graphics hardware before the new session can be activated. Otherwise, the new session is active, but you still get the images from the old one. Or worse, it might flicker between the images of both sessions. Without kernel-mode-setting it might even hang your graphics hardware.

(A similar interface to KDSETMODE exists for keyboards with KDSKBMODE.)

Unfortunately, the kernel has no interface to forcibly revoke graphics or input access. So VTs were extended by the VT_SETMODE ioctl. A process can issue VT_SETMODE on a VT and pass two signal-numbers (usually SIGUSR1 and SIGUSR2). If the kernel wants to perform a VT-switch, it sends one such signal to the active VT. This VT can cleanup resources, stop using graphics/input devices and acknowledge the VT-switch via the VT_RELDISP ioctl. If the process dead-locked and doesn’t acknowledge the request, the VT-switch will not happen! This is why a crashed XServer can hang your system. But if it correctly issues the VT_RELDISP ioctl, the kernel will perform the VT-switch. Once the kernel switches back to the given VT it sends the second signal as notification that it is now active again.

However, on every VT (precisely, on every session) only a single process can call VT_SETMODE. This already shows that the concept is flawed. For example, if the XServer takes the VT in posession for graphics and input devices, another audio-server in the session couldn’t do the same for audio-hardware. This applies to all other devices. An alternative involving logind is discussed in a followup article.

3) Seats without VTs

We discussed that VTs are always bound to seat0. So if you run a session on a seat other than seat0 or if you disabled VTs entirely, then the situation becomes pretty simple: no multi-session support is available. This is enforced by systemd-logind so the active session can run without interruptions.

Also important to note is that there is no text-mode. A session must run in graphics mode as the kernel facility to interpret text-commands is bound to VTs.

Session management without VTs

4) logind integration

As we discussed in the previous article, today systemd-logind tracks and manages sessions. But how does this integrate with VTs? VTs pre-date systemd by years, so to preserve backwards-compatibility, we need to keep the infrastructure as it is. That’s why logind watches /sys/class/tty/tty0/active for changes. Once a VT switch happens, logind notices it and marks the old session as inactive and the new as active. It also adjusts ACLs in /dev to keep access-restrictions in sync with the active session. However, this cannot be done properly without a race-condition. Therefore, logind provides a new dbus-API to replace the ageing VT API. New session-daemons are advised to use it in favor of VTs.

Session-Management on Linux

One thing I always admired on linux is fast session-switching. A simple key-press (ctrl+alt+Fx) and I can ditch my current session to switch to another one. While in principle session-interaction is fairly simple, you can easily get it wrong (as can be seen with XMir). But how exactly does session management work today? And how did systemd-logind change this situation?

First, we need to define a session. A session is a group of processes running under control of a single user. Only a single session is active on a system (more precisely on a seat; discussed below) and a user can only interact with the active (or foreground) session. A session can start and stop processes, but it cannot escape from the session. In other words, all started processes will always belong to the session they were started in. Only system-daemons (which are not attached to a session) can spawn new sessions.

A session usually has different user-daemons which provide services for the session. For instance, the X-Server provides graphics access to other session-processes, pulseaudio provides audio-access. Then there is a central process which controls and monitors a session. Whenever a session is spawned, this process is the first process that is created. It initializes the session, spawns other management processes and optionally restarts them if they crash. It might also exit once the session was initialized if no process-monitoring is required. An example would be gnome-session.

1) Session Management

Historically, every process on a system has a session-id which defines the session a process belongs to. The setsid() system call allows to create a new session. However, with systemd-logind a new more-versatile concept was introduced. To understand this, we need to first look at seats.

Seats

Multi-seat setup (source: Wikipedia)

On normal systems only a single seat exists (called seat0). But more seats can be created on request. A seat is a virtual object that describe a physical interface to a system. The combination of a monitor plus keyboard and mouse form a usual seat. One user can interact with the system through this seat. If you attach another monitor plus keyboard and mouse, another user can interact with the system in parallel via this new seat (eg., called seat1). It is important to note that these seats are independent of each other. The pictures you get on the monitors are different and keyboard input from one seat doesn’t affect the other.

Any device connected to your system can be attached to a seat, but only to one seat at a time! If a device is attached to seat, it cannot be used by processes running on another seat. Obviously, input devices, sound-devices and monitors are usually attached to a seat. Network-interfaces, for example, can be left unattached so all seats can share an internet-connection. It is up to the system-administrator to decide which devices to attach to which seat, or whether to leave them as global devices that can be shared.

Bootup

During system-boot, systemd starts several daemons to manage a system. All these daemons are global and not attached to a seat or session. This also means, these daemons must not access devices attached to a seat. Among these daemons is also systemd-logind, which is responsible for session and seat management.

To allow user interaction, systemd needs to automatically spawn a new session for each seat. This is usually a login-session (like gdm, kdm or LightDM) but might also be configured as auto-login and spawn directly a user-session like gnome-session. In case of a login-session, the greeter draws login and password fields and after verifying the input, it instructs systemd to spawn a user-session. Note that the greeter cannot start the user-session itself as it already runs in a session (you cannot escape your session, remember?).

But what exactly does systemd do to control sessions?

Sessions

Via kernel interfaces (namely cgroups) systemd can reliably track processes and their childs. Whenever systemd is instructed to start a new session, it first verifies the caller is allowed to do that and then it spawns the first process for the session. Internally, it attaches a new session-id to this process, saves the seat this session is run on and some other maintenance data. Whenever this session process creates new processes, systemd can use cgroups to track these. So it always knows which session a process belongs to.

The first process for a session is responsible of initializing the session. In case of gnome-session, it spawns the X-Server, some gnome-specific maintenance daemons and then monitors the session in case something goes wrong. You can now interact with this session (normally through the X-Server) and do your daily work.

A session can explicitly instruct systemd to close itself. However, this will only mark it as dead. Once all processes of a session exited, the session is automatically cleaned up and removed.

2) Case-Study: Text-mode sessions

Text-mode interface (source: Wikipedia)

As a concrete non-trivial scenario, I’ll explain how text-mode sessions integrate with this. First, we need to know that virtual-terminals are basically a combination of input devices and a monitor. They can be accessed via /dev/tty<num> (where <num> is between 1 and 63; the other /dev/ttyXY devices are no VTs!). If you read from them, you get input from connected keyboards, if you write to them, the kernel displays it as text on the monitor. Some rather complex control-sequences are available to do colors or more, but that’s beyond the scope. VTs are always bound to seat0. So only sessions on this seat can use VTs (which means, classic text-mode is only available on seat0).

Text-mode sessions are special in that they are not spawned by a login-session. systemd-logind spawns /bin/agetty right during boot for each VT (you can configure how many of the 63 possible should be started). agetty is running as system daemon and not in a session! However, it is bound to seat0, obviously. agetty initializes the VT and runs /bin/login. login then writes a greeter to the VT and reads username plus password from it. As you might notice, this is problematic as it accesses devices attached to a seat even though it’s not running in a session. But due to the design of VTs, this cannot be easily avoided.

Once login verified the user-input, it instructs systemd to start the given user-session in text-mode. It looks up the initial process in /etc/passwd (let’s assume it is /bin/bash) and runs it. So bash is the controlling process in this session. It also substitutes the X-Server in that it provides user-interaction (in text mode rather than graphics). It allows to switch between different processes (ctrl+Z, fg, bg and jobs) and allows starting other session-daemons during initialization (via .bashrc).

So while text-mode feels a lot different, it is in fact very similar to graphics mode and is internally handled almost equally.

3) Multi-Session

We now understand what a session is, how they are created and how a user interacts with them. However, we haven’t discussed what happens if there are multiple sessions on a seat.

On current systems, multiple sessions are only allowed on seat0 if VTs are enabled (which is usually the case). That means, on all other seats, once you spawned a session, you have to stick with it. You cannot ctrl+alt+Fx away from the session (the reason for this is historical and we’re about to change this). However, you can close the session and then start a new one.

But if we assume our seat supports multiple sessions, how does that work? systemd-logind is responsible to manage and track sessions. Therefore, it keeps an Active attribute for each session. It’s a boolean value that defines whether the session is currently active or not. systemd-logind takes care that always at most one session is active. If no session runs on a seat, no session might be active (also in other situations which we ignore here as it’s an implementation detail). Additionally, systemd-logind sets the correct access-restrictions on devices attached to a seat so only the active session can access these devices. This guarantees that no background session interferes with foreground activity.

To switch between sessions, each session has to listen for special keyboard input (normally ctrl+alt+Fx) and instruct systemd-logind to switch to another session once it’s pressed. Today, this is implemented in the X-Server or, if no x-server runs in the session, by the underlying VT. You can also send a dbus call to systemd-logind directly to perform the session-switch (see loginctl activate <sid>).

So for proper multi-session support, a login-session spawns a user-session during login but stays active in background. A user can now switch between the user-session and the login-session and optionally spawn additional session by logging in again. Session processes can listen for dbus signals from systemd-logind to be notified when they are activated or deactivated. This allows them to go asleep while being in background and start up again once being put in foreground.

For an in-depth view on session-switching, see this followup article.