Tag Archives: virtual terminal

Sane Session-Switching

In a previous article I talked about the history of VT switching. Created as a simple way to switch between text-mode sessions it has grown into a fragile API to protect one testosterone monster (also called XServer) from another. XServers used to poke in PCI bars, modified MMIO registers and messed around with DMA controllers. If they hauled out the big guns, it was almost absurd to believe a simple signal-flinging VT could ever successfully negotiate. Fortunately, today’s XServer is a repentant sinner. With common desktop hardware, all direct I/O is done in the kernel (thanks KMS!) and chances of screwing up your GPUs are rather minimal. This allows us to finally implement proper device-handover during session-switches.

If we look at sessions at a whole, the XServer isn’t special at all. A lot of session-daemons may run today that provide some service to the session as a whole. This includes pulseaudio, dbus, systemd –user, colord, polkit, ssh-keychain, and a lot more. All these daemons don’t need any special synchronization during session-switch. So why does the XServer require it?

Any graphics-server like the XServer is responsible of providing access to input and graphics devices to a session. When a session is activated, they need to re-initialize the devices. Before a session is deactivated, they need to cleanup the devices so the to-be-activated session can access them. The reason they need to do this is missing infrastructure to revoke their access. If a session would not cleanup graphics devices, the kernel would prevent any new session from accessing the graphics device. For input devices it is even worse: If a session doesn’t close the devices during deactivation, it would continue reading input events while the new session is active. So while typing in your password, the background session might send these key-strokes to your IRC client (which is exactly what XMir did). What we need is a kernel feature to forcibly revoke access to a graphics or input device. Unfortunately, it is not as easy at it sounds. We need to find some-one who is privileged and trusted enough to do this. You don’t want your background session to revoke your foreground session’s graphics access, do you? This is were systemd-logind enters the stage.

systemd-logind is already managing sessions on a system. It keeps track on which session is active and sets ACLs in /dev to give the foreground session access to device nodes. To implement device-handover, we extend the existing logind-API by a new function: RequestDevice(deviceNode). A graphics-server can pass a file-system path for a device-node in /dev to systemd-logind, which checks permissions, opens the node and returns a file-descriptor to the caller. But systemd-logind retains a copy of the file-descriptor. This allows logind to disable it as long as the session is inactive. During a session-switch, logind can now disable all devices of the old session, re-enable the devices of the new session and notify both of the session-switch. We now have a clean handover from one session to the other. With this technology in place, we can start looking at real scenarios.

1) Session-management with VTs

Session management using VTs for foreground control and logind for device management

Session management with VTs and logind

Based on the graphs for VT-switching, I drew a new one considering logind. VTs are still used to switch between sessions, but sessions no longer open hardware devices directly. Instead, they ask logind as described above. The big advantage is that VT-switches are no longer fragile. If a VT is active, it can be sure that it has exclusive hardware-access. And if a session is dead-locked, we can force a VT-switch and revoke their device-access. This allows to recover from situations where your XServer hangs without SSH’ing from a remote machine or using SysRq.

2) Session-management without VTs

Session management based solely on logind

Pure logind session management

While sane VT-switching is a nice feature, the biggest win is that we can implement proper multi-session support for seats without VTs. While previously only a single session could run on such seats, with logind device-management, we can now support session-switching on any seat.

Instead of using VTs to notify sessions when they are activated or deactivated, we use the logind-dbus-API. A graphics-server can now request input and graphics devices via the logind RequestDevice API and use it while active. Once a session-switch occurs, logind will disable the device file-descriptors and switch sessions. A dbus signal is sent asynchronously to the old and new session. The old session can stop rendering while inactive to save power.

3) Asynchonous events and backwards-compatibility

One thing changes almost unnoticed when using RequestDevice. An active graphics-server might be almost about to display an image on screen while a session-switch occurs. logind revokes access to graphics devices and sends an asynchronous event that the session is now inactive. However, the graphics-server might not have received this event, yet. Instead, it tries to invoke a system-call to update the screen. But this will fail with EACCES or EPERM as it doesn’t have access to it, anymore. Currently, for most graphics servers this is a fatal error. Instead of handling EACCES and interpreting it as “this device is now paused”, they don’t care for the error code and abort. We could fix all the graphics-servers, but to simplify the transition, we introduced negotiated session-switches.

Whenever logind is asked to perform a session-switch, it first sends PauseDevice signals for every open device to the foreground graphics-server. This must respond with a PauseDeviceComplete call to logind for each device. Once all devices are paused, the session-switch is performed. If the foreground session does not respond in a timely manner, logind will forcibly revoke device access and then perform the session-switch, anyway.

Note that negotiated session-switches are only meant for compatibility. Any graphics-server is highly encouraged to handle EACCES just fine!

All my local tests ran fine so far, but all this is still under development. systemd patches can be found at github (frequently rebased!). Most tests I do rely on an experimental novt library, also available at github (I will push it during next week; this is only for testing!). Feedback is welcome! The RFC can be found on systemd-devel. Now I need a day off..

Happy Switching!

How VT-switching works

Having multiple sessions on your system in parallel is a quite handy feature. It allows things like Fast User Switching or running two different DEs at the same time. Especially graphics developers like it, because they can test-run their experimental XServer/weston on the same machine they develop on.

To understand how it works, we need the concept of a session. See my previous introduction into session-management if you’re not familiar with it. I expect the reader to be familiar with basic session-management concepts (i.e., seats, systemd-logind, login-sessions, user-sessions, processes and daemons in a session).

1) Traditional text-mode VT-switching

Virtual terminals were introduced with linux-0.12. It’s the origin of multi-session support on linux. Before this, linux only supported a single TTY session (which was even available in the first tarball of linux-0.01). With virtual terminals, we have /dev/tty<num> devices, where <num> is between 1 and 63. They are always bound to seat0 and every session on seat0 is bound to a single VT. That means, there are at most 63 live sessions on seat0. Only one of them is active at a time (which can be read from /sys/class/tty/tty0/active).

Session management and device access with VTs in text-mode

Session management in text-mode

The kernel listens for keyboard events and switches between VTs on ctrl+alt+Fx shortcuts. Alternatively, you can issue a VT_ACTIVATE ioctl to politely ask the kernel to switch sessions.

In the old days, all sessions ran in text-mode. In text-mode, a process can get keyboard input by reading from a VT and can write onto the screen by writing to a VT. Some rather ugly control-sequences are supported to allow colors or other advanced features. The kernel interprets these and instructs the graphics hardware to print the given text.

Important to note is that in text-mode sessions don’t have direct hardware access. The kernel merges all keyboard events into one stream and a session cannot tell which device it came from. In fact, it cannot even tell how many devices there are. Same for graphics devices. As a VT has only a single output stream, there is only a single screen to write to. Multi-head support is not available. Neither are any advanced graphics-operations. But this allows the kernel to serialize access to hardware devices. Only the active VT gets input events and only the buffer of the active VT is displayed on the screen. No resource-conflicts can occur.

While there are ways to detect when a session is activated/deactivated, in text-mode a session normally doesn’t care. It just stops receiving keyboard input. If the session writes to the VT while deactivated, it will affect the internal buffer of the VT, but not the screen. Only the buffer of the active VT is shown on screen.

2) Session-switching in graphics-mode

System session management and device access with VTs in graphics mode

Session management in gfx-mode

Very soon it became clear that text-mode is not enough. We wanted more! That’s when the VT graphics mode was introduced. Graphics mode doesn’t change the setup, we still have 63 VTs and each session is bound to a VT. But a VT can now be switched from text-mode KD_TEXT into graphics-mode KD_GRAPHICS (via KDSETMODE ioctl). This doesn’t do anything spectacular. Really! The only effect is it disables the kernel-internal graphics routines. As long as a VT is in graphics-mode, the kernel will not instruct the graphics hardware to display the VT on screen. Once it is reset to text-mode and the VT is active, the kernel will display it again.

So the graphics-mode itself is useless. But at the same time, the kernel started providing separate interfaces to input and graphics devices. So while a VT is in graphics-mode, a session can access the graphics devices directly and render whatever they want. It can also ignore input from the VT and instead read input-events directly from the input-event interfaces. This is how the XServer works today.

But it is pretty obvious that this becomes problematic during session switches. If the kernel switches away from a graphics VT, the session needs to release the graphics hardware before the new session can be activated. Otherwise, the new session is active, but you still get the images from the old one. Or worse, it might flicker between the images of both sessions. Without kernel-mode-setting it might even hang your graphics hardware.

(A similar interface to KDSETMODE exists for keyboards with KDSKBMODE.)

Unfortunately, the kernel has no interface to forcibly revoke graphics or input access. So VTs were extended by the VT_SETMODE ioctl. A process can issue VT_SETMODE on a VT and pass two signal-numbers (usually SIGUSR1 and SIGUSR2). If the kernel wants to perform a VT-switch, it sends one such signal to the active VT. This VT can cleanup resources, stop using graphics/input devices and acknowledge the VT-switch via the VT_RELDISP ioctl. If the process dead-locked and doesn’t acknowledge the request, the VT-switch will not happen! This is why a crashed XServer can hang your system. But if it correctly issues the VT_RELDISP ioctl, the kernel will perform the VT-switch. Once the kernel switches back to the given VT it sends the second signal as notification that it is now active again.

However, on every VT (precisely, on every session) only a single process can call VT_SETMODE. This already shows that the concept is flawed. For example, if the XServer takes the VT in posession for graphics and input devices, another audio-server in the session couldn’t do the same for audio-hardware. This applies to all other devices. An alternative involving logind is discussed in a followup article.

3) Seats without VTs

We discussed that VTs are always bound to seat0. So if you run a session on a seat other than seat0 or if you disabled VTs entirely, then the situation becomes pretty simple: no multi-session support is available. This is enforced by systemd-logind so the active session can run without interruptions.

Also important to note is that there is no text-mode. A session must run in graphics mode as the kernel facility to interpret text-commands is bound to VTs.

Single-session setup on seats without VTs

Session management without VTs

4) logind integration

As we discussed in the previous article, today systemd-logind tracks and manages sessions. But how does this integrate with VTs? VTs pre-date systemd by years, so to preserve backwards-compatibility, we need to keep the infrastructure as it is. That’s why logind watches /sys/class/tty/tty0/active for changes. Once a VT switch happens, logind notices it and marks the old session as inactive and the new as active. It also adjusts ACLs in /dev to keep access-restrictions in sync with the active session. However, this cannot be done properly without a race-condition. Therefore, logind provides a new dbus-API to replace the ageing VT API. New session-daemons are advised to use it in favor of VTs.

KMSCON Introduction

KMSCON is a KMS/DRM-based system console with an integrated terminal emulator. It was designed to replace the linux-kernel-console and virtual terminals (VTs). When run in default-mode, KMSCON allocates a virtual terminal and provides a terminal-emulator on it. It can thus be used as drop-in replacement for the linux-console and agetty. Compared to the linux-console, KMSCON provides a rich set of enhanced features including full-internationalized keyboard handling, full UTF8 input/font support, hardware-accelerated rendering, multi-seat support and more.

If run in listen-mode, KMSCON can also replace virtual terminals. This allows to run multiple graphics-servers (like kmscon, X-Server, Wayland Compositors) simultaneously on all seats not only on the default seat seat0.

Parts of this article expect the reader to be familiar with the term multi-seat. KMSCON runs on single-seat and multi-seat setups, but to understand the terms used here, you should read up on Wikipedia. On linux, the default seat is called seat0. This is also the primary seat for single-seat setups or if KMSCON is compiled without multi-seat support.

Default Mode

In default-mode KMSCON opens a virtual terminal (VT) to provide a terminal-emulator. It first checks whether the controlling terminal is a VT, if it’s not it tries to find an unused VT via VT_OPENQRY. You can also specify a VT on the command line via vt=/dev/tty5, or –vt=tty5, or –vt=5. This syntax allows backwards compatibility to most getty implementations.

After startup the VT is automatically activated and switched to (this can be prevented with –no-switchvt). You can use the standard linux keyboard shortcuts ctrl+alt+F1 to ctrl+alt+F12 to switch between your VTs. KMSCON integrates seamlessly into existing setups and can be used in parallel with X-Servers, Wayland-Compositors or the linux-console. Only one VT is occupied by KMSCON, so you can still use the linux-console on all other VTs. But you can also run a KMSCON process on each VT replacing the linux-console everywhere.

By default, KMSCON spawns /bin/login on the new terminal session. You can change this with the –login option. However, for security reasons it is recommended to keep the default. If a session exits, it is automatically restarted. To terminate KMSCON send SIGTERM to the main process or use a daemon manager like systemd.

KMSCON tries to be backwards-compatible in terms of user-interaction to the linux-console. However, the internal terminal-emulator is developed to be compatible with xterm. This is because the linux-console‘s terminal-emulation layer is very limited and incompatible to many other terminal-emulators. On the other hand, xterm is the de-facto standard in terminal-emulation under linux and provides many enhanced control-sequences. Therefore, KMSCON tries to behave exactly like xterm does. This isn’t always as easy as it sounds so please file bug-reports if you encounter any incompatibilities.

With this knowledge we can now replace agetty. So instead of:

  /sbin/agetty --noclear tty5 38400 linux

we now run:

  /usr/bin/kmscon --vt=tty5 --no-switchvt

to replace the linux-console with KMSCON on VT-5.  KMSCON also ships optional systemd service files in ./kmscon/docs/*.service

Keyboard Input

KMSCON uses libxkbcommon for keyboard control. This library implements huge parts of the X11 Keyboard Specification without any dependencies to X11 source-code. It was developed to allow other applications than X-Server to provide internationalized keyboard handling. It recently saw its first release after several years of development but is currently not available in all major linux distributions. However, many distributions already provide experimental packages for it.

With XKB support on board, KMSCON can provide the same keyboard handling as the X-Server does. That is, keyboards are configured via the RMLVO options (Rules Model Layout Variant Options). The most commonly used option is probably –xkb-layout=<language> to change the keyboard layout. All other options are mostly unused.

Any keyboard setups that can be used with X11 can also be used with xkbcommon. Simply create your X11 keyboard layouts or use one of the many existing ones and configure KMSCON to use it.

Video Devices

KMSCON got its name from the linux kernel Direct Rendering Manager (DRM) subsystem (not to be confused with DRM: Digital Rights Management). DRM drivers provide a common set of APIs to perform Mode-Setting, called Kernel Mode-Setting (KMS). This is the API that is used by X-Server to program video output. KMSCON uses the same API and thus provides mostly the same compatibility to GPU drivers as X-Server does.

But KMSCON also supports linux fbdev devices for backwards-compatibility. In fact, the modular interface allows to provide drivers for all kinds of video hardware. However, it is recommended to write DRM drivers for your video-hardware instead of writing user-space drivers for KMSCON or X-Server. Because this allows all user-space programs to use the DRM API to access any type of video output device without extra user-space support for each driver.

Right from the beginning KMSCON provided multiple-GPU support. This means, KMSCON automatically picks up all connected GPUs and provides a system console on them. You can even hotplug new DisplayLink GPUs and KMSCON detects them during runtime and instantly provides a console on them. Unfortunately, GPU detection isn’t as straightforward as one would think. On some systems there are GPUs that shouldn’t be touched by KMSCON. That’s why KMSCON provides the –gpu switch to change the GPU selection algorithm. A GPU blacklist configuration file or option is also on its way.

On fbdev and dumb-DRM devices, KMSCON uses 2D rendering without any hardware acceleration. But if KMSCON is compiled with full DRM video backends, it can use mesa3D’s OpenGLESv2 implementation to provide hardware-accelerated rendering. Use –hwaccel to enable this during runtime. It can speed up rendering by multiple orders of magnitude. However, on older systems it might even slow down rendering. It is disabled by default and needs to be explicitly enabled with –hwaccel.

Seat Selection

By default, KMSCON runs on the seat that it was started on. At most times, this is the default seat seat0. However, you can run KMSCON on other seats with the –seats option. It takes a list of seat names that KMSCON will run on. Each seat is independent and you can either run one KMSCON process for each seat or you can even run a single KMSCON process on multiple seats. The latter will allow sharing of system-resources like font glyph-caches. However, if 3D-hardware-acceleration is enabled, unexpected latencies may occur in the DRM drivers and you shouldn’t share a single KMSCON process across multiple seats for decent user-experience.

On seat0 KMSCON can make use of VTs. However, on seats other than seat0 (or even on seat0 if VTs are disabled) linux does not provide VTs. Therefore, KMSCON has to run without them. This means, KMSCON will automatically activate itself and run unconditionally on this seat. Unfortunately this has the side-effect that no other graphics-server (like X-Server) can run on this seat. Without the synchronization API of VTs there is no way to dispatch the graphics/input devices between multiple graphics servers. Therefore, most users will prefer running an X-Server on those seats. However, if no X11 is needed, KMSCON runs perfectly fine on those seats, too.


In default-mode KMSCON runs only on seats that are available during startup and terminates after all seats it runs on are either gone or their controlling VT hung up. This is perfect for VTs on seat0 but it requires another process to start KMSCON for each new seat showing up.

In listen-mode KMSCON runs as daemon waiting for new seats to show up. It automatically spawns a new terminal session on each seat matching the –seats option. KMSCON will not exit until SIGTERM is caught. Note that KMSCON ignores seats with VTs (normally only seat0) in this mode as you should run KMSCON in default-mode on those seats.

VT Replacement

The linux-kernel-console and VTs are tightly bound to each other. There is only one kernel configuration option to control them both: CONFIG_VT. KMSCON can already replace the linux-kernel-console but to disable CONFIG_VT, we also need a proper replacement for virtual terminals. As previously noted, KMSCON can run smoothly without VTs, however, this means that you can only run a single graphics-server on a seat and most users will probably choose X-Server here (and I would do so, too).

Fortunately, KMSCON already provides a solution to this. The –cdev-session option provides fake VTs for all seats that do not have real VTs. This allows to run X-Server, KMSCON, a Wayland Compositor and any other graphics-server simultaneously without changing a single line of code. However, this is beyond the scope of this introduction. Stay tuned for a follow-up post that introduces KMSCON session support and cdev-sessions in more detail.

Current State

KMSCON works! No special setup, no special system requirements and no weird dependencies are needed. You can check out KMSCON on any major linux distribution. Right now. What you need is:

  • libxkbcommon: This library has no external dependencies and can easily be installed on any distribution. Many distributions even provide experimental packages for it. See xkbcommon.org.
  • libudev: Udev is used for hotplugging and device detection. Every major distribution provides it.
  • kmscon: You can get KMSCON from github.

Everything else is optional! For proper font-rendering you also need pango or freetype2. For DRM video output you need libdrm. For 3D-hardware-acceleration mesa3D is needed. For multi-seat support systemd-logind is used.

However, please note that KMSCON is still experimental. I do my best to keep backwards-compatibility and fix all bugs that are reported. But I am unable to test KMSCON on all imaginable setups. So please report all bugs to github.com/dvdhrm/kmscon and I promise to have a look at it.

Furthermore, the TODO list grows steadily and I still have lots of ideas how to improve KMSCON. New ideas are very welcome, but please bear with me if it takes more time than expected to implement them. There is probably lots of stuff with higher priority on my list. But patches are highly appreciated even if they implement low-priority features.

There is a man-page kmscon(1) that explains many concepts and command-line arguments right-away. But do not hesitate to contact me personally if you have any questions about KMSCON. And for all developers I am doing a presentation about deprecating CONFIG_VT and KMSCON during FOSDEM-13 in Brussels next year. Hope to see you there!