On Wifi-Display, Democratic Republics and Miracles

For a long time connecting your TV or other external monitors to your laptop has always been a hassle with cables and drivers. Your TV had an HDMI-port, but your laptop only DVI, your projector requires VGA and your MacBook can only do DP. Once you got a matching adapter, you noticed that the cable is far to short for you to sit comfortably on your couch. Fortunately, we’re about to put an end to this. At least, that was my impression when I first heard of Miracast.

Miracast is a certification program of the Wifi-Alliance based on their Wifi-Display specification. It defines a protocol to connect external monitors via wifi to your device. A rough description would be “HDMI over Wifi” and in fact the setup is modeled closely to the HDMI standard. Unlike existing video/audio-streaming protocols, Miracast was explicitly designed for this use-case and serves it very well. When I first heard of it about one year ago, I was quite amazed that it took us until late 2012 to come up with such a simple idea. And it didn’t take me long until I started looking into it.

For 4 months now I have been hacking on wifi, gstreamer and the kernel to get a working prototype. Despite many setbacks, I didn’t loose my fascination for this technology. I did complain a lot about the hundreds of kernel dead-locks, panics and crashes during development and the never-ending list of broken wifi-devices. However, to be fair I got the same amount of crashes and bugs on Android and competing proprietary Miracast products. I found a lot of other projects with the same goal, but the majority was closed-source, driver-specific, only a proof-of-concept or limited to sink-side. Nevertheless, I continued pushing code to the OpenWFD repository and registered as speaker for FOSDEM-2014 to present my results. Unfortunately, it took me until 1 week before FOSDEM to get the wifi-P2P link working realiably with an upstream driver. So during these 7 days I hacked up the protocol and streaming part and luckily got a working demo just in time. I never dared pushing that code to the public repository, though (among others it contains my birthday, which is surprisingly page-aligned, as magic mmap offset) and I doubt it will work on any setup but mine..

The first day after FOSDEM I trashed OpenWFD and decided to start all over again writing a properly integrated solution based on my past experience. Remembering the advice of a friend (“Projects with ‘Open’ in their name are just like countries named ‘Democratic Republic of …’”) I also looked for a new name. Given the fact that it requires a miracle to get Wifi-P2P working, it didn’t take me long to come up with something I liked. I hereby proudly announce: MiracleCast

MiracleCast

The core of MiracleCast is a daemon called miracled. It runs as system-daemon and manages local links, performs peer-discovery on request and handles protocol encoding and parsing. The daemon is independent of desktop environments or data transports and serves as local authority for all display-streaming protocols. While Miracast will stay the main target, other competing technologies like Chromecast, Airtame and AirPlay can be integrated quite easily.

The miraclectl command-line tool can be used to control the daemon. It can create new connections, modify parameters or destroy them again. It also supports an interactive mode where it displays events from miracled when they arrive, displays incoming connection attempts and allows direct user-input to accept or reject any requests.

miracled and miraclectl communicate via DBus so you can replace the command-line interface with graphical helpers. The main objects on this API are Links and Peers. Links describe local interfaces that are used to communicate with remote devices. This includes wifi-devices, but also virtual links like your local IP-Network. Peers are remote devices that were discovered on a local link. miracled hides the transport-type of each peer so you can use streaming protocols on-top of any available link-type (given the remote side supports the same). Therefore, we’re not limited to Wifi-P2P, but can use Ethernet, Bluetooth, AP-based Wifi and basically any other transport with an IP layer on top. This is especially handy for testing and development.

Local processes can now register Sources or Sinks with miracled. These describe capabilities that are advertised on the available links. Remote devices can then initiate a connection to your sink or, if you’re a source, you can connect to a remote sink. miracled implements the transmission protocol and hides it behind a DBus interface. It routes traffic from remote devices to the correct local process and vice versa. Regardless of whether the connection uses Miracast, Chromecast or plain RTSP, the same API is used.

Current Status

The main target still is Miracast! While all the APIs are independent of the protocol and transport layer, I want to get Miracast working as it is a nice way to test interoperability with Android. And for Miracast, we first need Wifi-P2P working. Therefore, that’s what I started with. The current miracled daemon implements a fully working Wifi-P2P user-space based on wpa_supplicant. Everyone interested is welcome to give it a try!

Everything on top, including the actual video-streaming is highly experimental and still hacked on. But if you hacked on Miracast, you know that the link-layer is the hard part. So after 1 week of vacation (that is, 1 week hacking on systemd!) I will return to MiracleCast and tackle local sinks.

As I have a lot more information than I could possible summarize here, I will try to keep some dummy-documentation on the wiki until I the first project release. Please feel free to contact me if you have any questions. Check out the git-repository if you want to see how it works! Note that MiracleCast is focused on proper desktop integration instead of fast prototyping, so please bear with me if API design takes some time. I’d really appreciate help on making Wifi-P2P work with as many devices as possible before we start spending all our efforts on the upper streaming layers.

During the next weeks, I will post some articles that explain step by step how to get Wifi-P2P working, how you can get some Miracast protoypes working and what competing technologies and implementations are available out there. I hope this way I can spread my fascination for Miracast and provide it to as many people as possible!

Splitting DRM and KMS device nodes

While most devices of the 3 major x86 desktop GPU-providers have GPU and display-controllers merged on a single card, recent development (especially on ARM) shows that rendering (via GPU) and mode-setting (via display-controller) are not necessarily bound to the same device. To better support such devices, several changes are being worked on for DRM.

In it’s current form, the DRM subsystem provides one general-purpose device-node for each registered DRM device: /dev/dri/card<num>. An additional control-node is also created, but it remains unused as of this writing. While in general a kernel driver is allowed to register multiple DRM devices for a single physical device, no driver made use of this, yet. That means, whatever hardware you use, both mode-setting and rendering is done via the same device node. This entails some rather serious consequences:

  1. Access-management to mode-setting and rendering is done via the same file-system node
  2. Mode-setting resources of a single card cannot be split among multiple graphics-servers
  3. Sharing display-controllers between cards is rather complicated

In the following sections, I want to look closer at each of these points and describe what has been done and what is still planned to overcome these restrictions. This is a highly technical description of the changes and serves as outline for the Linux-Plumbers session on this topic. I expect the reader to be familiar with DRM internals.

1) Render-nodes

While render-nodes have been discussed since 2009 on dri-devel, several mmap-related security-issues have prevented it from being merged. Those have all been fixed and 3-days ago, the basic render-node infrastructure has been merged. While it’s still marked as experimental and hidden behind the drm.rnodes module parameter, I’m confident we will enable it by default in one of the next kernel releases.

What are render-nodes?

From a user-space perspective, render-nodes are “like a big FPU” (krh) that can be used by applications to speed up computations and rendering. They are accessible via /dev/dri/renderD<num> and provide the basic DRM rendering interface. Compared to the old card<num> nodes, they lack some features:

  • No mode-setting (KMS) ioctls allowed
  • No insecure gem-flink allowed (use dma-buf instead!)
  • No DRM-auth required/supported
  • No legacy pre-KMS DRM-API supported

So whenever an application wants hardware-accelerated rendering, GPGPU access or offscreen-rendering, it no longer needs to ask a graphics-server (via DRI or wl_drm) but can instead open any available render node and start using it. Access-control to render-nodes is done via standard file-system modes. It’s no longer shared with mode-setting resources and thus can be provided for less-privileged applications.

It is important to note that render-nodes do not provide any new APIs. Instead, they just split a subset of the already available DRM-API off to a new device-node. The legacy node is not changed but kept for backwards-compatibility (and, obviously, for mode-setting).

It’s also important to know that render-nodes are not bound to a specific card. While internally it’s created by the same driver as the legacy node, user-space should never assume any connection between a render-node and a legacy/mode-setting node. Instead, if user-space requires hardware-acceleration, it should open any node and use it. For communication back to the graphics-server, dma-buf shall be used. Really! Questions like “how do I find the render-node for a given card?” don’t make any sense. Yes, driver-specific user-space can figure out whether and which render-node was created by which driver, but driver-unspecific user-space should never do that! Depending on your use-cases, either open any render-node you want (maybe allow an environment-variable to select it) or let the graphics-server do that for you and pass the FD via your graphics-API (X11, wayland, …).

So with render-nodes, kernel drivers can now provide an interface only for off-screen rendering and GPGPU work. Devices without any display-controller can avoid any mode-setting nodes and just provide a render-node. User-space, on the other hand, can finally use GPUs without requiring any privileged graphics-server running. They’re independent of the kernel-internal DRM-Master concept!

2) Mode-setting nodes

While splitting off render-nodes from the legacy node simplifies the situation for most applications, we didn’t simplify it for mode-setting applications. Currently, if a graphics-server wants to program a display-controller, it needs to be DRM-Master for the given card. It can acquire it via drmSetMaster() and drop it via drmDropMaster(). But only one application can be DRM-Master at a time. Moreover, only applications with CAP_SYS_ADMIN privileges can acquire DRM-Master. This prevents some quite fancy features:

  • Running an XServer without root-privileges
  • Using two different XServers to control two independent monitors/connectors of the same card

The initial idea (and Ilija Hadzic’s follow-up) to support this were mode-setting nodes. A privileged ioctl on the control-node would allow applications to split mode-setting resources across different device-nodes. You could have /dev/dri/modesetD1 and /dev/dri/modesetD2 to split your KMS CRTC and Connector resources. An XServer could use one of these nodes to program the now reduced set of resources. We would have one DRM-Master per node and we’d be fine. We could remove the CAP_SYS_ADMIN restriction and instead rely on file-system access-modes to control access to KMS resources.

Another discussed idea to avoid creating a bunch of file-system nodes, is to allocate these resources on-the-fly. All mode-setting-resources would now be bound to a DRM-Master object. An application can only access the resources available on the DRM-Master that it is assigned to. Initially, all resources are bound to the default DRM-Master as usual, which everyone gets assigned to when opening a legacy node. A new ioctl DRM_CLONE_MASTER is used to create a new DRM-Master with the same resources as the previous DRM-Master of an application. Via a DRM_DROP_MASTER_RESOURCE an application can drop KMS resources from their DRM-Master object. Due to their design, neither requires a CAP_SYS_ADMIN restriction as they only clone or drop privileges, they never acquire new privs! So they can be used by any application with access to the control node to create two new DRM-Master resources and pass them to two independent XServers. These use the passed FD to access the card, instead of opening the legacy or mode-setting nodes.

From the kernel side, the only thing that changes is that we can have multiple active DRM-Master objects. In fact, per DRM-Master one open-file might be allowed KMS access. However, this doesn’t require any driver-modifications (which were mostly “master-agnostic”, anyway) and only a few core DRM changes (except for vmwgfx-ttm-lock..).

3) DRM infrastructure

The previous two chapters focused on user-space APIs, but we also want the kernel-internal infrastructure to account for split hardware. However, fact is we already have anything we need. If some hardware exists without display-controller, you simply omit the DRIVER_MODESET flag and only set DRIVER_RENDER. DRM core will only create a render-node for this device then. If your hardware only provides a display-controller, but no real rendering hardware, you simply set DRIVER_MODESET but omit DRIVER_RENDER (which is what SimpleDRM is doing).

Yes, you currently get a bunch of unused DRM code compiled-in if you don’t use some features. However, this is not because DRM requires it, but only because no-one sent any patches for it, yet! DRM-core is driven by DRM-driver developers!

There is a reason why mid-layers are frowned upon in DRM land. There is no group of core DRM developers, but rather a bunch of driver-authors who write fancy driver-extensions. And once multiple drivers use them, they factor it out and move it to DRM core. So don’t complain about missing DRM features, but rather extend your drivers. If it’s a nice feature, you can count on it being incorporated into DRM-core at some point. It might be you doing most of the work, though!

Sane Session-Switching

In a previous article I talked about the history of VT switching. Created as a simple way to switch between text-mode sessions it has grown into a fragile API to protect one testosterone monster (also called XServer) from another. XServers used to poke in PCI bars, modified MMIO registers and messed around with DMA controllers. If they hauled out the big guns, it was almost absurd to believe a simple signal-flinging VT could ever successfully negotiate. Fortunately, today’s XServer is a repentant sinner. With common desktop hardware, all direct I/O is done in the kernel (thanks KMS!) and chances of screwing up your GPUs are rather minimal. This allows us to finally implement proper device-handover during session-switches.

If we look at sessions at a whole, the XServer isn’t special at all. A lot of session-daemons may run today that provide some service to the session as a whole. This includes pulseaudio, dbus, systemd –user, colord, polkit, ssh-keychain, and a lot more. All these daemons don’t need any special synchronization during session-switch. So why does the XServer require it?

Any graphics-server like the XServer is responsible of providing access to input and graphics devices to a session. When a session is activated, they need to re-initialize the devices. Before a session is deactivated, they need to cleanup the devices so the to-be-activated session can access them. The reason they need to do this is missing infrastructure to revoke their access. If a session would not cleanup graphics devices, the kernel would prevent any new session from accessing the graphics device. For input devices it is even worse: If a session doesn’t close the devices during deactivation, it would continue reading input events while the new session is active. So while typing in your password, the background session might send these key-strokes to your IRC client (which is exactly what XMir did). What we need is a kernel feature to forcibly revoke access to a graphics or input device. Unfortunately, it is not as easy at it sounds. We need to find some-one who is privileged and trusted enough to do this. You don’t want your background session to revoke your foreground session’s graphics access, do you? This is were systemd-logind enters the stage.

systemd-logind is already managing sessions on a system. It keeps track on which session is active and sets ACLs in /dev to give the foreground session access to device nodes. To implement device-handover, we extend the existing logind-API by a new function: RequestDevice(deviceNode). A graphics-server can pass a file-system path for a device-node in /dev to systemd-logind, which checks permissions, opens the node and returns a file-descriptor to the caller. But systemd-logind retains a copy of the file-descriptor. This allows logind to disable it as long as the session is inactive. During a session-switch, logind can now disable all devices of the old session, re-enable the devices of the new session and notify both of the session-switch. We now have a clean handover from one session to the other. With this technology in place, we can start looking at real scenarios.

1) Session-management with VTs

Session management using VTs for foreground control and logind for device management

Session management with VTs and logind

Based on the graphs for VT-switching, I drew a new one considering logind. VTs are still used to switch between sessions, but sessions no longer open hardware devices directly. Instead, they ask logind as described above. The big advantage is that VT-switches are no longer fragile. If a VT is active, it can be sure that it has exclusive hardware-access. And if a session is dead-locked, we can force a VT-switch and revoke their device-access. This allows to recover from situations where your XServer hangs without SSH’ing from a remote machine or using SysRq.

2) Session-management without VTs

Session management based solely on logind

Pure logind session management

While sane VT-switching is a nice feature, the biggest win is that we can implement proper multi-session support for seats without VTs. While previously only a single session could run on such seats, with logind device-management, we can now support session-switching on any seat.

Instead of using VTs to notify sessions when they are activated or deactivated, we use the logind-dbus-API. A graphics-server can now request input and graphics devices via the logind RequestDevice API and use it while active. Once a session-switch occurs, logind will disable the device file-descriptors and switch sessions. A dbus signal is sent asynchronously to the old and new session. The old session can stop rendering while inactive to save power.

3) Asynchonous events and backwards-compatibility

One thing changes almost unnoticed when using RequestDevice. An active graphics-server might be almost about to display an image on screen while a session-switch occurs. logind revokes access to graphics devices and sends an asynchronous event that the session is now inactive. However, the graphics-server might not have received this event, yet. Instead, it tries to invoke a system-call to update the screen. But this will fail with EACCES or EPERM as it doesn’t have access to it, anymore. Currently, for most graphics servers this is a fatal error. Instead of handling EACCES and interpreting it as “this device is now paused”, they don’t care for the error code and abort. We could fix all the graphics-servers, but to simplify the transition, we introduced negotiated session-switches.

Whenever logind is asked to perform a session-switch, it first sends PauseDevice signals for every open device to the foreground graphics-server. This must respond with a PauseDeviceComplete call to logind for each device. Once all devices are paused, the session-switch is performed. If the foreground session does not respond in a timely manner, logind will forcibly revoke device access and then perform the session-switch, anyway.

Note that negotiated session-switches are only meant for compatibility. Any graphics-server is highly encouraged to handle EACCES just fine!

All my local tests ran fine so far, but all this is still under development. systemd patches can be found at github (frequently rebased!). Most tests I do rely on an experimental novt library, also available at github (I will push it during next week; this is only for testing!). Feedback is welcome! The RFC can be found on systemd-devel. Now I need a day off..

Happy Switching!

How VT-switching works

Having multiple sessions on your system in parallel is a quite handy feature. It allows things like Fast User Switching or running two different DEs at the same time. Especially graphics developers like it, because they can test-run their experimental XServer/weston on the same machine they develop on.

To understand how it works, we need the concept of a session. See my previous introduction into session-management if you’re not familiar with it. I expect the reader to be familiar with basic session-management concepts (i.e., seats, systemd-logind, login-sessions, user-sessions, processes and daemons in a session).

1) Traditional text-mode VT-switching

Virtual terminals were introduced with linux-0.12. It’s the origin of multi-session support on linux. Before this, linux only supported a single TTY session (which was even available in the first tarball of linux-0.01). With virtual terminals, we have /dev/tty<num> devices, where <num> is between 1 and 63. They are always bound to seat0 and every session on seat0 is bound to a single VT. That means, there are at most 63 live sessions on seat0. Only one of them is active at a time (which can be read from /sys/class/tty/tty0/active).

Session management and device access with VTs in text-mode

Session management in text-mode

The kernel listens for keyboard events and switches between VTs on ctrl+alt+Fx shortcuts. Alternatively, you can issue a VT_ACTIVATE ioctl to politely ask the kernel to switch sessions.

In the old days, all sessions ran in text-mode. In text-mode, a process can get keyboard input by reading from a VT and can write onto the screen by writing to a VT. Some rather ugly control-sequences are supported to allow colors or other advanced features. The kernel interprets these and instructs the graphics hardware to print the given text.

Important to note is that in text-mode sessions don’t have direct hardware access. The kernel merges all keyboard events into one stream and a session cannot tell which device it came from. In fact, it cannot even tell how many devices there are. Same for graphics devices. As a VT has only a single output stream, there is only a single screen to write to. Multi-head support is not available. Neither are any advanced graphics-operations. But this allows the kernel to serialize access to hardware devices. Only the active VT gets input events and only the buffer of the active VT is displayed on the screen. No resource-conflicts can occur.

While there are ways to detect when a session is activated/deactivated, in text-mode a session normally doesn’t care. It just stops receiving keyboard input. If the session writes to the VT while deactivated, it will affect the internal buffer of the VT, but not the screen. Only the buffer of the active VT is shown on screen.

2) Session-switching in graphics-mode

System session management and device access with VTs in graphics mode

Session management in gfx-mode

Very soon it became clear that text-mode is not enough. We wanted more! That’s when the VT graphics mode was introduced. Graphics mode doesn’t change the setup, we still have 63 VTs and each session is bound to a VT. But a VT can now be switched from text-mode KD_TEXT into graphics-mode KD_GRAPHICS (via KDSETMODE ioctl). This doesn’t do anything spectacular. Really! The only effect is it disables the kernel-internal graphics routines. As long as a VT is in graphics-mode, the kernel will not instruct the graphics hardware to display the VT on screen. Once it is reset to text-mode and the VT is active, the kernel will display it again.

So the graphics-mode itself is useless. But at the same time, the kernel started providing separate interfaces to input and graphics devices. So while a VT is in graphics-mode, a session can access the graphics devices directly and render whatever they want. It can also ignore input from the VT and instead read input-events directly from the input-event interfaces. This is how the XServer works today.

But it is pretty obvious that this becomes problematic during session switches. If the kernel switches away from a graphics VT, the session needs to release the graphics hardware before the new session can be activated. Otherwise, the new session is active, but you still get the images from the old one. Or worse, it might flicker between the images of both sessions. Without kernel-mode-setting it might even hang your graphics hardware.

(A similar interface to KDSETMODE exists for keyboards with KDSKBMODE.)

Unfortunately, the kernel has no interface to forcibly revoke graphics or input access. So VTs were extended by the VT_SETMODE ioctl. A process can issue VT_SETMODE on a VT and pass two signal-numbers (usually SIGUSR1 and SIGUSR2). If the kernel wants to perform a VT-switch, it sends one such signal to the active VT. This VT can cleanup resources, stop using graphics/input devices and acknowledge the VT-switch via the VT_RELDISP ioctl. If the process dead-locked and doesn’t acknowledge the request, the VT-switch will not happen! This is why a crashed XServer can hang your system. But if it correctly issues the VT_RELDISP ioctl, the kernel will perform the VT-switch. Once the kernel switches back to the given VT it sends the second signal as notification that it is now active again.

However, on every VT (precisely, on every session) only a single process can call VT_SETMODE. This already shows that the concept is flawed. For example, if the XServer takes the VT in posession for graphics and input devices, another audio-server in the session couldn’t do the same for audio-hardware. This applies to all other devices. An alternative involving logind is discussed in a followup article.

3) Seats without VTs

We discussed that VTs are always bound to seat0. So if you run a session on a seat other than seat0 or if you disabled VTs entirely, then the situation becomes pretty simple: no multi-session support is available. This is enforced by systemd-logind so the active session can run without interruptions.

Also important to note is that there is no text-mode. A session must run in graphics mode as the kernel facility to interpret text-commands is bound to VTs.

Single-session setup on seats without VTs

Session management without VTs

4) logind integration

As we discussed in the previous article, today systemd-logind tracks and manages sessions. But how does this integrate with VTs? VTs pre-date systemd by years, so to preserve backwards-compatibility, we need to keep the infrastructure as it is. That’s why logind watches /sys/class/tty/tty0/active for changes. Once a VT switch happens, logind notices it and marks the old session as inactive and the new as active. It also adjusts ACLs in /dev to keep access-restrictions in sync with the active session. However, this cannot be done properly without a race-condition. Therefore, logind provides a new dbus-API to replace the ageing VT API. New session-daemons are advised to use it in favor of VTs.

Thoughts on Linux System Compositors

My work towards deprecating CONFIG_VT recently got stuck due to several kernel changes that are needed to proceed. DRM Render-nodes and Modeset-nodes are on their way and so I thought I’d provide a short update on what is being planned regarding session management in user-space.

This is no technical discussion or comparison between VTs and other session managers. Instead, I try to describe how we intend to proceed and why we need a new kind of user-space session manager. For technical details, feel free to browse through other articles in this blog or join the discussions on systemd-devel, wayland-devel and dri-devel MLs.

Virtual Terminals

While legacy systems with CONFIG_VT will always be supported, the current trend is towards a system without VTs. VTs are controversially discussed, but the major flaw of VTs is that they’re only available on the default seat. So if you’re a big fan of VTs, consider this whole article being about seats other than seat0, that is, no VTs are available! If you’re not so a big fan of VTs, join my dream of a world without VTs (and without fbcon, hurray!).

Systemd

With Ubuntu (and upstart) considering to emulate systemd-logind, it is becoming clear that the systemd-logind API is getting the de-facto standard for seat and session management on desktop linux systems. I don’t care what people think of systemd; for the purpose of this article I will call the session manager logind, feel free to replace it by your favorite seat/session manager.

Non-desktop systems are beyond the scope of this document. However, as long as you have a monitor+keyboard connected to your system, your setup is targeted by this article.

#1: The Problem

Most of the time, on a common linux desktop system, we interact with our favorite Desktop Environment; lets choose Gnome for today. Gnome manages the different screens, applications and tasks that we want to run. We can switch between them, stop them or start them. This is all managed inside a single Gnome-User-Session. The need for multiple sessions (and thus, multiple VTs) is barely noticeable to us. And that fact is very important! Whatever we do to extend session management, the single user session should still be the central part of the design. We must not make it, in any way slower, less stable, or harder to implement. Instead, we design our system around it.

So lets broaden our perspective and look at all the systems running in parallel to the user session: There is the boot-splash, linux-console, emergency-console, sleeping user-sessions, lock-screens, ssh-sessions, custom parallel Wayland/X11 sessions and probably a lot of crazy user-setups I am not even aware of. It is hard to work with such a diverse and moving target, however, they always have something in common: Only one desktop session is active at a time!

But which one?

#2: logind to the rescue

Layout and drawbacks of the VT Session Manager

It is pretty obvious that we need an IPC mechanism to notify the different sessions about who is active and when to switch between them. Historically, VTs provided a decentralized way by letting each session open a different VT and making the active VT responsible of session-management. To be fair, there is support for automatic session-switching, but most sessions take control over input devices and thus need to schedule session switches themselves. Major flaw is that if the active session deadlocks, your system becomes unresponsive. While the VT system has several other drawbacks, discussion is beyond the scope of this document as we always have the situation where we don’t have VTs (think: seats other than seat0).

Advanced access control with FD passing via systemd-logind

Without VTs, we are currently left with only a single session per seat. Pretty boring! And that’s probably also what the desktop group at RedHat thought. A broadly discussed idea from several RedHat developers is to extend logind to arbitrate between the different sessions (Note: this article is about my perception, but the idea of using logind was originally not from me). logind already knows about all sessions on a system, knows about all seats and is the perfect place to implement session management. So instead of opening a VT, sessions now use dbus to tell logind that they provide a new user session. The API hasn’t been figured out, yet, but the big advantage over VTs is that we can pass file-descriptors directly to each session. logind can mute input-devices (patches pending on linux-input ML) for inactive sessions and can revoke DRM access. It guarantees that the active session has exclusive hardware access even if a session dead-locks.

Left for discussion is the design of the logind dbus API. It will include a set of calls to register, activate, deactivate and unregister sessions. Due to the fact that we have control over hardware resources, we can also implement forced session switching. Moreover, we could make logind listen to magic sysrq keys which deactivates the current session and instead activates the emergency console.

#3: Limitations

logind is no system compositor. logind will never be a system compositor. In fact, if logind becomes the central session manager, the concept of a system compositor will be dead and buried. The problem is, a session manager requires exclusive access over any sessions and hardware. Same is true for a system-compositor. Both concepts don’t play nice together. But what are the benefits of a system-compositor over logind?

While anything a system-compositor can do, could theoretically be implemented in logind, we assume that logind is supposed to stay small and efficient. So logind will never provide any DRM display handling. All sessions must implement it themselves. Same holds true for input handling. logind can only provide FD forwarding, but the heavy lifting must be done directly in each session program. In every small session program! The boot-splash must include complex DRM and XKB code, the screen-lock, too. Even the emergency console has to do all the low-level hardware handling. This introduces complexity and, very likely, bugs. But on the other hand, it guarantees efficiency. An xserver can run with direct hardware access and no extra context-switch to logind is needed on every frame.

But DRM handling could be simplified via a shared library for every session that doesn’t need custom 3D acceleration. So what else do we abandon for the sake of simplicity?

Probably the most hurtful is that global keyboard management is impossible. We cannot make logind listen for keyboard events (other than real emergency shortcuts) at the same time as a session is active. Otherwise, we might end up with logind and the session reacting to a single shortcut, which is unacceptable in any non-emergency situation. So session-switching will be left to the active-session itself. But if the active session dead-locks, we have the same problem as with VTs: an unresponsive system

We can implement emergency-shortcuts which violate this rule, but it will never be a clean solution. Moreover, we will never be able to have global Volume-Up and Volume-Down shortcut handling, no Brightness-keys, no global NumLock handling. Global keyboard handling will be reliant on policies, if implemented at all, and otherwise left to each session to implement.

#4: Quirks

Layout of different quirks working with logind session manager

Upgrading logind to a session-manager has advantages on its own. Most of the code is already available, no external manager is needed, which would have to synchronize with logind. We have direct hardware access in each session and, as mentioned earlier, sessions are self-hosting. A session itself doesn’t need any major modifications to run under this scenario. A session controls itself over the whole runtime and, more importantly, doesn’t depend on another daemon to function properly. Yes, session-management depends on logind, but the session itself, once activated, runs standalone.

To overcome the issues mentioned above, I developed a set of quirks that sit between logind and a session:

UVTD

uvtd (User-Space Virtual-Terminal Daemon) is an implementation of VTs in user-space. It is available as experimental branch in the official kmscon repository. Even if you disable kernel VTs, uvtd can provide legacy VT nodes for backwards compatibility. But in contrast to kernel VTs, uvtd can control the “server side” of each VT and provide a bridge between a fake VT and logind. This allows to run legacy VT applications even if no VTs are available. The current code already allows to run multiple xservers in parallel without any real VT and without modifying a single xserver line of code.

wlsystemc

wlsystemc (Wayland System Compositor) is another research project I was working on. It is a Wayland compositor that displays each client as fullscreen surface. Due to the fast moving Wayland codebase it got slightly out of date during the last year, but it was once working with basic SHM buffers. Anyhow, together with logind, wlsystemc can be redesigned to take over modesetting for basic sessions. For instance, a boot-splash would create a surface in wlsystemc, which itself registers a session in logind for this surface. Once the session gets activated, wlsystemc performs all the heavy DRM operations and the boot-splash does nothing more than simple generic Wayland rendering (i.e., passing custom buffer).

Kernel Log

“No VTs, no kernel log!”. But especially developers want a kernel oops or panic printed on screen whenever they occur. The solution is fblog or drmlog, a kernel module, that does nothing else than printing the kernel log to all connected monitors. logind can register it like any other session and enable or disable it via sysfs. During oops/panic the kernel will activate them automatically and notify logind asynchronously via sysfs. Both modules are still under development, but at least fblog is already working.

Linux Console

For completeness: kmscon can easily replace the linux-console in this setup. It will register a text-console as a normal session just like any other session does.

#5: Conclusion

While I was an enthusiastic advocate of the system-compositor idea, it did become pretty clear, that the performance penalty for input-event forwarding and the huge required kernel API extensions make it very unlikely that it will get accepted in the community. Furthermore, besides global keyboard handling, most technical concerns on the logind idea can be solved with quirks like wlsystemc and uvtd.

This post didn’t discuss or highlight the low-level technical concerns on kernel VTs, but I hope it explains why we get a lot more flexibility and possibilities by moving session handling into user-space. logind can be taught boot-splash awareness, emergency-handling or forced session-switching, which will be very unlikely to get implemented in kernel VTs.

But independent of how we implement it, moving VT handling to userspace and disabling CONFIG_VT still has one major advantage: it simplifies kernel programming a lot. And I know, nearly every DRM developer would be very happy to see this happen!

DRM Render- and Modeset-Nodes

Another year, another Google Summer of Code. This time I got the chance to work on something that I had on my TODO list for quite a long time: DRM Render- and Modeset-Nodes

As part of the X.org Foundation mentoring organization, I will try to pick up the work from Ilija HadzicDave AirlieKristian HoegsbergMartin Peres and others. The idea is to extend the DRM user-space API of the linux kernel to split modeset and rendering interfaces apart. The main usage is to allow different access-modes for graphics-compositors (which require the modeset API) and client-side rendering or GPGPU-users (which both require the rendering API). We currently use the DRM-Master interface to restrict the modeset API to privileged applications. However, this requires SYS_CAP_ADMIN privileges, which is roughly equivalent to root-privileges. With two different interfaces for modeset and rendering APIs, we can apply a different set of filesystem-access-modes to each of them and thus get fine-grained access-control.

Apart from fine-grained access control, we also get some other nice features almost for free:

  • We will be able to run GPGPU clients without any running compositor or event without any display controller
  • We can split modeset objects across multiple nodes to allow multi-seat setups with a single display controller
  • Efficient compositor-stacking by granting page-flip access or full modeset access temporarily to sub-compositors

There are actually a lot of other ideas how to extend this. So I decided to concentrate on the modeset-node / render-node split first. Once that is done (and fully working), I will pick different ideas that depend on this and try to implement them. Considering the lot of work others have put in this already, I think if I get the split merged into mainline, the project will already be a great success. Everything on top of it will be some bonus that will probably take more time to get merged. But lets see, maybe it turns out to be easier than I think and we end up with some of the use-cases merged upstream, too.

Thanks to the X.org Foundation, Google and my GSoC-mentor Dave Airlie for giving me the chance to work on the DRM API! I hope it will be a productive summer.

GSoC-Proposal

If someone is interested in more details, some excerpts from my original GSoC proposal:

Project Description:
Since several years xserver is no longer the only user-space project that makes
use of the kernel DRM API. The introduction of KMS allowed many new projects to
emerge, including plymouth, weston and kmscon. On the other side, OpenCL support
allows applications to make use of DRM without requiring any KMS APIs. Even though
both use-cases work with the current APIs, there are a lot of restrictions that
need to be worked around.

The most problematic concept is DRM-Master. KMS applications are required to be
DRM-Master to perform modesetting, but DRM-Master is tightly coupled to
CAP_SYS_ADMIN/root. On the other side, render clients are required to be assigned
to a DRM-Master so they can get authenticated. This prevents off-screen/offline
rendering without a running compositor.

One possible solution is to split render- and modeset-nodes apart. The DRM control
node can be used as the management node (which is, as far as I understand, what
it was designed for, anyway). A separate static render-node is created for each
DRM device which is restricted to ioctls specific to rendering operations. Instead
of requiring drmAuth() for authorization, we can now use filesystem access-modes.
This allows slightly more dynamic access-control, but the biggest advantage is
that we can do off-screen/offline rendering without a running
compositor/DRM-Master.

On the other side, a modeset-node is a concept to have KMS separated from DRM.
The use-case is to split modeset objects (eg., crtcs, encoders, connectors) across
different modesetting applications. This allows one compositor to use one
CRTC+connector combination, while another compositor (maybe on another seat) can
use another CRTC+external-connector. This doesn't have to be a static setup. On
the contrary, one use-case I am very interested in is a dynamic modeset-object
assignment to temporary clients. This way, a fullscreen application can be granted
page-flip rights from the compositor to avoid context-switches to the compositor
for doing trivial page-flips only.

Deliverables
* Working render-node clients: Preferably an offline OpenCL example and
  a wayland EGL client
* Merged kernel render-node implementation with at least i915 support
* Dynamic kernel modeset-nodes
* "Zero-context-switches" wayland/weston fullscreen client (optional)

Known Problems:
There were several attempts to push render-nodes into the kernel, but all failed
due to missing motivation to finish the user-space clients. Writing up new fancy
APIs is one part, but pushing API changes to such big projects requires the whole
environment to work well with the changes. That's why I want to concentrate on
the user-space side of render-nodes. And I want to finish the render-nodes project
before continuing with modeset-nodes. The idea has been around long enough that
it's time that we get it done.

However, one problem is that I never worked with the low-level X11 stack. The
wayland environment is great for experiments and quite active. I am very familiar
with it and know how to get examples easily running. The xserver, however, is a
huge black box to me. I know the concepts and understand the input and graphics
drivers design. But I never read xserver core code. That's something I'd like to
change during this project. I will probably be limited to DRI and graphics
drivers, but that's a good start.

Another idea that came up quite often is something like gem-fs. It's far beyond
the scope of this project, but it's something I'd like to keep in mind when
designing the API. It's hard to account for something that's only an idea, but
the concepts seem related so I will try to understand the reasons behind gem-fs
and avoid orthogonal implementations.

A few smaller implementation-specific problems are already known, including the
mmap-security problem, static "possible_encoders"/"possible_crtcs" bitsets and
missing MMUs on GPUs. However, there already have been ideas how to solve them
so I don't consider them blockers for render-nodes.