hotplug - Dynamic Hardware Configuration

Posted on October 8, 2004 by Chris Lumens in hotplug, presentations.

hotplug is a Linux kernel subsystem that allows you to load modules and run arbitrary scripts based on events like hardware being attached and detached. It does this through a complex system of shell scripts, magic environment and map files, events triggered in the kernel, and so forth. Understanding hotplug is important since it’s only going to get more widely used in later kernel versions. I gave this presentation to the GT LUG to inform people of how hotplug works. No pictures.

What is hotplug?

hotplug is a system for managing devices that can be dynamically attached to and removed from the system while it’s running. The most obvious use for this system is handling USB and firewire devices, though it also handles PCI (32-bit PCMCIA - or CardBus - devices are really PCI in disguise), tape drives, SCSI devices, devices requiring firmware to be loaded into them, input devices, and more. It consists of a kernel part and a userland part. I’ll only cover the userland part in any depth, as users should not ever have to dive into the kernel half.

The problem hotplug tries to solve is a very difficult one, especially for the land of Unix where hardware is accessed through device nodes and includes real permission checking. Device nodes themselves are special little files in /dev that are referenced through static major and minor numbers which must be registered somewhere. They must also be given owners and groups, as well as permissions for access. For static devices like an internal hard disk or a sound card, this system works fairly well.

However once dynamic devices are added to the system, the traditional Unix approach falls apart. When a USB drive is plugged into the system, what is it even called? A device node needs to be automatically created before ownerships and permissions can be assigned to it. Once this device file is created, the system will also need to know who to assign ownership to and what permissions everyone else should have. Unfortunately, the kernel can’t really keep track of this information without some sort of outside help. This is where hotplug comes in, but it tried to be much more generic. Instead of just handling device node creation and permissions, hotplug allows arbitrary scripts to be run. hotplug was introduced in the 2.4 kernel series and has become much more important in the 2.6 series.

hotplug refers to anything that happens as an action or event. The two main kinds are add and remove, though they may also be referred to as register and unregister depending on which subsystem files you’re looking at. Get used to this sort of consistency.

In this presentation, I will first cover how hotplug is put together and a trace of an event going through the system. I’ll also cover some examples of what you can use hotplug to do, before finally discussing some of the problems with the hotplug system and outlining an alternate implementation.

Components of the hotplug System

The default hotplug implementation comes from the [Linux Hotplugging Project] (http://linux-hotplug.sourceforge.net) and is maintained by Greg Kroah-Hartman. It is written entirely in shell and little specially formatted config files, though there’s no reason it couldn’t be written in Perl, C, or any other language you want. The default implementation has a whole lot of files that call each other with some magic parameters and environment settings. Here’s what you have to deal with:

/proc/sys/kernel/hotplug - Contains the name of the hotplug program to run when an event occurs. By default, this is /sbin/hotplug, or /dev/null to disable hotplug.
/sbin/hotplug - The main hotplug script, called by the kernel for every event with a single argument - the name of a subsystem. Other information is passed to the script through environment variables.
/etc/hotplug.d/ - Subsystem-specific scripts run by the main script as well as default scripts that always get run. Basically, this file just decides which ones should be called next to do the real work.
/etc/hotplug/ - This directory contains all the important subsystem-specific scripts and some special configuration files. This includes a lot of stuff:
- /etc/hotplug/blacklist - Some modules cause problems when loaded, so any that are listed in this file will never be loaded by a hotplug script.
- /etc/hotplug/*.agent - Each subsystem has an agent file that handles the various types of actions and runs any special scripts. This is where the real work gets done. The agent files get one parameter - the action. All other information is passed through the environment.
- /etc/hotplug/*.rc - Most subsystems also have an rc file to handle something called coldplugging. When the kernel is booting, it fakes hotplug events for all the devices already plugged in, and these events are handled by the rc files. The rc files get one parameter - the action. All other information is passed through the environment.
- map files - These files map device descriptions to module names or arbitrary scripts to be run. USB makes the most extensive use of this feature, though other subsystems use it as well.

As you can see, there’s a lot of parts. This is good in that you can pretty thoroughly extend the system - you can hook in via a subsystem-specific script in /etc/hotplug.d, modifying the agent and rc files, or by modifying a map file and adding in your own script to be run when that device is detected. All this makes it easy on the package maintainer to add their own thing, but difficult on people who want to understand how it all fits together. Debugging is not hotplug’s strongest point.

How a hotplug Event Works

Just staring at a big list of files isn’t very helpful. To really understand how it all fits together, we should take a look at how the entire system works together to handle an event. We’ll examine plugging in a USB keychain drive, though the process will be similar for any hotplug-enabled driver. I won’t cover any special add-on functionality like automounting until the next section.

USB key is attached to computer. The USB hub driver detects this and prints the following:

kernel: hub.c: new USB device 00:1d.0-1, assigned address 2
kernel: usb.c: USB device 2 (vend/prod 0x90a/0x1001) is not claimed by any active driver.

The kernel calls the hotplug script, based on the contents of the /proc/sys/kernel/hotplug file with a subsystem (the word “usb”) as an argument. In this example, the equivalent shell command would be:

# $(cat /proc/sys/kernel/hotplug) usb

/sbin/hotplug calls any scripts in /etc/hotplug.d/usb (there are none by default) and then the scripts in /etc/hotplug.d/default via the following loop:

for I in "${DIR}/$1/"*.hotplug "${DIR}/"default/*.hotplug ; do
        if [ -f $I ]; then
                test -x $I &amp;&amp; $I $1 ;
        fi
done

Under the default hotplug installation, only default.hotplug is run. The single argument to this script is also the name of the subsystem - usb in our case.

/etc/hotplug.d/default/default.hotplug is called with the following environment:

DEVFS=/proc/bus/usb OLDPWD=/ PATH=/bin:/sbin:/usr/sbin:/usr/bin ACTION=add
PWD=/etc/hotplug HOME=/ SHLVL=2 DEVICE=/proc/bus/usb/001/002
INTERFACE=8/6/80 PRODUCT=90a/1001/100 TYPE=0/0/0 DEBUG=yes _=/bin/env

While this environment isn’t used by the default script, it will be useful to the agent scripts and gets inherited by them when called. default.hotplug figures out which agent scripts need to be run and calls them.

/etc/hotplug/usb.agent is called with the inherited environment from above. The important environment variables are ACTION - which is used to determine what to do - and PRODUCT, INTERFACE, and TYPE which are used to determine exactly which scripts to run. Here, PRODUCT is 90a/1001/100 which corresponds to the vendor ID and product ID. The USB agent script then goes through the following steps:
1. Since $ACTION is “add”, it starts by loading kernel modules for the hardware as described in the modules.usbmap. This file is generated by modutils. Each line in this file contains a module to load should the values on the rest of the line match what we’re looking for. In this case, we are looking for a line with the vendor ID of 0x90a and the product ID of 0x1001. The following line matches:
```
# usb module  match_flags idVendor idProduct bcdDevice_lo bcdDevice_hi bDeviceClass bDeviceSubClass bDeviceProtocol bInterfaceClass bInterfaceSubClass bInterfaceProtocol driver_info
usb-storage   0x000f      0x090a   0x1001    0x0100       0x0100       0x00         0x00            0x00            0x00            0x00               0x00               0x00000000
```
This line says to load the usb-storage module on match, so hotplug loads that module along with any others that match.
1. It then examines the usb.handmap file which has an identical format to the previous one. This file exists to load modules that somehow weren’t mapped in modules.usbmap and ships with hotplug. For our USB drive example, there are no matches so no new modules are loaded. Under 2.6, this file isn’t needed because these modules are taken care of elsewhere.
2. Finally, we get to the really configurable part. Here, the USB agent file does similar module loading for usb.usermap and anything in /etc/hotplug/usb/*.usermap. This allows you to add your own mappings for module loading and running arbitrary scripts. If a match is made by the first column is not a module, any script by the same name in /etc/hotplug/usb will be run, inheriting that same environment from earlier. In this basic case, no matches were made so nothing else happens.
At this point, everything has run that will be run for the USB drive so execution returns to the kernel.

Examples

Automating gphoto

gphoto is a program for manipulating digital cameras. For USB cameras, you can plug the camera into a USB port and have gphoto copy the pictures off the camera and onto your hard drive. This is a good place for using hotplug to automate the task.

The first step is to generate a map file that contains your camera’s entry. The print-usb-usermap program that ships with libgphoto2 can output a map file describing all the cameras it supports:

print-usb-usermap > /etc/hotplug/usb/usbcam.usermap

This map file will run a usbcam script every time a camera is attached to the system. You can then make a /etc/hotplug/usb/usbcam script to do whatever you wanted. The simplest script making use of gphoto would just copy the pictures off. Your script can make use of all those environment variables that hotplug keeps passing around, as well.

#!/bin/bash
( cd /dest && gphoto2 -P -R --auto-detect )

Backups

hotplug can be used to do automatic backups when a disk is attached. This quick little example isn’t very automatic, though, because figuring out the device name assigned to a hotplugged disk is very difficult. You’ll need things like udev and sysfs to help with that. As before, you’ll need to make a map file entry to match your particular disk, but for this match you should point it at a script named syncup. Then, your script could look something like this:

#!/bin/bash
mount -t ext3 /dev/sda1 /backup &&
rsync -Pavvz --delete /home /backup/home &&
umount /backup

In this example, I have assumed the name of the device as well as skipped the whole problem of unmounting the disk when the device is detached. A sophisticated backup script would be much longer and have lots of error checking, but this gives you the basic idea of what’s possible. I originally intended this example to use firewire, but the current agent script doesn’t support map file hooks like the USB agent does. You could, of course, change this on your own system.

Problems

The problem hotplug tries to solve is a very difficult one, and it provides an extremely flexible and extensible set of scripts. However, its current implementation is not without some significant problems. Unfortunately it’s also the most popular implementation right now so these are problems we are going to have to face for some time, unless significant patches are made. As the 2.6 kernel gets more widely used, hotplug is also only going to get more important.

Moves the kernel into shell scripts - hotplug represents a major chunk of kernel functionality. It involves modules loading, creating devices nodes, and other such kernel tasks. In order to have a user space shell script perform these tasks, the kernel must package up all its understanding of the device via the clumsy environment variable system and pass it into the script. Since the kernel already knows everything, why not handle this itself?
Firmware loader goofiness - Moving the kernel into user space results in code like this:

 if [ -f $FIRMWARE_DIR/$FIRMWARE ]; then
     echo 1 > $SYSFS/$DEVPATH/loading
     cp $FIRMWARE_DIR/$FIRMWARE $SYSFS/$DEVPATH/data
     echo 0 > $SYSFS/$DEVPATH/loading
 else
     echo -1 > $SYSFS/$DEVPATH/loading
 fi

Too complex - There are simply too many scripts executing each other, reading various map files, and passing around magic environment variables. The base hotplug system is already too large. A distribution making extensive use of the hooks for hardware components could easily end up with dozens of scripts for each subsystem. As it stands, hotplug is difficult to follow and debug. Simpler implementations can be achieved, as I will show below.
Tries to do everything - It includes hooks for handling network interfaces including automatically bringing them up and down, provides for map file matching on all sorts of parameters, attempts to handle coldplugging before it even has enough information to do so, and more. Much of this could be handled in better places - for example, the distribution’s network scripts, running scripts from modules.conf, or others.
Removing is an afterthought - Lots and lots of attention gets paid to the adding side of the process, but removing is almost completely overlooked. The USB agent wants you to set a REMOVER variable that points to a device-specific script that will be run on the remove action. No other agent script gives you any hook to run something on remove - or even handles the remove case. The network agent only tried to kill dhcpcd, and nothing else. For something that tries to work its way into everything, this is unacceptable.
Poorly written - For various reasons, this is not well-written shell scripting. It assumes Red Hat-specific things like /etc/sysconfig and /etc/init.d/functions that not everyone has (or wants). Additionally, it make gratuitous use of the colon operator, uses backticks instead of the $( ) construct, and uses inconsistent naming. If you’re going to move the kernel into the shell, at least write it well.

The Probulator - An Alternate Implementation

Anyone who’s motivated enough can write their own hotplug replacement. All that’s required is to write a program that accepts the arguments and environment variables the kernel passes and loads the appropriate modules. If you’re really motivated, you can also provide hooks for running arbitrary scripts. However, much of what hotplug does or tries to do is completely optional.

One such alternate implementation is the probulator, part of the Genthree distribution. The probulator handles the USB, firewire, PCI, and firmware agents and the add and remove events. It can load and unload modules and firmware blobs. It does not currently allow for arbitrary scripts being run or add-on map files but these would be trivial to implement. It currently does everything the developers require in a single 255 line shell script.

The current CVS version of the probulator can be downloaded from here.

Conclusions

hotplug is a complex subsystem that first appeared in 2.4 to handle USB and firewire devices, but has grown well beyond that original purpose. Under 2.6, it handles all sorts of hardware events now including firmware loading, PCI devices, network cards, and so forth. It provides an extensible system for writing your own scripts that hook in to the events. However, it is also a complex tangle of shell scripts that move kernel functionality out into user land. As 2.6 becomes more and more popular, this subsystem will become even more important to regular users who will have to deal with it every day.