When running servers I want to encrypt the data stored on them. The problem you then pretty quickly run into is that it’s hard to actually boot with an encrypted root. I’ve solved this problem in the past by having a tinysshd in my initramfs which prompts me for a password to unlock the volumes. Though this works, it’s annoying in that the server isn’t able to boot at all, causing any additional monitoring I have to not work. There are also some services on machines that don’t need any encrypted storage and that I’d be happy to have start unattended.

It turns out though, systemd provides us all the tools we need to achieve this kind of split boot.

Setting up the encrypted disks

In my servers there’s usually 2 disks, with a bunch of partitions including a data partition. The data partition is first ecnrypted with LUKS and then turned into a btrfs RAID1.

As a consequence, I have something along the following in /etc/crypttab:

data1	UUID=<UUID>	none	luks,noauto	
data2	UUID=<UUID>	none	luks,noauto

The noauto is very important here, we don’t want to attempt to unlock these disks at boot. The UUID comes from blkid /dev/nvme0n1pX etc.

This also provides us with our first building block. For every of these disks, systemd will automatically generate a service, systemd-cryptsetup@<NAME>.service. This is done automatically for you by systemd-cryptsetup-generator.

Having these services is rather handy. You can start them by hand using systemctl, and systemd will prompt you for the password on the TTY.

Mounting the encrypted disks

Having these cryptsetup services is rather handy, but we need more than that, we need them mounted. Without them being mounted, we won’t be able to use the data stored on them.

Filesystem mounts are defined in /etc/fstab, and I have something like this in it:

UUID=<UUID>	/var/lib/docker	btrfs	defaults,noauto,noatime,ssd,subvolid=256,subvol=/docker,x-mount.mkdir,x-systemd.requires=systemd-cryptsetup@data1.service,x-systemd.requires=systemd-cryptsetup@data2.service	0	2
UUID=<UUID>	/data	btrfs	defaults,noauto,noatime,ssd,subvolid=259,subvol=/data,x-mount.mkdir,x-systemd.automount,x-systemd.requires=systemd-cryptsetup@data1.service,x-systemd.requires=systemd-cryptsetup@data2.service	0	2
UUID=<UUID>	/var/cache/pacman/pkg	btrfs	defaults,noauto,noatime,ssd,subvolid=260,subvol=/pkgcache,x-mount.mkdir,x-systemd.automount,x-systemd.requires=systemd-cryptsetup@data1.service,x-systemd.requires=systemd-cryptsetup@data2.service	0	2

Make sure that the UUID here is that of blkid /dev/mapper/<NAME>, using the names you used for devices in /etc/crypttab.

Much like in /etc/crypttab, we need to pass noauto to our mount options, to ensure we don’t attempt to mount these filesystems on boot.

The big trick here is that we can declare dependencies using x-systemd.requires as part of our mount options. /etc/fstab in turn is parsed by systemd-fstab-generator, yielding .mount units. Their names is the mount point, which each / replaced by -, so for example var-lib-docker.mount. I strongly suggest you take a stroll through systemd.mount to understand all the options at your disposal.

The x-mount.mkdir option will automatically create the target directory for us if it doesn’t exist, prior to mounting. This avoids silly scenarios like a mount failing because we forgot to create /data.

Don’t worry about the x-systemd.automount option, we’ll discuss that a bit later.

Depending on the encrypted mount

Next up, we’ll need to update a service. What we want to do is ensure that a service is only started after that encrypted filesystem has been mounted. If the disk is already unlocked this will be a no-op, if not, it should trigger the system to prompt us for the password and mount the filesystem.

We achieve this through the Requires and After of a service unit. In them we specify additional dependencies on mount units (and anythign else you’d like). You can edit one using systemctl edit, lets try with docker.service:


All together now

With all of this in place, when you systemctl start docker.service systemd will now try to start var-lib-docker.mount. var-lib-docker.mount is generated by systemd-fstab-generator. We’ve specified through x-systemd.requires that in order to start var-lib-docker.mount, we need systemd-cryptsetup@<NAME> services started. This in turn will cause systemd to prompt you for the decryption passwords on the TTY. Once the filesystems have been unlocked and successfully mounted our docker.service will start. Since the disks are now decrypted, any other .mount units won’t prompt you for a password and will just mount instead.

Isn’t that lovely?

Be careful not to start these services at boot, i.e don’t systemctl enable them. If you don’t want to have to start all services individually, make a custom .target that you then start, and override the WantedBy of the units so they don’t get started as part of multi-user.target. Only once you’ve done that should you systemctl enable the services.

Bonus section

Now one last thing, that x-systemd.automount thing. automount is another type of unit. systemd effectively watches for file access to the mount point, and when it happens it will automatically start the associated mount unit, causing the cryptsetup services to start making systemd prompt you for the password. This means that if you were to cd /data based on the setup I’ve showed you here, and the disks aren’t decrypted and mounted yet, you’ll be prompted for you decryption passwords at that point.

You can see automount units in the mount output, as there will be an additional entry for them:

systemd-1 on /data type autofs (rw,relatime,fd=46,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18402)
/dev/mapper/data1 on /data type btrfs (rw,noatime,ssd,space_cache,subvolid=259,subvol=/data)

The second mount, the type btrfs one, only exists after data.mount has started, but the autofs one exists on boot.

Automounts are usually only used together with network mounted filesystems, but it provides a nice little usability improvement in our case too.

Now you might be wondering, why don’t you have the automount option on /var/lib/docker? Well, unfortunately because it trips up Docker. I run Docker with the btrfs volume driver, but unfortunately at startup Docker only notices the autofs mount and concludes it shouldn’t load the btrfs snapshotter plugin. In order to avoid that, and since realistically Docker is the only thing that should be doing stuff under /var/lib/docker, I omit the x-systemd.automount.