When running servers I want to encrypt the data stored on them. The problem you then pretty quickly run into is that it’s hard to actually boot with an encrypted root. I’ve solved this problem in the past by having a tinysshd in my initramfs which prompts me for a password to unlock the volumes. Though this works, it’s annoying in that the server isn’t able to boot at all, causing any additional monitoring I have to not work. There are also some services on machines that don’t need any encrypted storage and that I’d be happy to have start unattended.
It turns out though, systemd provides us all the tools we need to achieve this kind of split boot.
Setting up the encrypted disks
In my servers there’s usually 2 disks, with a bunch of partitions including a data partition. The data partition is first ecnrypted with LUKS and then turned into a btrfs RAID1.
As a consequence, I have something along the following in
data1 UUID=<UUID> none luks,noauto data2 UUID=<UUID> none luks,noauto
noauto is very important here, we don’t want to attempt to unlock these
disks at boot. The
UUID comes from
blkid /dev/nvme0n1pX etc.
This also provides us with our first building block. For every of these disks,
systemd will automatically generate a service,
systemd-cryptsetup@<NAME>.service. This is done automatically for you by
Having these services is rather handy. You can start them by hand using
systemctl, and systemd will prompt you for the password on the TTY.
Mounting the encrypted disks
Having these cryptsetup services is rather handy, but we need more than that, we need them mounted. Without them being mounted, we won’t be able to use the data stored on them.
Filesystem mounts are defined in
/etc/fstab, and I have something like this
UUID=<UUID> /var/lib/docker btrfs defaults,noauto,noatime,ssd,subvolid=256,subvol=/docker,x-mount.mkdir,firstname.lastname@example.org,email@example.com 0 2 UUID=<UUID> /data btrfs defaults,noauto,noatime,ssd,subvolid=259,subvol=/data,x-mount.mkdir,x-systemd.automount,firstname.lastname@example.org,email@example.com 0 2 UUID=<UUID> /var/cache/pacman/pkg btrfs defaults,noauto,noatime,ssd,subvolid=260,subvol=/pkgcache,x-mount.mkdir,x-systemd.automount,firstname.lastname@example.org,email@example.com 0 2
Make sure that the
UUID here is that of
blkid /dev/mapper/<NAME>, using the
names you used for devices in
Much like in
/etc/crypttab, we need to pass
noauto to our mount options,
to ensure we don’t attempt to mount these filesystems on boot.
The big trick here is that we can declare dependencies using
as part of our mount options.
/etc/fstab in turn is parsed by
.mount units. Their names is the mount point, which each
-, so for example
var-lib-docker.mount. I strongly suggest
you take a stroll through
to understand all the options at your disposal.
x-mount.mkdir option will automatically create the target directory
for us if it doesn’t exist, prior to mounting. This avoids silly scenarios
like a mount failing because we forgot to create
Don’t worry about the
x-systemd.automount option, we’ll discuss that a bit
Depending on the encrypted mount
Next up, we’ll need to update a service. What we want to do is ensure that a service is only started after that encrypted filesystem has been mounted. If the disk is already unlocked this will be a no-op, if not, it should trigger the system to prompt us for the password and mount the filesystem.
We achieve this through the
After of a service unit. In them
we specify additional dependencies on mount units (and anythign else you’d
like). You can edit one using
systemctl edit, lets try with
[Unit] Requires=var-lib-docker.mount After=var-lib-docker.mount
All together now
With all of this in place, when you
systemctl start docker.service systemd
will now try to start
systemd-fstab-generator. We’ve specified through
x-systemd.requires that in order to start
var-lib-docker.mount, we need
systemd-cryptsetup@<NAME> services started. This in turn will cause systemd
to prompt you for the decryption passwords on the TTY. Once the filesystems
have been unlocked and successfully mounted our
docker.service will start.
Since the disks are now decrypted, any other
.mount units won’t prompt you
for a password and will just mount instead.
Isn’t that lovely?
Be careful not to start these services at boot, i.e don’t
them. If you don’t want to have to start all services individually, make a
.target that you then start, and override the
WantedBy of the
units so they don’t get started as part of
multi-user.target. Only once
you’ve done that should you
systemctl enable the services.
Now one last thing, that
automount is another
type of unit. systemd effectively watches for file access to the mount point,
and when it happens it will automatically start the associated mount unit,
causing the cryptsetup services to start making systemd prompt you for the
password. This means that if you were to
cd /data based on the setup I’ve
showed you here, and the disks aren’t decrypted and mounted yet, you’ll be
prompted for you decryption passwords at that point.
You can see automount units in the
mount output, as there will be an
additional entry for them:
systemd-1 on /data type autofs (rw,relatime,fd=46,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18402) /dev/mapper/data1 on /data type btrfs (rw,noatime,ssd,space_cache,subvolid=259,subvol=/data)
The second mount, the
type btrfs one, only exists after
started, but the
autofs one exists on boot.
Automounts are usually only used together with network mounted filesystems, but it provides a nice little usability improvement in our case too.
Now you might be wondering, why don’t you have the
automount option on
/var/lib/docker? Well, unfortunately because it trips up Docker. I run
Docker with the
btrfs volume driver, but unfortunately at startup Docker
only notices the
autofs mount and concludes it shouldn’t load the btrfs
snapshotter plugin. In order to avoid that, and since realistically Docker
is the only thing that should be doing stuff under
/var/lib/docker, I omit