Since I bought a new root server at netcup.de I was finally able to realize a new setup I had in mind. I’m using NixOS on one of my private laptops for quite some time and really like the concepts. That’s why I decided to install NixOS on my new server.

I never worked with ZFS but heard and read a lot of good reasons why it make sense to use it. In addition, encrypted disks should be the default nowadays, so this was another requirement for my new setup. Having an encrypted disk requires me to enter the encryption password when booting the server. But at this point, the OpenSSH daemon is not yet loaded. Although it is possible to have a VNC connection to the server via netcup’s server control panel, this should not be the way to go. So I needed to find a tutorial written by smarter people explaining how to SSH to my server to enter the password.

netcup.de provides official images to install a new operating system. Among those images are the usual suspects like Archlinux, CentOS, Debian, … When using such a provided image, it is possible to let the automatic installation take care of partitioning and formatting the disks. NixOS was not in the list of provided images, so I had to take the manual way.

Disclaimer

I’m doing this in my free time and am not a professional. My knowledge is based on experience, tutorials and documentation. So the steps I took are from different sources and try-and-error until I was able to write a step-by-step guide that is working for me. Most of the sources that helped me successfully run the installation are linked in this post.

The purpose of this article is to have my steps written down for future-me. And maybe this article could help some people struggling with a similar setup.

Installation

My steps are mainly based on the following sources:

Booting to NixOS ISO

Download NixOS minimal installation iso. Get the FTP credentials from netcup.de’s server control panel (SCP) and upload the iso. Once the iso is uploaded, mount it and change the boot order using SCP. The server should now start from your uploaded NixOS iso.

Once the server is started you should be able to login as root.

$ sudo -i

Partioning

# Delete all existing partitions

$ sgdisk --zap-all /dev/sda

As far as I can say, by default a netcup.de server does not support UEFI. At least /sys/firmware/efi does not exist. So the following steps are BIOS-based.

# List all devices

$ ls -l /dev/disk/by-id/

lrwxrwxrwx 1 root root 9 Mar 6 11:10 scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-0 -> ../../sda

# Create variables for easier access

$ SDA_ID="$(ls /dev/disk/by-id/ | grep '^[scsi]')"
$ DISK=/dev/disk/by-id/$SDA_ID

Create necessary partitions. I’m using sgdisk with some arguments to create new partitions, specifically:

  • -n <a>:<b>:<c>: New partition with number a, starting at sector b and ending at sector c. Note that specifying a partition number of 0 always takes the first available number.
  • -t: The partition’s type code.
    • EF02: BIOS boot partition
    • EA00: Freedesktop $BOOT
    • 8200: Linux swap
    • BF01: Solaris /usr & Mac Z
  • -c: Name of the partition
  • The last argument ($DISK) is the device on which the partition will be created.
# Partition for GRUB bootloader

$ sgdisk -n 0 : 0 :+1MiB -t 0:ef02 -c 0:grub $DISK

# Boot partition

$ sgdisk -n 0 : 0 :+1GiB -t 0:ea00 -c 0:boot $DISK

# Swap partition

$ sgdisk -n 0 : 0 :+6GiB -t 0:8200 -c 0:swap $DISK

# Partition for ZFS

$ sgdisk -n 0 : 0 : 0 -t 0:BF01 -c 0:ZFS $DISK

Note: Remove the spaces between the 0 and :.

Create some variables for easier access in the following steps.

$ BOOT=$DISK-part2
$ SWAP = $DISK-part3
$ ZFS=$DISK-part4

Configure ZFS

On the ZFS partition the ZFS pool and two datasets will be created. Two datasets are a good idea, because it enables a separation between NixOS files and data. Everything NixOS related can be reinstalled, because it will be configured in /etc/nixos/configuration.nix. But data cannot be just reinstalled of course. So I want to create ZFS snapshots that can be backed up and restored if necessary. The structure will look like this:

  • BIOS (Partition)
  • BOOT (Partition)
  • SWAP (Partition)
  • ZFS (Partition)
    • rpool (Encrypted ZFS Pool)
    • home (ZFS dataset)
    • root (ZFS dataset)
      • nixos

Create encrypted ZFS pool

Let’s create an encrypted ZFS pool. We need to specify a password that will be asked for every time the server starts.

$ zpool create -o ashift=12 -o altroot="/mnt" -O mountpoint=none -O encryption=aes-256-gcm -O keyformat=passphrase rpool $ZFS
Enter passphrase:
Re-enter passphrase:

Create ZFS datasets

Next step is creating three ZFS datasets inside the pool.

# Root dataset which will not be mounted

$ zfs create -o mountpoint=none rpool/root

# Dataset for NixOS. Set mountpoint to legacy because it will be mounted by NixOS.

$ zfs create -o mountpoint=legacy rpool/root/nixos

# Dataset for data. This will also be mounted by NixOS.

# Note that automatic snapshots are activated.

$ zfs create -o mountpoint=legacy -o com.sun:auto-snapshot=true rpool/home

# Activate compression for our data.

$ zfs set compression=lz4 rpool/home

Create and mount filesystems

Now that our ZFS pool and datasets are configured we can mount them.

# Mount NixOS root dataset to /mnt

$ mount -t zfs rpool/root/nixos /mnt

# Create new directory to mount home dataset

$ mkdir /mnt/home
$ mount -t zfs rpool/home /mnt/home

# Create filesystem for boot partition

$ mkfs.vfat $BOOT

# Create directory and mount boot partition

$ mkdir /mnt/boot
$ mount $BOOT /mnt/boot

# Set up swap area and enable it

$ mkswap -L swap $SWAP
$ swapon $SWAP

Generate NixOS configuration

All partitions should be prepared now. That means that NixOS can now create a configuration based on /mnt. Remember that the created partitions are mounted into /mnt. So NixOS will see them und create corresponding configuration files.

$ nixos-generate-config --root /mnt

We should have two new files:

  1. /mnt/etc/nixos/configuration.nix
  2. /mnt/etc/nixos/hardware-configuration.nix

Check /etc/nixos/hardware-configuration.nix. You should see something similar to this:

...
fileSystems."/" =
{ device = "rpool/root/nixos";
fsType = "zfs";
};

fileSystems."/home" =
{ device = "rpool/home";
fsType = "zfs";
};

fileSystems."/boot" =
{ device = "/dev/disk/by-uuid/<...>";
fsType = "vfat";
};

swapDevices =
[ { device = "/dev/disk/by-uuid/<...>"; }
];
...

Determine and note down your hostId by running:

$ head -c 8 /etc/machine-id

Now we need to update some settings in /mnt/etc/nixos/configuration.nix before starting the installation of NixOS.

# Update the following lines

boot.loader.grub.device = "/dev/sda";
boot.supportedFilesystems = ["zfs"];
boot.zfs.requestEncryptionCredentials = true;

networking.hostId = "<whatever you noted down>";
networking.hostName ="<your choice>"
networking.useDHCP = true;

# ZFS services

services.zfs.autoSnapshot.enable = true;
services.zfs.autoScrub.enable = true;

Of course you can change the configuration according to your needs, like the time zone, some preinstalled tools or create a user for yourself that is member of the wheel group. And it is always a good idea to change the SSH port from 22 to something not so obvious.

...
environment.systemPackages = with pkgs; [
vim
git
];
...
services.openssh = {
enable = true;
ports = [<not 22>];
};
...
users.users.florian = {
isNormalUser = true;
extraGroups = [ "wheel" ]; # Enable ‘sudo’ for the user.
openssh.authorizedKeys.keys = ["<my SSH public key>"];
};
...

Now the configuration is finished and we can start the installation by running:

# Run the installer

$ nixos-install

The installation will take some minutes. Afterwards, you can shutdown the system.

$ poweroff

Reset boot order

With netcup’s server control panel I unmounted the NixOS iso file and started the server afterwards. If everything worked out the server should show a prompt to enter the disk encryption password. Until now, it is only possible to enter the password by establishing a VNC connection to the server via SCP.

For the first run(s) I had the problem that although I entered the correct password it was not accepted. My solution was to reboot and in GRUB choose “NixOS - All configurations”.

SSH connection to the server to enter encryption password

Assuming the server is starting as expected it should be configured in a way that it is possible to enter the disk encryption password via an established SSH connection. So there will be no need to log in to SCP and open a VNC connection.

For this I am using Dropbear, a small ssh server/client.

# Create a key and note down the public key that gets displayed.

$ nix-shell -p dropbear --command "dropbearkey -t ecdsa -f /tmp/initrd-ssh-key"

# Create a new directory

$ sudo mkdir /var/dropbear

# Move key

$ sudo mv /tmp/initrd-ssh-key /var/dropbear/

Open /etc/nixos/configuration.nix and add the following lines to an appropriate position:

{ config, pkgs, lib, ... }:
...
boot.initrd.network = {
  enable = true;
  ssh = {
    enable = true; # Use a different port than your usual SSH port!
    port = 2233;
    # hostECDSAKey is deprecated with NixOS 20.09, see Update 1
    hostECDSAKey = /var/dropbear/initrd-ssh-key; # All users being a member of the "wheel" group are allowed to connect and enter the password.
    authorizedKeys = with lib; concatLists (mapAttrsToList (name: user: if elem "wheel" user.extraGroups then user.openssh.authorizedKeys.keys else []) config.users.users);
  };
  postCommands = ''
  echo "zfs load-key -a; killall zfs" >> /root/.profile
  '';
};

To activate the configuration, run:

# Check if we can build the configuration

$ sudo nixos-rebuild build

# Activate the new configuration

$ sudo nixos-rebuild switch
$ sudo reboot

On your local machine, add the following lines to .ssh/config.

Host <some-name-for-booting>
HostName <ip address>
User root # important!
Port 2233
UserKnownHostsFile ~/.ssh/known_hosts.boot

This is an SSH configuration for your server. We will use this connection for entering the disk encryption password.

So let’s connect to the server and enter the disk encryption password:

$ ssh <some-name-for-booting>
The authenticity of host [<ip address>]:2233 ([<ip address>]:2233) cannot be established.
ECDSA key fingerprint is SHA256:<...>.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added [<ip address>]:2233 (ECDSA) to the list of known hosts.
Enter passphrase for rpool:
1 / 1 key(s) successfully loaded
~ # Connection to <ip address> closed by remote host.
Connection to <ip address> closed.

In my local .ssh/config I added the following lines:

Host <some-name>
HostName <ip address>
User florian
Port <not 22>

The disk should now be decrypted and it is possible to connect to our server:

$ ssh <some-name>
Last login: ...

That should be it. If you are reading this and it helped a bit, I’m always happy to hear from you.

Update 1

With NixOS 20.09, boot.initrd.network.ssh.hostECDSAKey is deprecated, read this for necessary changes.

So long