NixOS: Enabling LXD virtual machines using Flakes

Nixpkgs is an ever-increasing collection of software packages for Nix and NixOS. Even with more than 80,000 packages, you easily run in a situation where there is a functionality that is not yet implemented.

Earlier this year, we wrote a tutorial on how to implement your own package in NixOS. The implementation of the package in the system was tedious and required keeping track of the package you wanted to implement.

Let’s dive into a practical example: LXD is a container and virtual machine manager. It endorses virtual machines natively since LXD 4.0. However, running on NixOS breaks the feature, and we have to find workarounds to use it. To allow LXD to start virtual machines on NixOS, we need a fix, and we want it to be both reproducible and portable.

To do so, we are using the lxd-agent feature which effectively fix the aforementionned issue. It was released by astridyu, a contributor to nixpkgs. It has not yet been implemented in the Master branch of nixpkgs and is, as of 19/04/2022, a pull request.

Our goal is to integrate the pull request to our nixpkgs to be able to start virtual machines. For this, we will use Flakes. It is an experimental feature of the Nix package manager that allows reproducibility in the deployment of dependencies. We use it to assemble different nixpkgs together. It is a trait known as composability.

If the pull request has been merged, this tutorial becomes irrelevant. To get the functionality, update your Nix or NixOS to the current version of the master branch.

Prerequisites

Step 1 – Installing ZFS

ZFS is a filesystem that we use to generate storage pools in LXD. It is the only filesystem supported by this method, for now.

ZFS must be installed on the NixOS machine prior to the installation of LXD. To do so, go to your configuration.nix file located in /etc/nixos/ and add the following lines to your configuration.

boot = 

  initrd.supportedFilesystems = [ "zfs" ];
  supportedFilesystems = [ "zfs" ];
  zfs.requestEncryptionCredentials = true;


  loader.grub.copyKernels=true;


  kernelParams= [ "nohibernate" ];
;

services.zfs = 

  autoScrub.enable = true;
  autoSnapshot.enable = true;
;

In the file, the networking.hostId property must be set. It is a bit tricky because it only accepts 32bit ID that you generate on the terminal with head -c 8 /etc/machine-id.

With this ID, go to your configuration file and set it.

networking = 
  hostId = "<id>";
;

Now, rebuild the switch drive. This is achieved by doing sudo nixos-rebuild switch.

If the build fails because you already have LXD installed, refer to this section of the article.

Step 2 – Installing Flakes

To install Flakes, simply put, in configuration.nix:

nix = 
  package = pkgs.nixFlakes;
  extraOptions = ''experimental-features = nix-command flakes'';
;

You must rebuild your switch drive again.

Getting started with Flakes

We will draw on the power of Flakes to patch our system by combining different versions of nixpkgs. The versions we will use are:

You can track the pull request with its number: #166858.

Then, we will proceed to override the LXD package from the nixpkgs master branch version with astridyu’s nixpkgs lxd-vms branch version to get the lxd-agent feature applied onto LXD.

To use Flakes, it is required to create a flake.nix file in /etc/nixos. This is done by doing sudo nano /etc/nixos/flake.nix.

In this file, we describe which nixpkgs repository we want to add to our configuration, and how to merge them.

Note: Verify that your hostname, is defined under the networking.hostName field in your configuration. It is required.

  description = "NixOS Configuration of LXD";


  inputs = 

    nixpkgs.url = "nixpkgs/master";


    nixpkgs-lxdvm = 
      url = "github:astridyu/nixpkgs/lxd-vms";
    ;
  ;


  outputs =  self, nixpkgs, nixpkgs-lxdvm :
    let
      system = "x86_64-linux";

      overlay-lxdvm = final: prev: 
         lxdvm = import nixpkgs-lxdvm 
           inherit system;

           config.allowUnfree = true;
         ;
      ;
    in 
      nixosConfigurations."<hostname>" = nixpkgs.lib.nixosSystem 
        inherit system;
        modules = [

          ( config, pkgs, ... :  nixpkgs.overlays = [ overlay-lxdvm ]; )
          ./configuration.nix
        ];
      ;
    ;

Installing LXD

We must modify our configuration.nix to enable LXD virtualization.

boot.kernelModules = ["vhost_vsock"];


virtualisation = 
  lxd = 
    enable=true;


    package = pkgs.lxdvm.lxd.override useQemu = true;;
    recommendedSysctlSettings=true;
  ;


;

We rebuild the switch drive. It is required to use the --impure option upon rebuilding if it is asked by the system.

The reason behind this is that Flakes runs in pure evaluation mode which is underdocumented. Flakes forbids the usage of absolute paths, which could cause the evaluation to be impure. This may happen because we are running unsupported packages.

Getting started with LXD and Virtual Machines

To get started with LXD, we recommend that you initialize a storage pool. This is done by sudo lxd init.

Afterwards, questions are asked concerning the first storage pool that LXD creates. We recommend this configuration for the first usage.

Would you like to use LXD clustering? (yes/no) [default=no]: no
Do you want to configure a new storage pool? (yes/no) [default=yes]: yes

Name of the new storage pool [default=default]: test-storage

Name of the storage backend to use (btrfs, dir, zfs) [default=zfs]: zfs

Create a new ZFS pool? (yes/no) [default=yes]:

Would you like to use an existing empty block device (e.g. a disk or partition)? (yes/no) [default=no]:

Size in GB of the new loop device (1GB minimum) [default=30GB]:

Would you like to connect to a MAAS server? (yes/no) [default=no]:

Would you like to create a new local network bridge? (yes/no) [default=yes]:

What should the new bridge be called? [default=lxdbr0]: test-bridge

What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]:

What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]:

Would you like the LXD server to be available over the network? (yes/no) [default=no]:

Would you like stale cached images to be updated automatically? (yes/no) [default=yes]:

Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]:

When there is no response, default is assumed.

Once the storage pool is configured, new virtual machines can be initialized with lxc launch --vm -s . It automatically downloads the image of your virtual machine and sets one up.

To check your virtual machines, do lxc ls.

Common issues

If the rebuild fails, it is due to LXD already being installed. This leads to nixos-rebuild switch not succeeding because an LXD storage pool already exists. It means that volumes are still mounted, and you need to delete them because it is necessary to rebuild LXD from scratch.

Start by regenerating your hardware-configuration.nix by doing nixos-generate-config.

In this file, the path to your mounted volumes is mentioned, and you need to dismount and remove them in the following way:

• sudo umount -v /var/lib/lxd/storage-pools/ to dismount the storage pool.
• sudo rm -r /var/lib/lxd to delete the whole LXD folder.

Rebuilding your system is necessary again.

Conclusion

Congratulations! You are now able to launch virtual machines using LXD on NixOS, and you learned how to use flakes to create an overlay for your nixpkgs.

This method will become obsolete once the pull request is merged, but it stays relevant if you want to integrate features that are not in the master branch.