NixOS within NixOS?

Question

I'm starting to play around with NixOS deployments. To that end, I have a repo with some packages defined, and a configuration.nix for the server.

It seems like I should then be able to test this configuration locally (I'm also running NixOS). I imagine it's a bad idea to change my global configuration.nix to point to the deployment server's configuration.nix (who knows what that will break); but is there a safe and convenient way to "try out" the server locally - i.e. build it and either boot into it or, better, start it as a separate process?

I can see docker being one way, of course; maybe there's nothing else. But I have this vague sense Nix could be capable of doing it alone.

score 14 · Accepted Answer · answered Dec 04 '18 at 18:12

14

There is a fairly standard way of doing this that is built into the default system.

Namely nixos-rebuild build-vm. This will take your current configuration file (by default /etc/nixos/configuration.nix, build it and create a script allowing you to boot the configuration into a virtualmachine.

once the script has finished, it will leave a symlink in the current directory. You can then boot by running ./result/bin/run-$HOSTNAME-vm which will start a boot of your virtualmachine for you to play around with.

TLDR;

nixos-rebuild build-vm
./result/bin/run-$HOSTNAME-vm

answered Dec 04 '18 at 18:12

hamhut1066

406
3
13

1

How costly is that VM? Does it clone the entire store? – 0fnt Nov 20 '19 at 20:50
3

@0fnt: No, not at all, it uses the store of the host (mounted readonly). – Ingo Blechschmidt Apr 13 '20 at 04:26
2

Alternatively I believe `-I nixos-config=./path/to/configuration.nix` can be used to point to config file. – Chris Stryczynski Jun 29 '21 at 14:44

score 5 · Answer 2 · edited Aug 23 '21 at 11:11

As you may already know, system configurations can coexist without any problems in the Nix store. The problem here is running more than one system at once. For this, you need an isolation or virtualization tools like Docker, VirtualBox, etc.

NixOS Containers

NixOS provides an efficient implementation of the container concept, backed by systemd-nspawn instead of an image-based container runtime.

These can be specified declaratively in configuration.nix or imperatively with the nixos-container command if you need more flexibility.

Docker

Docker was not designed to run an entire operating system inside a container, so it may not be the best fit for testing NixOS-based deployments, which expect and provide systemd and some services inside their units of deployment. ~~While you won't get a good NixOS experience with Docker,~~ Nix and Docker are a good fit. UPDATE: Both 'raw' Nix packages and NixOS run in Docker. For example, Arion supports images from plain Nix, NixOS modules and 'normal' Docker images.

NixOps

To deploy NixOS inside NixOS it is best to use a technology that is designed to run a full Linux system inside.

It helps to have a program that manages the integration for you. In the Nix ecosystem, NixOps is the first candidate for this. You can use NixOps with its multiple backends, such as QEMU/KVM, VirtualBox, the (currently experimental) NixOS container backend, or you can use the none backend to deploy to machines that you have created using another tool.

Here's a complete example of using NixOps with QEMU/KVM.

Tests

If the your goal is to run automated integration tests, you can make use of the NixOS VM testing framework. This uses Linux KVM virtualization (expose /dev/kvm in sandbox) to run integrations test on networks of virtual machines, and it runs them as a derivation. It is quite efficient because it does not have to create virtual machine images because it mounts the Nix store in the VM. These tests are "built" like any other derivation, making them easy to run.

Nix store optimization

A unique feature of Nix is that you can often reuse the host Nix store, so being able to mount a host filesystem in the container/vm is a nice feature to have in your solution. If you are creating your own solutions, depending on you needs, you may want to postpone this optimization, because it becomes a bit more involved if you want the container/vm to be able to modify the store. NixOS tests solve this with an overlay file system in the VM. Another approach may be to bind mount the Nix store forward the Nix daemon socket.

This is a great answer, but there is so much to unpack (e.g., "_Tests being derivations makes for a nice development workflow_"). Do you happen to have blog posts or go-to articles on these topics? — toraritte, Nov 14 '20 at 16:29
@toraritte I've edited that statement to be more specific. ( ... -> These tests are "built" like any other derivation, making them easy to run.) I've added NixOS containers (how could I forget??) and I've added links to the docs. — Robert Hensing, Nov 15 '20 at 19:42

score 5 · Answer 3 · edited Nov 14 '20 at 16:34

nixos-rebuild build-vm is the easiest way to do this, however; you could also import the configuration into a NixOS container (see Chapter 47. Container Management in the NixOS manual and the nixos-container command).

This would be done with something like:

containers.mydeploy = {
  privateNetwork = true;
  config = import ../mydeploy-configuration.nix;
};

Note that you would not want to specify the network configuration in mydeploy-configuration.nix if it's static as that could cause conflicts with the network subnet created for the container.