Pareto Security

Being a happy Nixer on a Mac

—

by

Neyts Zupan
in

At Pareto Security, we recently shipped Linux support. Building a Linux app also means writing automated tests for said app. This entails building up a number of Virtual Machines, with different distros, and different (mis-)configurations, to test that checks are running correctly.

Since I like determinism and infrastructure-as-code, my immediate thought was to use NixOS Integration Tests, and sure enough, we were able to quickly whip up integration tests for Debian, Ubuntu, Fedora and NixOS itself.

Since I still like using macOS, I had to go through a bit of research on how to run Nix, NixOS and NixOS Tests on macOS. During ThaigerSprint I collected all my notes and polished them up into this guide. Share with anyone that is starting out with Nix and wants to keep using their Mac!

Nix natively on macOS

First, I install Nix on my macOS. Using the Official Installer works fine, but there is no uninstall support. Instead, the fine folks in the #macos:nixos.org Matrix channel suggested using the Nix installer from Determinate Systems, especially since I planned on using flakes.

When I got stuck, I found the Setting up Nix on macOS post on Nixcademy super helpful!

Controlling my macOS settings with Nix

Imagine having a single file for all your macOS configuration, including the shell environment, installed tooling and libraries, git config, even Dock and Finder settings.

This is actually entirely possible, using nix-darwin! I followed the README, played around a bit and now whenever I get a new MacBook, setting it up is as easy as installing Nix followed by running darwin-rebuild switch --flake ~/.dotfiles#zbook!

You can find my configuration on GitHub, feel free to take parts from it!

Linux Builder

Since macOS is a different architecture (“darwin”) than NixOS, I cannot build NixOS derivations directly on macOS. Luckily, Nix knows how to offload the building of Linux packages to a Linux machine. We could set up a remote NixOS server and tell Nix to use that, but there is an even better way: run a local NixOS Virtual Machine!

Installing and maintaining a NixOS VM might sound tedious, but fear not! nix-darwin actually ships with one, called Linux Builder, I just needed to enable it!

With nix-darwin installed, I added the following to my darwin-configuration.nix to get the Linux Builder:

nix.linux-builder.enable = true;

I rebuilt with $ darwin-rebuild switch and verified that the Linux Builder VM is actually running with $ sudo launchctl list org.nixos.linux-builder.

{
	"StandardOutPath" = "/var/log/darwin-builder.log";
	"LimitLoadToSessionType" = "System";
	"StandardErrorPath" = "/var/log/darwin-builder.log";
	"Label" = "org.nixos.linux-builder";
	"OnDemand" = false;
	"LastExitStatus" = 0;
	"PID" = 5952;
	"Program" = "/bin/sh";
	"ProgramArguments" = (
		"/bin/sh";
		"-c";
		"/bin/wait4path /nix/store && exec /nix/store/rn0agv8m87p4xzrs4jxczg6sq4z7lbnw-linux-builder-start";
	);
};

Finally, I followed performance optimizations recommended in another Nixcademy blog post.

NixOS Tests on macOS

What I was initially after was not building NixOS packages but rather to be able to build and run NixOS Integration Tests. Although macOS is a different architecture (called darwin) than NixOS, running the NixOS Tests directly on macOS is still possible, because the Linux Builder provided by nix-darwin is used to build the Linux VMs that the tests run against.

Now that I had the Linux Builder installed and running, I was able to run an example NixOS test with $ nix -L build github:tfc/nixos-integration-test-example and got test script finished in ...s in my output after a few minutes, yay!

Reminder to self: When running $ nix -L build github:tfc/nixos-integration-test-example the second time, nothing happens. Huh? Well, NixOS Tests are just a Nix derivation, and Nix sees that nothing has changed, the sourcecode is the same, and hence nothing needs to be built, just exit. To force Nix otherwise, I used the --rebuild flag.

But what about when tests fail? I use the Interactive Mode to see what’s going on! First, $ nix -L build github:tfc/nixos-integration-test-example#default.driverInteractive to build the test driver with Interactive Mode enabled, then start it with $ ./result/bin/nixos-test-driver --keep-vm-state.

When the driver starts, I am dropped into a Python shell of the NixOS test driver. Usually, I would type start_all() to start all VMs that the tests define. Since the test example I am using defines SSH access for testing VMs, I am actually able to SSH into the VMs and debug their state: ssh root@localhost -p 2222

I use the --keep-vm-state flag to retain the SSH host keys between runs, to save me accepting them every time I start a new SSH session. See --help if you are interested in more options.

The folks at Nixcademy have written an extensive two-part blog post on NixOS Integration Tests that is well worth a read!

Troubleshooting

Here are a few things I usually try when I get strange errors setting up or using Nix on macOS:

  • Upgrade to the latest nixpkgs. Currently, macOS has second-tier support in Nix ecosystem and as such is sometimes broken for a few days until macOS-specific fixes are pushed after large changes to other parts of the ecosystem. Try both nixpkgs-unstable and a stable release such as 24.11.
  • Add nix.linux-builder.ephemeral.enable = true to darwin-configuration.nix. This aggressively purges the Linux Builder VMs between nix builds, ensuring a clean slate.
  • Sometimes, after reinstalling nix-darwin, I need to sudo su - and then ssh linux-builder to manually accept the SSH host key, so Nix can actually SSH to the Linux Builder VM.
  • If I get -bash: /run/current-system/sw/bin/hello: cannot execute binary file: Exec format error in the test VM, this usually means the VM is trying to run a package built for darwin instead of linux. Is the test using the correct pkgs.?

NixOS in a Virtual Machine

Sometimes, I still need an actual full-fledged NixOS to work on something, for example polishing the UI of Pareto Security menubar widget. I could set up a NixOS server with remote desktop and connect into it, but a better way is to run it locally, with UTM, a light-weight GUI layer around the QEMU machine virtualizer.

Getting started

First, I install UTM and download the 64-bit ARM Minimal ISO image from nixos.org/download.

Start up UTM and Create a New Virtual Machine. Choose Virtualize and Linux. Select the downloaded image as Boot ISO Image and keep pressing Continue to create the VM.

When the VM is up, I am dropped into a shell. Log in as root with sudo su - and then follow the UEFI Partitioning part of the NixOS Manual. Use /dev/vda instead of /dev/sda. When done, follow the Formatting and Installing sections (including the UEFI systems chapter for both).

Finish it off by clearing out the CD/DVD in the VM settings and rebooting into a fresh NixOS virtual machine!

I learned a ton from https://krisztianfekete.org/nixos-on-apple-silicon-with-utm/ and https://adrianhesketh.com/2024/04/20/setting-up-nixos-remote-builder-m1-mac/ blog posts.

Convenience

I don’t want to type all commands manually into the VM window. A better way is to configure the VM for SSH access and then use macOS native Terminal app to connect and run commands in the VM.

Add services.openssh.enable = true; and services.openssh.permitRootLogin = "yes"; to /etc/nixos/configuration.nix, then run $ nixos-rebuild switch.

The last step is to run ifconfig inside the VM to get its IP address, and to SSH into the VM from the Terminal app:

$ ssh [email protected]
([email protected]) Password: 
Last login: Thu Feb 13 06:11:33 2025

[root@nixos:~]#

Success!

NixOS Tests

Before I could run NixOS Tests in the NixOS VM, I needed to enable some Nix features. Add nix.settings.experimental-features = [ "nix-command" "flakes" ]; to /etc/nixos/configuration.nix and run $ nixos-rebuild switch.

The NixOS test example from the [Nix natively on macOS section](TODO) above now works in the UTM VM: $ nix -L build github:tfc/nixos-integration-test-example.

Nested Virtualization

I have noticed that NixOS tests running in UTM are slower than those running natively on macOS. Why is that?

The reason is that NixOS Tests build and start a QEMU VM, then run a python test script against this VM. And UTM is also a QEMU VM! VM inside a VM! Obviously, it would be slower!

Luckily, there is a workaround, called “nested virtualization” which allows the guest VM to pass-through CPU commands through the host VM up to the hardware host (your Mac). It’s fairly easy to enable, but there is a caveat: nested virtualization is only supported on Macs with the M3 CPU or newer.

Also, the UTM VM needs to be recreated. Follow the same process as above, but when creating the UTM image, tick the Use Apple Virtualization checkbox.

The difference in speed is about 10x!

x86 Virtualization

But I had to run NixOS tests for a specific CPU architecture, what then? Specifically, I needed to verify that NixOS tests run well on an Intel CPU, with the --system x86_64-linux flag:

$ nix -L build --system x86_64-linux github:tfc/nixos-integration-test-example --rebuild
error: some outputs of '/nix/store/...-vm-test-run-An-awesome-test..drv' are not valid, so checking is not possible

The UTM VM created above is based on the ARM architecture and as such can’t build packages for Intel. However, we are in luck again, Apple has provided us with another trick, called Rosetta, which enables us to pretend our CPU supports x86 instructions!

With Apple Virtualization VM shut down, in Settings, under Virtualization, tick the Enable Rosetta on Linux checkbox. This should work on any Apple Silicon Mac.

Boot the VM back up, add virtualisation.rosetta.enable = true; to configuration.nix and run nixos-rebuild switch. Building for x86 now works!

$ nix -L build --system x86_64-linux github:tfc/nixos-integration-test-example
vm-test-run-An-awesome-test.> Machine state will be reset. To keep it, pass --keep-vm-state
vm-test-run-An-awesome-test.> start all VLans
vm-test-run-An-awesome-test.> start vlan
vm-test-run-An-awesome-test.> running vlan (pid 9; ctl /build/vde1.ctl)
vm-test-run-An-awesome-test.> (finished: start all VLans, in 0.02 seconds)
vm-test-run-An-awesome-test.> Test will time out and terminate in 3600 seconds
...

Editor Integration

I wanted another bit of convenience: to use macOS VS Code to edit files inside the UTM VM.

Add (fetchTarball "https://github.com/nix-community/nixos-vscode-server/tarball/master") to the imports section of the configuration.nix, add services.vscode-server.enable = true;, then run nixos-rebuild switch to apply the changes.

Finally, install the Remote - SSH VS Code extension, and use the Remote SSH: Connect to Host... command in VS Code to connect to [email protected].

On some UTM VMs I got Could not start dynamically linked executable: ... error in VS Code, which I fixed by adding services.vscode-server.enableFHS = true;, rebuilding and rebooting.

Kudos to everyone who has worked on the Nix on Mac story! Compared to just a few years ago, it’s now an absolute joy to work with Nix on a Mac!

Neyts Zupan

Neyts is the Pareto Security co-founder and Tech Lead.