nixos/binfmt: Add option to use static emulators when available #334859

jcaesar · 2024-08-15T13:26:30Z

Motivation

This PR makes running binaries via boot.binfmt.emulatedSystems in chroots "just work", at the cost of adding/setting a single option.

The goal is to change

$ uname -m
x86_64
$ docker run --rm -ti --platform linux/arm alpine uname -m
exec /bin/uname: no such file or directory

into

$ docker run --rm -ti --platform linux/arm alpine uname -m
armv7l

The other prominent use-case is to nixos-enter/chroot into a broken linux install mounted to /mnt of an other machine for debugging, where that linux install is for a different platform (e.g. a raspi).

Description of changes

Add an option boot.binfmt.preferStaticEmulators, which changes boot.binfmt.registrations.*.interpreter to an executable from pkgsStatic where possible.
Add a test for the option
Some misc related cleanup

Things I'd like advice on

Note how the error message in the motivational examples is abysmal? /bin/uname exists in both examples. What's not being found is /run/binfmt/armv7l-linux.

Instead of making this an option, would it be conceivable to make this always-on?

Notes

This is the third attempt to land this in nixpkgs. Related PRs/issues:

Original Issue: Cannot chroot to a different CPU architecture using boot.binfmt.emulatedSystems #160300
First PR: nixos/binfmt: Add support for using statically-linked QEMU #160802
(abandoned)
Second PR: nixos/binfmt: Add support for using statically-linked QEMU (continuation of #160802) #300070
(abandoned)
Third PR: qemu: add user space emulator only variant #314998
(only made pkgsStatic.qemu.override { userOnly = true; } work)

Things done

Add a 👍 reaction to pull requests you find important.

lib/systems/default.nix

jcaesar · 2024-08-17T13:46:16Z

I was hoping to get away without having to modify qemu again, but alas, pkgsStatic.qemu-user hasn't been merged for a week but already got broken [Edit:] and fixed again. I added a commit that minifies dependencies so hopefully that will happen less frequently. ~~The qemu-user and qemu-utils binaries (which had dependencies removed) are bit-for-bit equal before and after the change, so it shouldn't have broken anything.~~ [Edit:] Aren't, but see comments below on how to verify that it's still fine.

Result of nixpkgs-review pr 334859 run on x86_64-linux 1

5 packages marked as broken and skipped:

cot
cot.dist
ocamlPackages.ocaml-freestanding
python312Packages.cot
python312Packages.cot.dist

1 package blacklisted:

nixos-install-tools

8 packages failed to build:

lima-bin
nemu
qemu_full
qemu_full.debug
qemu_full.ga
quickemu
quickgui
virtualboxKvm

86 packages built:

OVMFFull
OVMFFull.fd
alpine-make-vm-image
cloud-init
cloud-init.dist
cloud-utils
cloud-utils.guest
colima
diffoscope
diffoscope.dist
diffoscope.man
gnome.gnome-boxes
guestfs-tools
libguestfs
libguestfs-with-appliance
lima
linuxKernel.packages.linux_4_19.virtualbox
linuxKernel.packages.linux_4_19_hardened.virtualbox
linuxKernel.packages.linux_5_10.virtualbox
linuxKernel.packages.linux_5_10_hardened.virtualbox
linuxKernel.packages.linux_5_15.virtualbox
linuxKernel.packages.linux_5_15_hardened.virtualbox
linuxKernel.packages.linux_5_4.virtualbox
linuxKernel.packages.linux_5_4_hardened.virtualbox
linuxKernel.packages.linux_6_1.virtualbox
linuxKernel.packages.linux_6_10.virtualbox
linuxKernel.packages.linux_6_1_hardened.virtualbox
linuxKernel.packages.linux_6_6.virtualbox
linuxKernel.packages.linux_hardened.virtualbox (linuxKernel.packages.linux_6_6_hardened.virtualbox)
linuxKernel.packages.linux_latest_libre.virtualbox
linuxKernel.packages.linux_libre.virtualbox
linuxKernel.packages.linux_lqx.virtualbox
linuxKernel.packages.linux_xanmod.virtualbox
linuxKernel.packages.linux_xanmod_latest.virtualbox (linuxKernel.packages.linux_xanmod_stable.virtualbox)
linuxKernel.packages.linux_zen.virtualbox
lxd-lts
multipass
nixpkgs-manual
open-watcom-bin
open-watcom-bin-unwrapped
out-of-tree
python311Packages.cot
python311Packages.cot.dist
python311Packages.guestfs
python311Packages.guestfs.dist
python312Packages.guestfs
python312Packages.guestfs.dist
qemu
qemu-user
qemu-user.debug
qemu-utils
qemu-utils.debug
qemu.debug
qemu.ga
qemu_kvm
qemu_kvm.debug
qemu_kvm.ga
qemu_test
qemu_test.debug
qemu_test.ga
qemu_xen (qemu_xen_4_19)
qemu_xen.debug (qemu_xen_4_19.debug)
qemu_xen.ga (qemu_xen_4_19.ga)
qemu_xen_4_16
qemu_xen_4_16.debug
qemu_xen_4_16.ga
qemu_xen_4_17
qemu_xen_4_17.debug
qemu_xen_4_17.ga
qemu_xen_4_18
qemu_xen_4_18.debug
qemu_xen_4_18.ga
qtemu
solo5
solo5.debug
vagrant
virtualbox
virtualbox.modsrc
virtualboxHardened
virtualboxHardened.modsrc
virtualboxHeadless
virtualboxHeadless.modsrc
virtualboxWithExtpack
virtualboxWithExtpack.modsrc
vmctl
zpool-auto-expand-partitions

nixos/modules/system/boot/binfmt.nix

SuperSandro2000 · 2024-08-28T12:07:06Z

pkgs/applications/virtualization/qemu/default.nix

+  buildInputs = [ glib zlib ]
+    ++ lib.optionals (!minimal) [ dtc pixman vde2 lzo snappy libtasn1 gnutls nettle libslirp ]
+    ++ lib.optionals (!userOnly) [ curl ]


I am not really sure how to verify this but disabling features we don't need is probably a good idea.

What I did:

Verify manually that for !minimal && !userOnly (i.e. all the normal qemus like qemu, qemu-full, qemu_kvm…), only the order of inputs changes (which is hopefully irrelevant).

qemu-user working as intended is covered by a test added here

For qemu-utils:

nix build github:NixOS/nixpkgs/{c46d43c625f161d3bd22052a5f8f9df703d81be8,f69411bcfb0dd3565a19667fdf509bbadb38f008}#qemu-utils

The sizes of the output binaries are all the same: ls -l result*/bin/* | cut -d\ -f5- | sort, so it's unlikely that some larger functionality went missing

Looking e.g. through nix run nixpkgs#diffoscope result*/bin/qemu-storage-daemon shows some probably irrelevant differences in the length of some dummy strings (Probably due to error: derivation '…-qemu-utils-9.0.2.drv' may not be deterministic: output '…-qemu-utils-9.0.2' differs -.-)

alyssais · 2024-09-06T10:47:36Z

lib and pkgs changes look fine to me.

SuperSandro2000

otherwise LGTM and I would say we merge this

nixos/modules/system/boot/binfmt.nix

…sStatic

The fixBinary flag will be enabled if a static emulator is in use.

jcaesar · 2024-10-01T06:57:52Z

@SuperSandro2000 Can I do something to make this easier to merge? Maybe split off the changes to pkgs/ into a different PR? (They're not strictly necessary, but should help to avoid future build failures.) Or is this simply ready to hit the button?

Rebased due to merge conflicts. nemu and quickgui fail as before.

Result of nixpkgs-review run on x86_64-linux 1

8 packages marked as broken and skipped:

cot
cot.dist
linuxKernel.packages.linux_5_4_hardened.virtualbox
ocamlPackages.ocaml-freestanding
python311Packages.subunit2sql
python311Packages.subunit2sql.dist
python312Packages.cot
python312Packages.cot.dist

1 package blacklisted:

nixos-install-tools

4 packages failed to build:

nemu
quickgui
quickgui.debug
quickgui.pubcache

157 packages built:

OVMFFull
OVMFFull.fd
alpine-make-vm-image
barbicanclient (python311Packages.python-barbicanclient)
barbicanclient.dist (python311Packages.python-barbicanclient.dist)
cloud-init
cloud-init.dist
cloud-utils
cloud-utils.guest
colima
diffoscope
diffoscope.dist
diffoscope.man
glanceclient (python311Packages.python-glanceclient)
glanceclient.dist (python311Packages.python-glanceclient.dist)
gnome-boxes
guestfs-tools
heatclient (python311Packages.python-heatclient)
heatclient.dist (python311Packages.python-heatclient.dist)
ironicclient (python311Packages.python-ironicclient)
ironicclient.dist (python311Packages.python-ironicclient.dist)
kata-runtime
libguestfs
libguestfs-with-appliance
lima
lima-bin
linuxKernel.packages.linux_5_10.virtualbox
linuxKernel.packages.linux_5_10_hardened.virtualbox
linuxKernel.packages.linux_5_15.virtualbox
linuxKernel.packages.linux_5_15_hardened.virtualbox
linuxKernel.packages.linux_5_4.virtualbox
linuxKernel.packages.linux_6_1.virtualbox
linuxKernel.packages.linux_6_10.virtualbox
linuxKernel.packages.linux_6_11.virtualbox
linuxKernel.packages.linux_6_1_hardened.virtualbox
linuxKernel.packages.linux_6_6.virtualbox
linuxKernel.packages.linux_hardened.virtualbox (linuxKernel.packages.linux_6_6_hardened.virtualbox)
linuxKernel.packages.linux_latest_libre.virtualbox
linuxKernel.packages.linux_libre.virtualbox
linuxKernel.packages.linux_lqx.virtualbox
linuxKernel.packages.linux_xanmod.virtualbox
linuxKernel.packages.linux_xanmod_latest.virtualbox (linuxKernel.packages.linux_xanmod_stable.virtualbox)
linuxKernel.packages.linux_zen.virtualbox
lxd-lts
magnumclient (python311Packages.python-magnumclient)
magnumclient.dist (python311Packages.python-magnumclient.dist)
manilaclient (python311Packages.python-manilaclient)
manilaclient.dist (python311Packages.python-manilaclient.dist)
mistralclient (python311Packages.python-mistralclient)
mistralclient.dist (python311Packages.python-mistralclient.dist)
multipass
nixpkgs-manual
open-watcom-bin
open-watcom-bin-unwrapped
openstackclient (python311Packages.python-openstackclient)
openstackclient-full
openstackclient-full.dist
openstackclient.dist (python311Packages.python-openstackclient.dist)
out-of-tree
python311Packages.cot
python311Packages.cot.dist
python311Packages.guestfs
python311Packages.guestfs.dist
python311Packages.keystoneauth1
python311Packages.keystoneauth1.dist
python311Packages.openstacksdk
python311Packages.openstacksdk.dist
python311Packages.openstacksdk.man
python311Packages.os-client-config
python311Packages.os-client-config.dist
python311Packages.osc-lib
python311Packages.osc-lib.dist
python311Packages.osc-placement
python311Packages.osc-placement.dist
python311Packages.oslo-concurrency
python311Packages.oslo-concurrency.dist
python311Packages.oslo-db
python311Packages.oslo-db.dist
python311Packages.oslo-log
python311Packages.oslo-log.dist
python311Packages.oslo-serialization
python311Packages.oslo-serialization.dist
python311Packages.oslo-utils
python311Packages.oslo-utils.dist
python311Packages.osprofiler
python311Packages.osprofiler.dist
python311Packages.python-aodhclient
python311Packages.python-aodhclient.dist
python311Packages.python-cinderclient
python311Packages.python-cinderclient.dist
python311Packages.python-designateclient
python311Packages.python-designateclient.dist
python311Packages.python-keystoneclient
python311Packages.python-keystoneclient.dist
python311Packages.python-neutronclient
python311Packages.python-neutronclient.dist
python311Packages.python-novaclient
python311Packages.python-novaclient.dist
python311Packages.python-octaviaclient
python311Packages.python-octaviaclient.dist
swiftclient (python311Packages.python-swiftclient)
swiftclient.dist (python311Packages.python-swiftclient.dist)
troveclient (python311Packages.python-troveclient)
troveclient.dist (python311Packages.python-troveclient.dist)
watcherclient (python311Packages.python-watcherclient)
watcherclient.dist (python311Packages.python-watcherclient.dist)
python311Packages.python-zaqarclient
python311Packages.python-zaqarclient.dist
zunclient (python311Packages.python-zunclient)
zunclient.dist (python311Packages.python-zunclient.dist)
python311Packages.swift
python311Packages.swift.dist
python311Packages.tempest
python311Packages.tempest.dist
python312Packages.guestfs
python312Packages.guestfs.dist
qemu
qemu-user
qemu-user.debug
qemu-utils
qemu-utils.debug
qemu.debug
qemu.ga
qemu_full
qemu_full.debug
qemu_full.ga
qemu_kvm
qemu_kvm.debug
qemu_kvm.ga
qemu_test
qemu_test.debug
qemu_test.ga
qemu_xen
qemu_xen.debug
qemu_xen.ga
qemu_xen_4_17
qemu_xen_4_17.debug
qemu_xen_4_17.ga
qemu_xen_4_18
qemu_xen_4_18.debug
qemu_xen_4_18.ga
qtemu
quickemu
solo5
solo5.debug
vagrant
virtualbox
virtualbox.modsrc
virtualboxHardened
virtualboxHardened.modsrc
virtualboxHeadless
virtualboxHeadless.modsrc
virtualboxKvm
virtualboxWithExtpack
virtualboxWithExtpack.modsrc
vmctl
zpool-auto-expand-partitions

SuperSandro2000 · 2024-10-01T23:24:34Z

Looks all good to me. Just give me a ping when you fixed the test and as long as the changes done there are looking good, I would just merge this.

jcaesar · 2024-10-02T04:28:59Z

The test failure on linux seems to be flaky, I couldn't reproduce it in 5 tries, a no-change force push made it go away. (darwin times out as always.) I'd guess it's unrelated to this PR.

nixos-discourse · 2024-10-31T20:06:40Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/docker-ignoring-platform-when-run-in-nixos/21120/18

Luflosi · 2024-11-03T09:01:07Z

4658a06 has a downside: the closure size of QEMU is now bigger, since the package now contains 34 emulators instead of just one. To measure this, I added qemu-user-orig-aarch64 = qemu.override { smartcardSupport = false; spiceSupport = false; openGLSupport = false; virglSupport = false; vncSupport = false; gtkSupport = false; sdlSupport = false; pulseSupport = false; smbdSupport = false; seccompSupport = false; hostCpuTargets = [ "aarch64-linux-user" ]; }; and qemu-user-aarch64 = qemu-user.override { hostCpuTargets = [ "aarch64-linux-user" ]; }; to pkgs/top-level/all-packages.nix, built the two new packages and qemu-user and looked at the result with nix-tree ./result. The original qemu-user that was used before 4658a06 (built with a current Nixpkgs version) has a NAR size of 48.55 MiB and a total closure size of 135.71 MiB, while qemu-user has a NAR size of 112.63 MiB and a total closure size of 163.54 MiB. The size increase was less than I expected. But if we take qemu-user and let it build only for one architecture, we can make it much smaller than even the original qemu-user version, coming in with a NAR size of only 6.51 MiB and a total closure size of 57.42 MiB.

I'm not sure how to do it better though. If we go back to overriding qemu in lib/systems/default.nix, users will again have to compile QEMU from source, which I don't want. If we add a new package for every single architecture to pkgs/top-level/all-packages.nix, so Hydra will build them, I fear that this would become a maintenance burden as new architectures need to be added manually.

jcaesar requested review from alyssais and Ericson2314 as code owners August 15, 2024 13:26

github-actions bot added 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: module (update) This PR changes an existing module in `nixos/` 6.topic: lib The Nixpkgs function library labels Aug 15, 2024

jcaesar marked this pull request as draft August 15, 2024 13:27

jcaesar force-pushed the pr-14 branch 5 times, most recently from d0777e3 to d80ca52 Compare August 16, 2024 02:13

ofborg bot requested a review from edolstra August 16, 2024 03:18

ofborg bot added 10.rebuild-darwin: 11-100 10.rebuild-linux: 11-100 labels Aug 16, 2024

alyssais reviewed Aug 16, 2024

View reviewed changes

lib/systems/default.nix Outdated Show resolved Hide resolved

jcaesar force-pushed the pr-14 branch from d80ca52 to b8cd7b7 Compare August 17, 2024 11:46

ofborg bot requested a review from alyssais August 17, 2024 12:20

jcaesar marked this pull request as ready for review August 17, 2024 13:46

jcaesar requested a review from infinisil as a code owner August 17, 2024 13:46

jcaesar mentioned this pull request Aug 26, 2024

nixos/binfmt: Add support for using statically-linked QEMU (continuation of #160802) #300070

Closed

14 tasks

SuperSandro2000 approved these changes Aug 28, 2024

View reviewed changes

nixos/modules/system/boot/binfmt.nix Outdated Show resolved Hide resolved

SuperSandro2000 reviewed Aug 28, 2024

View reviewed changes

jcaesar force-pushed the pr-14 branch 2 times, most recently from 8d1bf60 to f69411b Compare August 28, 2024 23:40

ofborg bot added 10.rebuild-linux: 101-500 and removed 10.rebuild-linux: 11-100 labels Aug 29, 2024

SuperSandro2000 approved these changes Aug 29, 2024

View reviewed changes

wegank added the 12.approvals: 1 This PR was reviewed and approved by one reputable person label Aug 31, 2024

Cryolitia approved these changes Sep 6, 2024

View reviewed changes

wegank added 12.approvals: 2 This PR was reviewed and approved by two reputable people and removed 12.approvals: 1 This PR was reviewed and approved by one reputable person labels Sep 7, 2024

SuperSandro2000 approved these changes Sep 12, 2024

View reviewed changes

nixos/modules/system/boot/binfmt.nix Outdated Show resolved Hide resolved

jcaesar force-pushed the pr-14 branch from f69411b to fe4d094 Compare September 12, 2024 12:02

ofborg bot requested a review from SigmaSquadron September 12, 2024 17:47

SigmaSquadron removed the 12.approvals: 2 This PR was reviewed and approved by two reputable people label Sep 12, 2024

wegank added the 2.status: merge conflict This PR has merge conflicts with the target branch label Sep 27, 2024

jcaesar and others added 3 commits October 1, 2024 15:05

qemu: reduce userOnly dependencies so its less likely to break in pkg…

1f82156

…sStatic

lib/systems: use qemu-user package instead of custom definition

4658a06

nixos/binfmt: Add option to use static emulators when available

b8c1ef9

The fixBinary flag will be enabled if a static emulator is in use.

jcaesar force-pushed the pr-14 branch from fe4d094 to e6ce356 Compare October 1, 2024 06:34

ofborg bot removed the 2.status: merge conflict This PR has merge conflicts with the target branch label Oct 1, 2024

ofborg bot requested review from CertainLach, hehongbo and digitalrane October 1, 2024 07:17

nixos/tests/systemd-binfmt: Add chroot test

fc0cc3f

jcaesar force-pushed the pr-14 branch from e6ce356 to fc0cc3f Compare October 1, 2024 23:48

SuperSandro2000 approved these changes Oct 2, 2024

View reviewed changes

SuperSandro2000 merged commit d44717c into NixOS:master Oct 2, 2024
28 checks passed

Cryolitia mentioned this pull request Oct 6, 2024

Cannot chroot to a different CPU architecture using boot.binfmt.emulatedSystems #160300

Closed

Atemu mentioned this pull request Oct 6, 2024

nixos-enter not working from x86_64 nixos host to aarch64 nixos target #346504

Closed

Luflosi mentioned this pull request Nov 3, 2024

boot.binfmt.emulatedSystems requires compiling QEMU #221056

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nixos/binfmt: Add option to use static emulators when available #334859

nixos/binfmt: Add option to use static emulators when available #334859

jcaesar commented Aug 15, 2024 •

edited

Loading

jcaesar commented Aug 17, 2024 •

edited

Loading

SuperSandro2000 Aug 28, 2024

jcaesar Aug 29, 2024

alyssais commented Sep 6, 2024

SuperSandro2000 left a comment

jcaesar commented Oct 1, 2024 •

edited

Loading

SuperSandro2000 commented Oct 1, 2024

jcaesar commented Oct 2, 2024 •

edited

Loading

nixos-discourse commented Oct 31, 2024

Luflosi commented Nov 3, 2024

nixos/binfmt: Add option to use static emulators when available #334859

nixos/binfmt: Add option to use static emulators when available #334859

Conversation

jcaesar commented Aug 15, 2024 • edited Loading

Motivation

Description of changes

Things I'd like advice on

Notes

Things done

jcaesar commented Aug 17, 2024 • edited Loading

SuperSandro2000 Aug 28, 2024

Choose a reason for hiding this comment

jcaesar Aug 29, 2024

Choose a reason for hiding this comment

alyssais commented Sep 6, 2024

SuperSandro2000 left a comment

Choose a reason for hiding this comment

jcaesar commented Oct 1, 2024 • edited Loading

SuperSandro2000 commented Oct 1, 2024

jcaesar commented Oct 2, 2024 • edited Loading

nixos-discourse commented Oct 31, 2024

Luflosi commented Nov 3, 2024

jcaesar commented Aug 15, 2024 •

edited

Loading

jcaesar commented Aug 17, 2024 •

edited

Loading

jcaesar commented Oct 1, 2024 •

edited

Loading

jcaesar commented Oct 2, 2024 •

edited

Loading