Binfmt and QEMU are not configured for x86

[HugoFlorentino]

I am trying to use the tool from Ubuntu 20.04, but it fails. I prepared the script with this command, which gave no error:

create-alpine-disk-image --debug --script-filename create.sh --virtual vmware --arch x86 --cpu-vendor intel --enable-watchdog --os-device-media disk --os-device-type scsi --ipv4only --keymap 'us us-intl' --optimise --username cloud --password cloud --auth-control both --root-part-size 128 --add-packages 'linux-virt mesa-dri-gallium open-vm-tools-deploypkg open-vm-tools-guestinfo open-vm-tools-openrc open-vm-tools-static open-vm-tools-timesync open-vm-tools-vmbackup busybox-extras binutils wget ca-certificates'

However, when I try to run it (as root), following error message appears:

Binfmt and QEMU are not configured for x86

I think I have all dependencies installed, but I couldn't see a list of required packages, I installed them basicaly through trial and error.

Edit: I found out what the problem was. My system had x86_64 as architecture, but my intention was to build a strict x86 alpine image. Apparently, in order to build an image for a different architecture, some extra configuration must be done which the script doesn't currently support.

[dermotbradley]

The issue with creating a x86 image appears to be due the way that Ubuntu (and Debian) set up binfmt/qemu-user - it appears what (on a x86_64 system) they set up binfmt entries for armv7 and aarch64 they do not set up an entry for x86 (unlike binfmt on Alpine which does). So it is not exactly an issue with the script, more of a difference in behaviour on Ubuntu/Debian regarding x86 emulation.

I suspect this is done as Ubuntu and Debian tend to have multilib installed, permitting them to run x86 binaries directly on a x86_64 system. I will have to think how to resolve this for Ubuntu and Debian in the script.

I am somewhat confused by the options you are passing to create-alpine-disk-image:

"--root-part-size 128" - from a brief test this is insufficient size for a Alpine rootfs to install. How did you determine the 128MB/MiB figure?

Why are you adding the "linux-virt" package? the appropriate kernel package is automatically selected by the script.

Likewise open-vm-tools packages are automatically selected when you pass the "--virtual vmware" option. Also I only just noticed that Alpine does not actually provide open-vm-tools packages for x86 - I have changed the script accordingly.

Why are you adding the ca-certificates and wget packages? ca-certificates and wget should be installed automatically by the script.

Why are you installing "mesa-dri-gallium"? That is graphics related, not something typically installed on a server.

I noticed you have made some changes in your fork of my repo. In particular I am confused why you have added Alpine 3.12.x to the script. Firstly Alpine 3.12 is out-of-support since the end of May. Secondly there is no cloud-init package for Alpine 3.12 (it was added to Alpine 3.13.0) and, as my script relies on cloud-init, I specifically did not add 3.12.x support for that reason.

In also don't understand why you added a parted check to the create-alpine-disk-image script when it makes no use of parted at all - the secondary script created by create-alpine-disk-image does have a dependancy but a check for that it already present here https://github.com/HugoFlorentino/alpine-image/blob/main/lib/common-functions#L755

[HugoFlorentino]

Eventually I did find a way to get past that error:

if [ ! -f /proc/sys/fs/binfmt_misc/qemu-i386 ]; then echo ':qemu-i386:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x03\x00:\xff\xff\xff\xff\xff\xfe\xfe\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-i386-static:' | sudo tee /proc/sys/fs/binfmt_misc/register > /dev/null; fi

Apparently to keep this registry persistent, it has to be added to /etc/binfmt.d/qemu.conf

Some changes I made in my fork were rather blind attempts, because I could not find comments explaining the purpose of some options. I am learning as I go. You are correct, many packages were redundant. I added v3.12 because I couldn't find some drivers which I thought were needed to be manually declared.

--root-part-size ended up needing around 300 MB because later I added more packages

I tried the tool in a system without parted installed and it failed without outputting useful errors, that's why I added it. Better to fail early if it's not installed, IMHO. Same thing goes for shellcheck. The script failed without any useful error.

[dermotbradley]

Eventually I did find a way to get past that error:

if [ ! -f /proc/sys/fs/binfmt_misc/qemu-i386 ]; then echo ':qemu-i386:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x03\x00:\xff\xff\xff\xff\xff\xfe\xfe\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-i386-static:' | sudo tee /proc/sys/fs/binfmt_misc/register > /dev/null; fi

Apparently to keep this registry persistent, it has to be added to /etc/binfmt.d/qemu.conf

Right, this is working around the issue I mentioned earlier that the Ubuntu and Debian binfmt-support/qemu-user-static packages do not set this up for x86.

I will look at modifying my script to set this up on Debian/Ubuntu machines.

I added v3.12 because I couldn't find some drivers which I thought were needed to be manually declared.

I don't see how selecting Alpine Release 3.12 would make any difference regarding kernel drivers, the enabled virtio & VMware-specific drivers have not changed between kernels in Alpine releases for quite some time.

Were you perhaps trying to use emulated physical devices on VMware (rather than virtualised virtio/vmware devices)? The Alpine linux-virt does not support the full set of physical drivers as the linux-lts package as it is designed for virtual machines (rather than virtual machines "pretending" to be physical machines).

I tried the tool in a system without parted installed and it failed without outputting useful errors, that's why I added it. Better to fail early if it's not installed, IMHO.

The create-alpine-disk-image script does not use parted at all and therefore it does not check that parted is installed.

Any script, such as create.sh, generated by create-alpine-disk-image will make use of parted but such scripts do already include a check that the parted package is installed. If the parted package check failed then the create.sh script should produce a "The following packages need to be installed:" message and stop.

So I'm confused how you would see a problem that was resolved by simply installing the parted package, unless you are testing on a non-Alpine/Debian/Ubuntu system - those are the only distros the script currently supports.

Same thing goes for shellcheck. The script failed without any useful error.

I'm using shellcheck as a debugging aid and so didn't add it to dependencies. I've changed the script to now make it a conditional dependency, I'll do a new MR in the next few hours to push this change.

[HugoFlorentino]

I tried the tool in a system without parted installed and it failed without outputting useful errors, that's why I added it. Better to fail early if it's not installed, IMHO.

The create-alpine-disk-image script does not use parted at all and therefore it does not check that parted is installed.

Any script, such as create.sh, generated by create-alpine-disk-image will make use of parted but such scripts do already include a check that the parted package is installed. If the parted package check failed then the create.sh script should produce a "The following packages need to be installed:" message and stop.

So I'm confused how you would see a problem that was resolved by simply installing the parted package, unless you are testing on a non-Alpine/Debian/Ubuntu system - those are the only distros the script currently supports.

This is right now on Ubuntu 20.04 without parted installed:

$ sudo ./create.sh
Creating sparse disk image of 265MiB
Partitioning disk image for BIOS
$

It simply breaks and finishes, no error message at all.

[dermotbradley]

I just merged some changes including the Debian/Ubuntu setup for binfmt x86.

[dermotbradley]

It simply breaks and finishes, no error message at all.

It you create the script with the "--debug" flag then once you run it there will be a logfile in the same directory with more information about the script run - that may highlight what is going on.

[HugoFlorentino]

It simply breaks and finishes, no error message at all.

It you create the script with the "--debug" flag then once you run it there will be a logfile in the same directory with more information about the script run - that may highlight what is going on.

Right:

[2022-06-16 00:43:15] Creating sparse disk image of 265MiB
[2022-06-16 00:43:15] Partitioning disk image for BIOS
[2022-06-16 00:43:15]   Creating msdos disk label
./create.sh: 350: parted: not found

Perhaps displaying the error would make the script more intuitive. Apparently this is what you intended with the message "The following packages need to be installed:" but it's simply not shown, either with or without --debug

[dermotbradley]

[2022-06-16 00:43:15] Creating msdos disk label
./create.sh: 350: parted: not found

Hmm, that part of create.sh is after it calls the check_for_required_packages function which should give an error if the parted package is not installed.

The script doesn't check if the parted binary exists as it already has checked that the parted package is installed. I know I tested this on Debian not too long ago - Ubuntu generally functions 99% the same as Debian.

I'll need to set up a Ubuntu VM to test further.

[HugoFlorentino]

[2022-06-16 00:43:15] Creating msdos disk label
./create.sh: 350: parted: not found

Hmm, that part of create.sh is after it calls the check_for_required_packages function which should give an error if the parted package is not installed.

Actually I don't see that this function checks presence of packages at all in Debian/Ubuntu

There is a dpkg-query in common-functions but it just checks for presence of "binfmt-support qemu-user-static"

[dermotbradley]

The are 2 calls to dpkg-query - one in check_for_required_packages, the other in check_binfmt_packages.

[HugoFlorentino]

Sorry, I missed that. Funny it's not working though.

[dermotbradley]

Indeed. Perhaps you could try pasting that code into a separate script or on the command-line to check what is going on?

[HugoFlorentino]

I tried querying all at once and also each at a time using a loop:

#! /bin/sh
_package_list="coreutils dosfstools mount parted wget util-linux e2fsprogs shellcheck"
# shellcheck disable=SC2086
if ! dpkg-query -W -f='\${Status}\n' "${_package_list}" > /dev/null 2>&1; then
  echo "Packages '${_package_list}' required, but not all of them installed."
fi
for package in $(echo "${_package_list}"); do
# shellcheck disable=SC2086
  if ! dpkg-query -W -f='\${Status}\n' "${package}" > /dev/null 2>&1; then
    echo "Package ${package} is required, but not installed."
  fi
done
exit 0

The first variant did not report things correctly (I have all packages installed right now):

Packages 'coreutils dosfstools mount parted wget util-linux e2fsprogs shellcheck' required, but not all of them installed.

[dermotbradley]

Yeah I was "lazy" in not using a loop and checking each individually as the existing code certainly works fine for me on Debian.

Using a loop is the obvious solution.

Out of interest what does this show with the same $_package_list value?

dpkg-query -W -f='\${Status}\n' "${_package_list}" 2>&1
echo $?

[HugoFlorentino]

It exits with a return value of 1. Go figure.

Plus, I uninstalled parted and the loop didn't work as expected either.

[HugoFlorentino]

I cloned project with latest changes, prepared the script with debug enabled and this is the output, when trying to build an image for x86:

[2022-06-23 00:11:49] Creating sparse disk image of 339MiB
[2022-06-23 00:11:49] Partitioning disk image for BIOS
[2022-06-23 00:11:49]   Creating msdos disk label
[2022-06-23 00:11:49]   Creating 337MiB Root partition
[2022-06-23 00:11:49]   Setting partition boot flag on
[2022-06-23 00:11:49]   Resultant partition layout:
BYT;
/home/cloud/create-alpine-disk-image/alpine-images/alpine-edge-x86-virtual-vmware.img:339MiB:file:512:512:msdos::;
1:2.00MiB:339MiB:337MiB:::boot;
[2022-06-23 00:11:49] Setting up loop device for disk image
[2022-06-23 00:11:49] Formatting and mounting filesystems
[2022-06-23 00:11:49]   Formatting Ext4 root filesystem on partition
[2022-06-23 00:11:49]   Mounting root filesystem onto ./chroot
[2022-06-23 00:11:50]   Size of root filesystem:
Filesystem     1M-blocks  Used Available Use% Mounted on
/dev/loop0p1        300M    1M      276M   1% /home/cloud/create-alpine-disk-image/chroot
[2022-06-23 00:11:50]   Filesystems' UUIDs
NAME      FSTYPE LABEL UUID FSAVAIL FSUSE% MOUNTPOINT
loop0
`-loop0p1                      276M     0% /home/cloud/create-alpine-disk-image/chroot
[2022-06-23 00:11:50]   Blkid output:
NAME      FSTYPE LABEL       UUID                                 FSSIZE FSAVAIL FSUSED FSUSE% MOUNTPOINT                                  PARTLABEL
loop0
└─loop0p1 ext4   alpine-root 29afb518-115f-4bf7-bb82-803b6c9e0620 299.9M    276M   360K     0% /home/cloud/create-alpine-disk-image/chroot
[2022-06-23 00:11:50] Downloading statically built APK tool for x86_64 arch
[2022-06-23 00:11:56] Copying system's /etc/resolv.conf into chroot filesystem
[2022-06-23 00:11:56] Creating /etc/apk/repositories file inside chroot
[2022-06-23 00:11:56] Bootloader packages to be installed are: grub grub-bios
[2022-06-23 00:11:56] Install base Alpine (plus bootloader packages) for x86 arch inside chroot
fetch https://dl-cdn.alpinelinux.org/alpine/edge/main/x86/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/edge/community/x86/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/edge/testing/x86/APKINDEX.tar.gz
(1/36) Installing alpine-baselayout-data (3.2.0-r22)
(2/36) Installing musl (1.2.3-r0)
(3/36) Installing busybox (1.35.0-r17)
Executing busybox-1.35.0-r17.post-install
ERROR: busybox-1.35.0-r17.post-install: script exited with error 127
(4/36) Installing alpine-baselayout (3.2.0-r22)
Executing alpine-baselayout-3.2.0-r22.pre-install
ERROR: alpine-baselayout-3.2.0-r22.pre-install: script exited with error 127
Executing alpine-baselayout-3.2.0-r22.post-install
ERROR: alpine-baselayout-3.2.0-r22.post-install: script exited with error 127
(5/36) Installing ifupdown-ng (0.12.1-r0)
(6/36) Installing openrc (0.44.10-r7)
Executing openrc-0.44.10-r7.post-install
ERROR: openrc-0.44.10-r7.post-install: script exited with error 127
(7/36) Installing alpine-conf (3.14.2-r0)
(8/36) Installing ca-certificates-bundle (20211220-r0)
(9/36) Installing libcrypto1.1 (1.1.1o-r0)
(10/36) Installing libssl1.1 (1.1.1o-r0)
(11/36) Installing ssl_client (1.35.0-r17)
(12/36) Installing zlib (1.2.12-r1)
(13/36) Installing apk-tools (2.12.9-r5)
(14/36) Installing busybox-suid (1.35.0-r17)
(15/36) Installing mdev-conf (4.1-r1)
(16/36) Installing busybox-initscripts (4.1-r1)
Executing busybox-initscripts-4.1-r1.post-install
ERROR: busybox-initscripts-4.1-r1.post-install: script exited with error 127
(17/36) Installing scanelf (1.3.4-r0)
(18/36) Installing musl-utils (1.2.3-r0)
(19/36) Installing libc-utils (0.7.2-r3)
(20/36) Installing alpine-keys (2.4-r1)
(21/36) Installing alpine-base (3.16.0-r0)
(22/36) Installing lddtree (1.26-r2)
(23/36) Installing xz-libs (5.2.5-r1)
(24/36) Installing zstd-libs (1.5.2-r3)
(25/36) Installing kmod (29-r2)
(26/36) Installing kmod-openrc (29-r2)
(27/36) Installing libblkid (2.38-r2)
(28/36) Installing argon2-libs (20190702-r1)
(29/36) Installing device-mapper-libs (2.02.187-r2)
(30/36) Installing json-c (0.16-r0)
(8/36) Installing ca-certificates-bundle (20211220-r0)
(9/36) Installing libcrypto1.1 (1.1.1o-r0)
(10/36) Installing libssl1.1 (1.1.1o-r0)
(11/36) Installing ssl_client (1.35.0-r17)
(12/36) Installing zlib (1.2.12-r1)
(13/36) Installing apk-tools (2.12.9-r5)
(14/36) Installing busybox-suid (1.35.0-r17)
(15/36) Installing mdev-conf (4.1-r1)
(16/36) Installing busybox-initscripts (4.1-r1)
Executing busybox-initscripts-4.1-r1.post-install
ERROR: busybox-initscripts-4.1-r1.post-install: script exited with error 127
(17/36) Installing scanelf (1.3.4-r0)
(18/36) Installing musl-utils (1.2.3-r0)
(19/36) Installing libc-utils (0.7.2-r3)
(20/36) Installing alpine-keys (2.4-r1)
(21/36) Installing alpine-base (3.16.0-r0)
(22/36) Installing lddtree (1.26-r2)
(23/36) Installing xz-libs (5.2.5-r1)
(24/36) Installing zstd-libs (1.5.2-r3)
(25/36) Installing kmod (29-r2)
(26/36) Installing kmod-openrc (29-r2)
(27/36) Installing libblkid (2.38-r2)
(28/36) Installing argon2-libs (20190702-r1)
(29/36) Installing device-mapper-libs (2.02.187-r2)
(30/36) Installing json-c (0.16-r0)
(31/36) Installing libuuid (2.38-r2)
(32/36) Installing cryptsetup-libs (2.4.3-r0)
(33/36) Installing kmod-libs (29-r2)
(34/36) Installing mkinitfs (3.6.1-r3)
Executing mkinitfs-3.6.1-r3.post-install
ERROR: mkinitfs-3.6.1-r3.post-install: script exited with error 127
(35/36) Installing grub (2.06-r5)
(36/36) Installing grub-bios (2.06-r5)
Executing busybox-1.35.0-r17.trigger
ERROR: busybox-1.35.0-r17.trigger: script exited with error 127
5 errors; 28 MiB in 36 packages

Script aborts at this point, without installing in the image the list of additional packages I declared.

[dermotbradley]

(3/36) Installing busybox (1.35.0-r17)
Executing busybox-1.35.0-r17.post-install
ERROR: busybox-1.35.0-r17.post-install: script exited with error 127

This appears to be a fundamental issue with (packages) installing - the busybox-1.35.0-r17.post-install script just runs busybox.

All I can assume is that the binfmt/qemu-user workaround for x86 on debian/ubuntu is not working as expected.

Do you see the same errors when creating a x86_64 disk image?

[HugoFlorentino]

Do you see the same errors when creating a x86_64 disk image?

No, with that architecture the script finishes building image normally.

Even while building image for x86, there is a log line which attracted my attention:

Downloading statically built APK tool for x86_64 arch

Shouldn't it download a statically built APK tool for x86?

[dermotbradley]

No, with that architecture the script finishes building image normally.

Right, which seems to confirm it is a binfmt/qumu-user-x86 issue on Ubuntu x86_86.

Shouldn't it download a statically built APK tool for x86?

No. The static APK is run on the machine before any binfmt/qemu-user activity (which happens inside the chroot).

[HugoFlorentino]

But x86 code is compatible with x86_64 hardware anyway, so it should work

[dermotbradley]

But x86 code is compatible with x86_64 hardware anyway, so it should work

Debian and Ubuntu support multiarch, Alpine does not.

Most binaries are not 100% statically linked and so require some libraries (the C library at the very least) in order to run. Therefore you can't run an Alpine x86 binary on a x86_64 machine if you do not also have the x86 C library (Musl in Alpine's case) available.

Also Alpine uses Musl as its C library (and so Alpine binaries are linked to that), Debian and Ubuntu typically use Glibc.

[HugoFlorentino]

Most binaries are not 100% statically linked and so require some libraries (the C library at the very least) in order to run. Therefore you can't run an Alpine x86 binary on a x86_64 machine if you do not also have the x86 C library (Musl in Alpine's case) available.

Also Alpine uses Musl as its C library (and so Alpine binaries are linked to that), Debian and Ubuntu typically use Glibc.

I thought the whole point of a static build was to not depend on libraries.

[HugoFlorentino]

I tried something else, after using the script with --arch x86 :

$  sudo grep -i 'x86_64' ~/create-alpine-disk-image/create.sh
          x86_64 )
            _required_packages="$_required_packages qemu-x86_64" ;;
      x86_64 )
        _binfmt_file="/proc/sys/fs/binfmt_misc/qemu-x86_64" ;;

Cloud that be the reason it fails?

[dermotbradley]

I thought the whole point of a static build was to not depend on libraries.

If a binary is completely statically linked, yes. However static linking means that at least 1 library has been statically linked, not all.

Alpine binaries always dynamically link at least the C library (musl). If you run "ldd" on something like Alpine's /bin/busybox you will see this.

[dermotbradley]

Cloud that be the reason it fails?

Rather than grepping did you actually look at those sections of the create.sh script?

No, those lines are not relevant.