buildah from is not working for emulated architectures #5322
Replies: 18 comments 3 replies
-
|
3 Files: |
Beta Was this translation helpful? Give feedback.
-
|
Do you have the qemu-user-static package installed? |
Beta Was this translation helpful? Give feedback.
-
|
Hi Daniel, thanks for looking into this. |
Beta Was this translation helpful? Give feedback.
-
|
If the user static is installed then it should work. Buildah does not do anything to effect this. When the host kernel sees an executable compiled for a different arch, the kernel is supposed to use the emulation from qemu-user-static to execute the code. |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
Thanks Nalin for pointing this out. The problem occurs already in the command buildah from fedora. |
Beta Was this translation helpful? Give feedback.
-
|
I also tried to pull the correct arch by adding the tag, same result. |
Beta Was this translation helpful? Give feedback.
-
|
I ran the error case with a higher loglevel: |
Beta Was this translation helpful? Give feedback.
-
|
@rhatdan can you please take a look again? Do you have an idea why buildah from does not work on emulated platforms even with the original quay buildah image? Am I doing something wrong? |
Beta Was this translation helpful? Give feedback.
-
|
Just to be sure this is not a silly user error I did install a fresh KVM with fedora 39, installed podman and tried to use a Dockerfile with buildah bud, and it fails the same way. So it seems to me all usage of podman/buildah is broken when using it in an emulated container. |
Beta Was this translation helpful? Give feedback.
-
|
Could you give me the actual commands you are executing. The last comment above is Buildah within a Podman container? What happens when you execute something like: |
Beta Was this translation helpful? Give feedback.
-
podman run --arch s390x --privileged --rm -it quay.io/buildah/stable:v1.33.2-s390x buildah from fedora
This works. Also, the result is a usable s390x container. @rhatdan could you please take a look and give some advice how to continue here? |
Beta Was this translation helpful? Give feedback.
-
|
Anybody who can shed some light on this problem? |
Beta Was this translation helpful? Give feedback.
-
|
How about doing: |
Beta Was this translation helpful? Give feedback.
-
|
This is starting an x86 container so that works. In the use case I presented I think that is quite a normal straight forward usage. On an x86 we spin off an emulated container with --arch s390x or armhf... in order to compile some binaries and then pack them in a container. The first container started with podman is the build container also used to build in OCP on Z. Thats why we use the same in the developer compile environment on an x86 machine. So it seems the straight forward use case does not work with podman/buildah, do you agree Dan? |
Beta Was this translation helpful? Give feedback.
-
I'd recommend using the
Emulation is very, very good, but it isn't perfect. Some of that is bugs, yes, but some of that is limitations of how emulation is configured, and some of it comes down to how the emulator interacts with the kernel. |
Beta Was this translation helpful? Give feedback.
-
|
In a perfect world we would prefer to build all containers on native platforms and then assemble them into a multi-arch image. This is the goal of Emulating as little as possible is the best way forward. |
Beta Was this translation helpful? Give feedback.
-
|
should this work? podman run --privileged --rm --platform linux/s390x -it quay.io/buildah/stable:v1.33.2 buildah from fedoraCurrently this fails for me in the way described above. I believe this had worked once but I can not reproduce it anymore. |
Beta Was this translation helpful? Give feedback.
-
|
It might work, but you are running a lot of code in emmulation there. if you do this as. Then only the buildah container processes will run in emulation mode. |
Beta Was this translation helpful? Give feedback.
-
|
This appears to be an aspect of user-space emulation that interferes with the reexec mechanism we use to have subprocesses perform tasks that would adversely affect global state if they were done in-process. reexec sets argv[0] when it execs the subprocess to communicate to it what it should do instead of the normal buildah main() logic, and at startup, buildah checks argv[0] for one of the handful of recognized special values that indicates that it should do something else for its parent process and then exit. When starting a process using an emulator, the kernel passes the binary as the emulator's first argument rather than the argv[0] value we set, so that the emulator can find the binary. The kernel docs mention a "P" flag, which could be used when registering an emulator, which causes the kernel to pass argv[0] to the interpreter as a second argument, presumably with any other flags that were passed on the command line following it. The interpreter has to be expecting that extra argument, though, and I don't think qemu-user does. Except for reworking how the reexec package works, which depending on how it's done would end up requiring big changes to projects that use it, I don't see that there's a change we could make in this repository that would make the combination work. |
Beta Was this translation helpful? Give feedback.
-
I'm not familiar to QEMU, but I'm hitting this problem, so I dig a bit deeper, to see if I could find something of value. If Also, it seems that |
Beta Was this translation helpful? Give feedback.
-
Description
When using buildah from on an x86_64 host with --arch s390x for example, the run command fails. This happens with all emulated architectures.
Steps to reproduce the issue:
Run the following in a container on an x86_64 host using --arch s390x
1.work_container=$(buildah from fedora)
2.buildah run "${work_container}" /bin/bash -c "echo hello"
Describe the results you received:
Describe the results you expected:
hello
(this is produced by the echo hello)
Output of
rpm -q buildahorapt list buildah:Using the official image from quay
Output of
buildah version:Output of
podman versionif reporting apodman buildissue:Output of
cat /etc/*release:Output of
uname -a:Output of
cat /etc/containers/storage.conf:I am also attaching some scripts and a Dockerfile to easily reproduce this problem.
The host system I used was Fedora 39. The following is a list of qemu and podman rpms installed.
qemu-common-8.1.3-1.fc39.x86_64
qemu-audio-jack-8.1.3-1.fc39.x86_64
qemu-audio-pipewire-8.1.3-1.fc39.x86_64
qemu-device-display-vhost-user-gpu-8.1.3-1.fc39.x86_64
qemu-ui-opengl-8.1.3-1.fc39.x86_64
qemu-ui-spice-core-8.1.3-1.fc39.x86_64
qemu-char-spice-8.1.3-1.fc39.x86_64
qemu-ui-spice-app-8.1.3-1.fc39.x86_64
qemu-audio-spice-8.1.3-1.fc39.x86_64
qemu-device-display-qxl-8.1.3-1.fc39.x86_64
qemu-ui-egl-headless-8.1.3-1.fc39.x86_64
qemu-ui-sdl-8.1.3-1.fc39.x86_64
qemu-audio-sdl-8.1.3-1.fc39.x86_64
qemu-block-curl-8.1.3-1.fc39.x86_64
qemu-audio-alsa-8.1.3-1.fc39.x86_64
qemu-audio-dbus-8.1.3-1.fc39.x86_64
qemu-audio-oss-8.1.3-1.fc39.x86_64
qemu-audio-pa-8.1.3-1.fc39.x86_64
qemu-block-blkio-8.1.3-1.fc39.x86_64
qemu-block-dmg-8.1.3-1.fc39.x86_64
qemu-block-gluster-8.1.3-1.fc39.x86_64
qemu-block-iscsi-8.1.3-1.fc39.x86_64
qemu-block-nfs-8.1.3-1.fc39.x86_64
qemu-block-rbd-8.1.3-1.fc39.x86_64
qemu-char-baum-8.1.3-1.fc39.x86_64
qemu-device-display-virtio-gpu-8.1.3-1.fc39.x86_64
qemu-device-display-virtio-gpu-ccw-8.1.3-1.fc39.x86_64
qemu-device-display-virtio-gpu-gl-8.1.3-1.fc39.x86_64
qemu-device-display-virtio-gpu-pci-8.1.3-1.fc39.x86_64
qemu-device-display-virtio-gpu-pci-gl-8.1.3-1.fc39.x86_64
qemu-device-display-virtio-vga-8.1.3-1.fc39.x86_64
qemu-device-display-virtio-vga-gl-8.1.3-1.fc39.x86_64
qemu-device-usb-host-8.1.3-1.fc39.x86_64
qemu-device-usb-redirect-8.1.3-1.fc39.x86_64
qemu-device-usb-smartcard-8.1.3-1.fc39.x86_64
qemu-ui-curses-8.1.3-1.fc39.x86_64
qemu-pr-helper-8.1.3-1.fc39.x86_64
qemu-ui-gtk-8.1.3-1.fc39.x86_64
qemu-user-static-xtensa-8.1.3-1.fc39.x86_64
qemu-user-static-x86-8.1.3-1.fc39.x86_64
qemu-user-static-sparc-8.1.3-1.fc39.x86_64
qemu-user-static-sh4-8.1.3-1.fc39.x86_64
qemu-user-static-s390x-8.1.3-1.fc39.x86_64
qemu-user-static-riscv-8.1.3-1.fc39.x86_64
qemu-user-static-ppc-8.1.3-1.fc39.x86_64
qemu-user-static-or1k-8.1.3-1.fc39.x86_64
qemu-user-static-nios2-8.1.3-1.fc39.x86_64
qemu-user-static-mips-8.1.3-1.fc39.x86_64
qemu-user-static-microblaze-8.1.3-1.fc39.x86_64
qemu-user-static-m68k-8.1.3-1.fc39.x86_64
qemu-user-static-loongarch64-8.1.3-1.fc39.x86_64
qemu-user-static-hppa-8.1.3-1.fc39.x86_64
qemu-user-static-hexagon-8.1.3-1.fc39.x86_64
qemu-user-static-cris-8.1.3-1.fc39.x86_64
qemu-user-static-arm-8.1.3-1.fc39.x86_64
qemu-user-static-alpha-8.1.3-1.fc39.x86_64
qemu-user-static-aarch64-8.1.3-1.fc39.x86_64
qemu-block-ssh-8.1.3-1.fc39.x86_64
qemu-system-x86-core-8.1.3-1.fc39.x86_64
qemu-system-x86-8.1.3-1.fc39.x86_64
qemu-kvm-8.1.3-1.fc39.x86_64
qemu-user-static-8.1.3-1.fc39.x86_64
qemu-guest-agent-8.1.3-1.fc39.x86_64
qemu-img-8.1.3-1.fc39.x86_64
libvirt-daemon-driver-qemu-9.7.0-2.fc39.x86_64
podman-4.8.3-1.fc39.x86_64
container-selinux-2.227.0-1.fc39.noarch
Beta Was this translation helpful? Give feedback.
All reactions