cross-architecture broken with recent bootc

[ported-pw]

I cannot build any cross-arch (ARM to AMD64) images on my M1 MacBook. It fails on some interesting layer mounting issue:

Steps:

1. Installed Podman Desktop

2. podman machine stop
   podman machine set --rootful
   podman machine start
   

3. The most minimal example I could break is this:

   sudo podman run \
      --rm \
      -it \
      --privileged \
      --pull=newer \
      --security-opt label=type:unconfined_t \
      -v $(pwd)/output:/output \
      --platform linux/amd64 \
      quay.io/centos-bootc/bootc-image-builder:latest \
      --type qcow2 \
      --target-arch amd64 \
      quay.io/fedora/fedora-bootc:40
   

which results in:

[...]
Copying blob 8f11ed82e8bb done   |
[...]
Error: copying system image from manifest list: writing blob: adding layer with blob "sha256:3f4b29abb14575e3968cf042ba63d35deed74aca2d746f96b547e6ca5da7539e"/""/"sha256:b1bc5c0fb77df66e970553b0fac7730ac3440fc85e5c619fc01929a83b926a8e": unpacking failed (error: exit status 125; output: Error: unrecognized command `podman /`

Did you mean this?
       cp
       ps
       rm

Try 'podman --help' for more information
)
2024/09/09 12:33:47 error: cannot build manifest: failed to pull container image: exit status 125, stderr:

Just pulling that image normally with podman pull works fine.

I also originally ran into something very similar when trying to use --local:

sudo podman pull quay.io/fedora/fedora-bootc:40
sudo podman run \
    --rm \
    -it \
    --privileged \
    --pull=newer \
    --security-opt label=type:unconfined_t \
    -v $(pwd)/output:/output \
    -v /var/lib/containers/storage:/var/lib/containers/storage \
    --platform linux/amd64 \
    quay.io/centos-bootc/bootc-image-builder:latest \
    --local \
    --type qcow2 \
    --target-arch amd64 \
    quay.io/fedora/fedora-bootc:40

which results in:

Generating manifest manifest-qcow2.json
2024/09/09 12:42:09 error: cannot build manifest: running quay.io/fedora/fedora-bootc:40 container failed: exit status 126
stderr:
WARNING: image platform (linux/arm64) does not match the expected platform (linux/amd64)
time="2024-09-09T12:42:09Z" level=error msg="Unmounting /var/lib/containers/storage/overlay/19aa28066f9c3a9a39ee20638c390f9890f17fd0bf3760ca004bf5129ad7026e/merged: invalid argument"
Error: mounting storage for container 30b6258182819f6b1cb6616cf505973fa71da50b6511e8055b012d4fb614f5b6: creating overlay mount to 19aa28066f9c3a9a39ee20638c390f9890f17fd0bf3760ca004bf5129ad7026e/merged, mount_data="lowerdir=/var/lib/containers/storage/overlay/l/WG7RWIHG44KOACMBKSDCFNP4UF:/var/lib/containers/storage/overlay/l/5E2NKIQA2XCJ3T4AQKVZP2R3UG:/var/lib/containers/storage/overlay/l/CPULKYB5OGOZNY7PDH6YKTQGCX:/var/lib/containers/storage/overlay/l/SQB6BFAROQNDVC6IX3CUFWSXFS:/var/lib/containers/storage/overlay/l/NBXZ7UEQXEL442FJ7GT5ACECY7:/var/lib/containers/storage/overlay/l/LD5N23GP5BWBQ32J3PSDKGGMSL:/var/lib/containers/storage/overlay/l/LOGZXIKFLKLIUTUX25U6TSBY3V:/var/lib/containers/storage/overlay/l/DGF44NJRKCLXE56QG35BKISWJE:/var/lib/containers/storage/overlay/l/JZNGUSGVYCGJTSYJAS42FYFJFE:/var/lib/containers/storage/overlay/l/62YV6HD2WSN45ENJQTUPNAYQAH:/var/lib/containers/storage/overlay/l/RZGBK6XY3BMFT7UAIONE2HWFDK:/var/lib/containers/storage/overlay/l/CFJIH5DM3AVXZNIMZOPEF44NP6:/var/lib/containers/storage/overlay/l/QLSMDFUFLGNCP7KFMMO4AEBIIX:/var/lib/containers/storage/overlay/l/EUR5I2TKHKZ25ZCZBYRESFKSKI:/var/lib/containers/storage/overlay/l/LV5LC2O23JARNKOWIXI5QLT673:/var/lib/containers/storage/overlay/l/5IBKL3MJ2PTFQLRI2PGEGW7JXZ:/var/lib/containers/storage/overlay/l/YYZAWX4TZVE2GJ4BCUBWMFGFM3:/var/lib/containers/storage/overlay/l/2MIR75TEUU2QC423UZCCTYEJ7P:/var/lib/containers/storage/overlay/l/T75WTLZ3BHVB6P73O6IRRQCPTE:/var/lib/containers/storage/overlay/l/W4CCR23R4FOMHGHFUSQ4EJZ6Y7:/var/lib/containers/storage/overlay/l/KLBKYDRVROGMGWN6QOBSKAJGZE:/var/lib/containers/storage/overlay/l/RWUKGPXV4GABE72P27TJ2BDDZL:/var/lib/containers/storage/overlay/l/W6VXGFGHRI6JYHFNOVHFEEPQ4U:/var/lib/containers/storage/overlay/l/5YEV7IODOM5BTO6AVUNRSX7EHR:/var/lib/containers/storage/overlay/l/XDWEFVS3QEIWIGPT5TT4BAU7PE:/var/lib/containers/storage/overlay/l/RSC3IISHFHHRK5JDRNQAJ6DTG5:/var/lib/containers/storage/overlay/l/YKDO6Q645735TKL2ICPUSP4HM3:/var/lib/containers/storage/overlay/l/BP65ZK6C7MVOLTYDOALUDUBFYY:/var/lib/containers/storage/overlay/l/AIBGFSGZQHGEJS67VL5G5BMK4T:/var/lib/containers/storage/overlay/l/XHFFVJZYQNEZRMKELOKKCQ2AFH:/var/lib/containers/storage/overlay/l/BG7UI7MF4LBFRGQURYIJQC5PDC:/var/lib/containers/storage/overlay/l/DKEODBUCU2KEYWJPD6J4XJQVOS:/var/lib/containers/storage/overlay/l/5N7BWBOCZ7PVLJDQUZB6LB75TS:/var/lib/containers/storage/overlay/l/6Y4VQO33NEIRNZUQPY3CBVEBED:/var/lib/containers/storage/overlay/l/3ZS4MCW77LAYRC6WXPIDSJ2HEU:/var/lib/containers/storage/overlay/l/H6JROLXARBSXEE3CBCJEHQ5ZAL:/var/lib/containers/storage/overlay/l/OX3SNSJM2UQW7MORLJ234DC6BP:/var/lib/containers/storage/overlay/l/YFAOEEOC2VCIRYYKUO6QREOO7H:/var/lib/containers/storage/overlay/l/TOBA7H7IXQYNFBAZQCVVXI3MOS:/var/lib/containers/storage/overlay/l/YT5EBCLSWFUUCHNI7YFTZIG2VJ:/var/lib/containers/storage/overlay/l/KLBD7GSW7RSEOULCAL7LY4NHCU:/var/lib/containers/storage/overlay/l/YSIV6BPFJW4XK2UWP6QVTKFZGO:/var/lib/containers/storage/overlay/l/METUZVYK753ZKNBVTVMXXQG2JQ:/var/lib/containers/storage/overlay/l/U4ZILNEHWLJLBJZYEF2F7DZN6U:/var/lib/containers/storage/overlay/l/4XDP4SSO7EWOGGA7DJ5HN73IGH:/var/lib/containers/storage/overlay/l/OL7MO6HJEPEKT7UVTUB5RWX3CY:/var/lib/containers/storage/overlay/l/3EJCJAT32TX7XGQ26ADQJJDZ7A:/var/lib/containers/storage/overlay/l/5MZCBFMCYSCTF7V5GJ5Y4KSNQC:/var/lib/containers/storage/overlay/l/N4ZG3X6ZZLT6ND3CHWECEYSZUS:/var/lib/containers/storage/overlay/l/3JZI6TJVTPOE2I4YBW64SE6IPI:/var/lib/containers/storage/overlay/l/PD6AGFRORP4WYJ7JLK3MXXQMSL:/var/lib/containers/storage/overlay/l/SO4NGGOJLD4AU3MGZSL5TAZDF5:/var/lib/containers/storage/overlay/l/QDJJUIUWYCVI5BX3LTL6ZAIIWQ:/var/lib/containers/storage/overlay/l/VJULNCEKKJQYNCGQODNUDPFKEV:/var/lib/containers/storage/overlay/l/Q3WWGFSNKVLMTXCTZBBQVHDV7A:/var/lib/containers/storage/overlay/l/TLIBJ2OH5D2NHN27WLZHU7DKDQ:/var/lib/containers/storage/overlay/l/D3YNL72ZOVWIHEEXFB55KKQDN5:/var/lib/containers/storage/overlay/l/BJPJVJRD2QMK4VTWPS4ZEMG3FX:/var/lib/containers/storage/overlay/l/KPS7ETLLPXT7OGSXK4BFZF7ZFU:/var/lib/containers/storage/overlay/l/CBFTKHUR56TDHWRFATJMJ4ZSLK:/var/lib/containers/storage/overlay/l/KBMR3ARDG5XV6Z2UCWXCHQAV7J:/var/lib/containers/storage/overlay/l/G7C2JS2HTYL5KO3XK5462JTFFE:/var/lib/containers/storage/overlay/l/NA5WHIZIE5X5HZBSXTXUERGT5E:/var/lib/containers/storage/overlay/l/FRRRIYN2BTWTZOGY3FCLSBEP5R:/var/lib/containers/storage/overlay/l/5TDPBY7VZYX342JKRNLQVJL3NJ,upperdir=19aa28066f9c3a9a39ee20638c390f9890f17fd0bf3760ca004bf5129ad7026e/diff,workdir=19aa28066f9c3a9a39ee20638c390f9890f17fd0bf3760ca004bf5129ad7026e/work,nodev,metacopy=on,volatile": mountfrom re-exec output: Error: unrecognized command `podman /var/lib/containers/storage/overlay`
Try 'podman --help' for more information
: error: exit status 125

All of this works fine on a Linux AMD64 machine/VM.
Would appreciate some help to make the workflow painless again :)

[cgwalters]

Maybe fallout from containers/bootc@0527ca9 - we're doing a lot more in bootc's install path now? I haven't tested cross arch in a while

[ported-pw]

Just noticed, my second example obviously has an issue as I pulled the wrong image by not specifying --arch.

sudo podman pull --arch amd64 quay.io/fedora/fedora-bootc:40

but the result is still the same:

Generating manifest manifest-qcow2.json
2024/09/09 14:31:40 error: cannot build manifest: running quay.io/fedora/fedora-bootc:40 container failed: exit status 126
stderr:
time="2024-09-09T14:31:40Z" level=error msg="Unmounting /var/lib/containers/storage/overlay/28498a00edd3e0f6be8e497d6ea6b5160cd6b199bd426591353507a633e4765a/merged: invalid argument"
Error: mounting storage for container 8456b0c200e84d9910d8a473b74f48f76cc930f8be4c9f7c14a4780ea387c643: creating overlay mount to 28498a00edd3e0f6be8e497d6ea6b5160cd6b199bd426591353507a633e4765a/merged, mount_data="lowerdir=/var/lib/containers/storage/overlay/l/DBHJ3TOIS4EK5VDIIT66EDJ7O6:/var/lib/containers/storage/overlay/l/6FP6MWRSJCAD3D6K5D4V6AM5NU:/var/lib/containers/storage/overlay/l/MV2AEFEG2DTO2OEU5S6VNIOYDC:/var/lib/containers/storage/overlay/l/KIGRTROGJXDCC237LRIBDWDJ3I:/var/lib/containers/storage/overlay/l/PGZH34LG4ATCHQOZSZQCMZEWLX:/var/lib/containers/storage/overlay/l/P2MGABDPKNOIXLZFGPYE3TYP2C:/var/lib/containers/storage/overlay/l/SN4EZWQLXDK5H7OQ73ZUHDN2LN:/var/lib/containers/storage/overlay/l/3Q6T647UMUCPNHJOIP3OLOTSXI:/var/lib/containers/storage/overlay/l/DGFCSZOIZRTTGRCF3N2OGTIUKX:/var/lib/containers/storage/overlay/l/ZW7U3GCVFBXQ6FMQPFTBNZBL7L:/var/lib/containers/storage/overlay/l/LE3MCF2DYHIYRDFJVPVGD34CGC:/var/lib/containers/storage/overlay/l/W4H3VXE5L3OITEILTKUVDJXZUP:/var/lib/containers/storage/overlay/l/JAX4533QZKFYSZAICJIFE7FHGF:/var/lib/containers/storage/overlay/l/ZJTUIB5K4O7GHW2MRXZ64S4RUO:/var/lib/containers/storage/overlay/l/DKUU4XJYEN2NVTIOFQB7ENOPNU:/var/lib/containers/storage/overlay/l/CGXNU2UK6CL6ELUPYMX5INEPGR:/var/lib/containers/storage/overlay/l/YKXKPBN74OKNMXM5NSID72RYZV:/var/lib/containers/storage/overlay/l/GAAJ6JD5YD6ZB2TXBOLG3XRUZO:/var/lib/containers/storage/overlay/l/EV6TILQS32YQAEHTCMUJVFRPX2:/var/lib/containers/storage/overlay/l/5RACCUIZSRAPTNICFNHWLEJ4MP:/var/lib/containers/storage/overlay/l/2KF42HEV3GZRQGHZMKPDNS7UYT:/var/lib/containers/storage/overlay/l/6OFFOR4TW7MQOKLMYK7YRM4V3O:/var/lib/containers/storage/overlay/l/T22CH5HJM4FUAS2NXTQPQPD3E5:/var/lib/containers/storage/overlay/l/MBSN7U2M6XJVQA3E3FPBQTTXWI:/var/lib/containers/storage/overlay/l/Z5NSFYFDKV627PP4GBVQVP3ZFZ:/var/lib/containers/storage/overlay/l/VBO5ZVU6C5LPP4QWZQ4FGE4I2E:/var/lib/containers/storage/overlay/l/UICGV2FPCT2SIJU6ZPJXKRUJEX:/var/lib/containers/storage/overlay/l/PTWUO3PWPMHDTCQATMFMYPAMC2:/var/lib/containers/storage/overlay/l/LET4MOTZHEA2YEF6JJ66MVA3ZO:/var/lib/containers/storage/overlay/l/BWJA3MSENPM7NQR6EQ64SLBKH2:/var/lib/containers/storage/overlay/l/YK3FYRK7PJM3KTDNU7VRI3K7RZ:/var/lib/containers/storage/overlay/l/TPIRFOQ4MB2LA6X7U2H734KYPP:/var/lib/containers/storage/overlay/l/7TRND6RHHS7XGKFZ5IHHUO23X6:/var/lib/containers/storage/overlay/l/IVBX5GKJ5LBZ2TJ3GXMX7BGOZI:/var/lib/containers/storage/overlay/l/MVQV7AAGPNCFYVLZPOYLUZRCIC:/var/lib/containers/storage/overlay/l/TAJPG443OA6OUPYZE3MPXL2MLW:/var/lib/containers/storage/overlay/l/XLK2UPFFB3L2HDEOE5IHCOAC3D:/var/lib/containers/storage/overlay/l/MSCRM23E4WUT3FGBMAFPV5RRHL:/var/lib/containers/storage/overlay/l/JYGGKS6N3YQSEIALANVAJIMX4F:/var/lib/containers/storage/overlay/l/22AIXX7YMH5W23LMKDBLT4ZMWP:/var/lib/containers/storage/overlay/l/EHQDWMERWFV4M66RIZ3OYPECQD:/var/lib/containers/storage/overlay/l/KR7VKYRL4XHKKRIRRIX34HX57P:/var/lib/containers/storage/overlay/l/THERXJR2RDXTUUQPMHBBMCR76D:/var/lib/containers/storage/overlay/l/JOEYPB2TCZHPGGMXQUV57JOHR6:/var/lib/containers/storage/overlay/l/2NBJHZHSNVZS4DGZYQTRVCXRPB:/var/lib/containers/storage/overlay/l/DSJBKRQCB4PDFPPLMBPTOQUEZE:/var/lib/containers/storage/overlay/l/ZXPSDZWLKQZCKNI26ODPZUVRFN:/var/lib/containers/storage/overlay/l/ISB3GNAAB5P5WFAC6RFLUWGQMR:/var/lib/containers/storage/overlay/l/YTUFMH52KLXEE3FV5FVEP5PUVG:/var/lib/containers/storage/overlay/l/VIKJXVNR7DPRHPCQZXB6P5HY4X:/var/lib/containers/storage/overlay/l/MADKRI7YIBA5DKOJV56YZUYK3F:/var/lib/containers/storage/overlay/l/QL3K4RQZOMTDZEAZU7T4IGA6BS:/var/lib/containers/storage/overlay/l/NKTIGYQZJ2NPSOHUYFFRAXKGDN:/var/lib/containers/storage/overlay/l/ICZTCBX2SU6HKXAC3IMAF6H3LN:/var/lib/containers/storage/overlay/l/BXIEMJX2XLEYRST7Q46SQHC654:/var/lib/containers/storage/overlay/l/VDY5OPWGAVBGERBVMCWV2RGNXZ:/var/lib/containers/storage/overlay/l/ZP5NTB4JJO6HTDO4UWZA54C4VZ:/var/lib/containers/storage/overlay/l/WY2N4OGB6ULM63T5CBASO4BWYP:/var/lib/containers/storage/overlay/l/7GIN2ATCGPQFDCPNE3QIZJWIZZ:/var/lib/containers/storage/overlay/l/BTQH6SDALHKIPS6QWVZUJEX5R5:/var/lib/containers/storage/overlay/l/B3V4K2EN5QBV55THT4ALDUGCQE:/var/lib/containers/storage/overlay/l/5SRR54EVAH6BJL3OYYKJY5DIUE:/var/lib/containers/storage/overlay/l/YNV2O4FSGJRB3WSNGV6OQCNB3M:/var/lib/containers/storage/overlay/l/76SFTAL3I75HMFG4OK6TXVHUOE:/var/lib/containers/storage/overlay/l/MAGBO7PRFFCEUEQAFHWG6DAUCM,upperdir=28498a00edd3e0f6be8e497d6ea6b5160cd6b199bd426591353507a633e4765a/diff,workdir=28498a00edd3e0f6be8e497d6ea6b5160cd6b199bd426591353507a633e4765a/work,nodev,metacopy=on,volatile": mountfrom re-exec output: Error: unrecognized command `podman /var/lib/containers/storage/overlay`
Try 'podman --help' for more information
: error: exit status 125

[mvo5]

Maybe fallout from containers/bootc@0527ca9 - we're doing a lot more in bootc's install path now? I haven't tested cross arch in a while

We had cross arch as part of our bootc-image-builder CI until about a week or so ago, then it started breaking because of an unimplemented openat2 syscall in qemu (which is fixed in https://www.mail-archive.com/[email protected]/msg1064233.html but not accepted upstream yet) so this is currently on hold on our side.

[cgwalters]

Bigger picture it's going to be a bit tricky for us to maintain the qemu-emulated path as it is today; in bootc we really want to use some sophisticated Linux kernel features.

Probably the most fruitful avenue to pursue will be moving some cross-arch logic into bootc itself. It may to start just look something like bootc install ... --target-arch and just overriding the image we fetch. We'd need to work through any other details but I don't think there's a whole lot there.

[mvo5]

FTR, the qemu-user openat2 support is now merged upstream, we "just" need to backport it to the podman-machine image (which is f40 afaik(?)).

[MoralCode]

Looks like:

• qemu-user 9.2 is the version that will contain this feature as far as i can tell (its the one whose feature freeze date was after this got merged).
• fedora 41 currently has version 9.1 packaged in the repos (sudo dnf --releasever=41 --forcearch=aarch64 list qemu-user)
• it seems like there used to be a build of this under f40 at https://packages.fedoraproject.org/pkgs/qemu/qemu-user/

Am gonna see if i can compile this from latest sources and see if itll work

[mvo5]

Looks like:

* qemu-user 9.2 is the version that will contain this feature as far as i can tell (its the one whose [feature freeze date](https://wiki.qemu.org/Planning/9.2) was after this got merged).

* fedora 41 currently has version 9.1 packaged in the repos (`sudo dnf --releasever=41 --forcearch=aarch64 list qemu-user`)

[..]
The fix got backported via https://src.fedoraproject.org/rpms/qemu/pull-request/70 into f41, the backport for f40 would be a bit more involved though as qemu upstream changed a bit more.

[MoralCode]

we "just" need to backport it to the podman-machine image (which is f40 afaik(?)).

I tried modifying the containerfile to include this copr repo, but that didn't seem to change the error message - maybe i did something wrong

my changes are here: https://github.com/MoralCode/bootc-image-builder/tree/f40

[mvo5]

we "just" need to backport it to the podman-machine image (which is f40 afaik(?)).

I tried modifying the containerfile to include this copr repo, but that didn't seem to change the error message - maybe i did something wrong

my changes are here: https://github.com/MoralCode/bootc-image-builder/tree/f40

The qemu-user binaries need to come from the system, so either the host or the machine inside podman machine. The namespace support for the underlying binfmt_misc is relatively new [0] we should probably look into using this, then we could potentially have qemu-user as part of the bib container. Maybe worth filing a separate issue that we should look into supporting this via https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git/commit/?h=vfs.binfmt_misc&id=ecddcab2d1b15fea782889237093bd069979c8c7 as it will benefit fedora/rhel10/centos10

There is also an extra complication when running under the mac, here rosetta2 needs to be disabled as it does not support openat2 (I did not find much documentation around rosetta2 but it seems to be also using syscall emulation (to some extend at least)).

[0] microsoft/WSL#8203