Replace ubuntu-jammy-rpi with ubuntu-noble-rpi

[IsaacJT]

Hi @schaefi,

I'm jumping back into some Kiwi topics after a long time away ;)

ubuntu-jammy-rpi no longer builds due to the repo being moved under Deprecated on download.opensuse.org. This updates the OBS URL to bring it in line with the change made in 7d41510.

[IsaacJT]

Unfortunately the build is still broken (with box build) after updating the URL:

E: flAbsPath on /var/lib/dpkg/status.kiwi failed - realpath (2: No such file or directory)
E: Could not open file  - open (2: No such file or directory)
E: Problem opening
E: The package lists or status file could not be parsed or opened.

I'm not sure what it's looking for or expecting here.

[schaefi]

Hi @IsaacJT

I'm jumping back into some Kiwi topics after a long time away ;)

That's great, we missed you :)

The proposed change looks good and it can be merged. Due to python issues I could not build the latest version for Ubuntu Jammy anymore. The oldest entry-point I could built it for was xUbuntu_23.04, that's why you need to jump to the deprecated space for anything older and I hope it doesn't cause trouble

E: flAbsPath on /var/lib/dpkg/status.kiwi failed - realpath (2: No such file or directory)

However, I could reproduce the build problem you found and its also related to an older version of apt. This part I could "fix" in kiwi. See a PR here: OSInside/kiwi#2639

As background information; We dropped the use of debootstrap and use a bootstrap system that only uses apt. If you want to see the details on this change: OSInside/kiwi#2602

So with the kiwi fix I don't see the apt issue anymore but the installation fails on the post script of systemd in Jammy

chfn: PAM: Critical error - immediate abort
adduser: `/usr/bin/chfn -f systemd Network Management systemd-network' returned error code 1. Exiting.
dpkg: error processing package systemd (--configure):
 installed systemd package post-installation script subprocess returned error exit status 1

If I repeat the call chfn tells me that there is no "Network" user. I'm not sure if this is a problem of how kiwi created the bootstrap or rather an issue of the packaging. I appreciate your help at this point.

Thanks much

[schaefi]

@IsaacJT thinking more about this and also what Neal said on the kiwi PR, we actually don't support Ubuntu <= 24.04 with the new kiwi version 10.x.x. The fix to support older versions is also not nice and as you are using boxbuild you should be able to build the image with the following command

kiwi-ng system boxbuild --box universal --kiwi-version 9.25.22 -- --description ...  --target-dir ...

I have tested a jammy build and that worked for me

Let me know if this also works for you

Thanks

[IsaacJT]

Hi Marcus, I originally ran into this issue while trying to demonstrate how Kiwi does cross architecture builds to a coworker using this description. If the newest versions of Kiwi are no longer supporting 22.04, maybe the "correct" solution for this is to uplift this description to support 24.04 instead? It would be great to have a working foreign architecture reference description that people can just be pointed to and use. I'm happy to give it a go and modify the PR. Cheers, Isaac

…

-------- Original Message -------- On 5/09/24 22:21, Marcus Schäfer wrote: @IsaacJT thinking more about this and also what Neal said on the kiwi PR, we actually don't support Ubuntu <= 24.04 with the new kiwi version 10.x.x. The fix to support older versions is also not nice and as you are using boxbuild you should be able to build the image with the following command kiwi-ng system boxbuild --box universal --kiwi-version 9.25.22 -- --description ... --target-dir ... I have tested a jammy build and that worked for me Let me know if this also works for you Thanks — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[schaefi]

maybe the "correct" solution for this is to uplift this description to support 24.04 instead?

I would agree to this plan

[schaefi]

I updated the lunar and noble descriptions for x86_64 and tested the boxbuild. So far so good. Please take a look at the changes I did to the package list. When we moved from debootstrap to an apt based process for the bootstrap some packages needs to be mentioned explicitly now.

[IsaacJT]

Thanks for the pointers. I'm not sure exactly which bootstrap package you mean (I added libpam-runtime and language-pack-en from the other recipes) but I'm running into the following error when trying to cross build it in a boxbuild:

[ ERROR   ]: 08:46:05 | KiwiDebianBootstrapError: KiwiCommandError: apt-get: stderr: E: Package 'apt:amd64' has no installation candidate
, stdout: Reading package lists...
Building dependency tree...
Package apt:amd64 is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

It's trying to install the amd64 package even though it's running in an arm64 box.

Command line is:

kiwi-ng --debug system boxbuild --aarch64 --box ubuntu --cpu cortex-a53 --machine virt --box-memory=8G
 --box-smp-cpus=8 --no-accel -- --description . --target-dir  ~/tmp/arm64

[schaefi]

Thanks for the pointers. I'm not sure exactly which bootstrap package you mean (I added libpam-runtime and language-pack-en from the other recipes)

yep I meant those

It's trying to install the amd64 package even though it's running in an arm64 box.

yeah and I stumbled over the exact same issue. The only solution I found was dropping the security repo, then it resolves clean. See: commit f91f44d

I find this apt behavior completely strange. As you can see from my patch for the x86_64 image the issue was vice versa. So it was requesting apt:arm64 on an amd64 box. It seems there is something in that security repo that makes app to think it has to request a multiarch package

I only see this behavior on noble, not on lunar

[schaefi]

Hmm, but I see for the arm build the ports.ubuntu.com repo is used, so the same "fix" I did on the x86 side cannot applied here. I have no idea how to debug this because all you get from apt is

Package apt:amd64 is not available, but is referred to by another package.

yeah which one ?

[IsaacJT]

Yeah unfortunately it doesn't help :( I'll push the current state if you want to have a look

[schaefi]

This is very strange, I could reproduce it once but now it works... without having changed anything.

Can you try to run with --box-debug

this keeps the box open with a shell. You can re-run the build by

run_kiwi

[IsaacJT]

It's quite bizarre. Seems to work about 10% of the time...

[schaefi]

yes really strange. I have the following patch

diff --git a/ubuntu/aarch64/ubuntu-jammy-rpi/appliance.kiwi b/ubuntu/aarch64/ubuntu-jammy-rpi/appliance.kiwi
index c7286d7..58a1c50 100644
--- a/ubuntu/aarch64/ubuntu-jammy-rpi/appliance.kiwi
+++ b/ubuntu/aarch64/ubuntu-jammy-rpi/appliance.kiwi
@@ -29,12 +29,14 @@
         <user password="$1$wYJUgpM5$RXMMeASDc035eX.NbYWFl0" home="/home/ubuntu" name="ubuntu" groups="users,kvm" shell="/bin/bash"/>
     </users>
     <repository type="apt-deb" repository_gpgcheck="false" package_gpgcheck="false" imageinclude="true">
-        <source path="obs://Virtualization:Appliances:Builder/xUbuntu_22.04"/>
+        <source path="obs://Virtualization:Appliances:Builder/xUbuntu_23.04"/>
     </repository>
-    <repository type="apt-deb" alias="Jammy" distribution="jammy" components="main multiverse restricted universe" repository_gpgcheck="false">
+    <repository type="apt-deb" alias="Lunar" distribution="lunar" components="main multiverse restricted universe" repository_gpgcheck="false">
         <source path="http://ports.ubuntu.com/ubuntu-ports"/>
     </repository>
     <packages type="image">
+        <package name="libpam-runtime"/>
+        <package name="language-pack-en"/>
         <package name="linux-raspi"/>
         <package name="ubuntu-raspi-settings"/>
         <package name="linux-firmware"/>
@@ -63,13 +65,11 @@
         <package name="mawk"/>
         <package name="openssh-client"/>
         <package name="openssh-server"/>
-        <package name="netcat"/>
         <package name="zstd"/>
         <package name="util-linux"/>
         <package name="less"/>
         <package name="vim"/>
         <package name="python3-kiwi"/>
-        <package name="python3-kiwi-boxed-plugin"/>
         <package name="kiwi-systemdeps"/>
         <package name="jing"/>
         <package name="qemu-kvm"/>
diff --git a/ubuntu/aarch64/ubuntu-jammy-rpi/pre_disk_sync.sh b/ubuntu/aarch64/ubuntu-jammy-rpi/pre_disk_sync.sh
index 770f803..95bee9a 100644
--- a/ubuntu/aarch64/ubuntu-jammy-rpi/pre_disk_sync.sh
+++ b/ubuntu/aarch64/ubuntu-jammy-rpi/pre_disk_sync.sh
@@ -17,7 +17,7 @@ cp -a /usr/lib/firmware/*-raspi/device-tree/broadcom/* /boot/efi/
 #==========================================
 # copy initrd and kernel
 #------------------------------------------
-cp /boot/initrd.img-*-raspi /boot/efi/initrd.img
+cp /boot/initrd-*-raspi /boot/efi/initrd.img
 cp /boot/vmlinuz-*-raspi /boot/efi/vmlinuz
 
 #==========================================

and this builds in 1 out of 10 times

[schaefi]

So here is a reproducer

create /tmp/foo.config

Dir "/";
Dir::State "/var/cache/kiwi/apt-get/";
Dir::State::status "/var/lib/dpkg/status.kiwi";
Dir::Cache "/var/cache/kiwi/apt-get/";
Dir::Etc   "/var/cache/kiwi/apt-get/";
APT
{
    Get
    {
        AllowDowngrades "true";
        AllowRemoveEssential "true";
        AllowChangeHeldPackages "true";
        AllowUnauthenticated "true";
    }
};
DPkg::Options {
    "--path-exclude=/usr/share/man/*";
    "--path-exclude=/usr/share/doc/*";
    "--path-include=/usr/share/doc/*/copyright";
};

now call

apt-get -q -c /tmp/foo.config -y --no-install-recommends install "-oDebug::pkgDPkgPm=1" "-oDPkg::Pre-Install-Pkgs::=cat >/tmp/foo" "?essential" "?exact-name(usr-is-merged)" base-passwd apt

Sometimes this works, sometimes not

what the heck

[schaefi]

@IsaacJT I think I found it. If I remove the OBS created kiwi repo

Virtualization:Appliances:Builder/xUbuntu_23.04

then it works reliable. I have no idea why though. Do you see anything suspicious here: https://download.opensuse.org/repositories/Virtualization:/Appliances:/Builder/xUbuntu_23.04/

I have pushed an ubuntu-noble-rpi/ description without using the kiwi repo, this one builds reliable for me

[schaefi]

My guess is that apt has an issue when it should resolve on multiarch flat repos, but it doesn't look like a bug in kiwi

[IsaacJT]

I wonder if it would make sense to add Architectures: arm64 to the *.sources files that Kiwi generates? That way it will never load anything to do with amd64 from the apt mirrors.

[IsaacJT]

Annoyingly it's now working every time I try it.... Maybe it was some intermittent problem on some mirror?

[schaefi]

I wonder if it would make sense to add Architectures: arm64 to the *.sources files that Kiwi generates? That way it will never load anything to do with amd64 from the apt mirrors.

yes this is a very good idea. I coded it in a way that you can specify the architectures via the kiwi definition file.

• Add support for architectures in deb source file kiwi#2640

It would be great if you can give it a review. Thanks much

[schaefi]

Annoyingly it's now working every time I try it.... Maybe it was some intermittent problem on some mirror?

🤣 really annyoing if it works every time

I think this PR can be merged, it looks good to me, or do you plan any further changes ? If we agree to the change in kiwi, we can pin the architecture for the Virt:App repo in a new PR once a new kiwi version with the feature is out

[IsaacJT]

I unfortunately don't have a RPi to test the output right now. I'll see if I can dig one up or get it working in QEMU (I think there's an RPi emulation mode?)

[schaefi]

I unfortunately don't have a RPi to test the output right now. I'll see if I can dig one up or get it working in QEMU (I think there's an RPi emulation mode?)

yes qemu has a --machine emulation for several pi's. I have some PI's here on my desk. If you want I can test your result build.

btw: I never build something for production with the slow cross arch emulation mode of a boxbuild. For aarch64 I usually build on PI 's 4/5 with 8G of RAM and using boxbuild together with KVM. KVM is supported on the PI's and works really well. In the past I also had access to a NXP lx2160 which was really fast.

The cross building model in boxbuild works, but I think it's really more for debugging as it's super slow. Using a cross-arch container with binfmts is faster in this case

[schaefi]

@IsaacJT kiwi >= v10.1.7 has the architectures flag, the boxes will auto rebuild in a few minutes.

I'm going to merge this PR now

[IsaacJT]

Sorry only just got around to testing it again now. It doesn't build in its current form (KiwiCommandError: chroot: stderr: Keymap us is not installed. and a couple of other issues), so I'll open another PR :)

[IsaacJT]

Interestingly I'm now getting

[ ERROR   ]: 16:11:58 | KiwiDebianBootstrapError: KiwiCommandError: apt-get: stderr: E: Package 'apt:amd64' has no installation candidate
, stdout: Reading package lists...
Building dependency tree...
Reading state information...
Package apt:amd64 is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

again...