[s390x] Reboot failed after bootc switch and upgrade

[henrywang]

bootc switch succeed but reboot failed with the following error. Looks the required OSTree root does not change to new one(bootc switch or bootc upgrade one).

ostree-prepare-root.service   
 - OSTree Prepare OS/...  
[    2.222936] ostree-prepare-root[754]: ostree-prepare-root: Couldn't find specified OSTree root '/sysroot//ostree/boot.1/default/b4cc77944674566ca72921ec06ceed86d7d946d9ee55a6966d6388fa4a8a3366/0': No such file or directory 
[ FAILED ] Failed to start         

bootc switch console log: http://lab-02.hosts.prod.upshift.rdu2.redhat.com/beaker/logs/recipes/17210+/17210771/console.log
bootc upgrade console log: https://beaker-archive.prod.engineering.redhat.com/beaker-logs/2024/10/100253/10025395/17231599/console.log

• Same test on ppc64le does not have this issue.

bootc install to-existing-root works:

Writing manifest to image destination
Installing image: docker://quay.io/fedora/fedora-bootc:41
Digest: sha256:c5045a3ee6d2f546348caaaa4f39cfe5a869e4367611a0304e94ca9d26444913
----------------------------
WARNING: This operation will OVERWRITE THE BOOTED HOST ROOT FILESYSTEM and is NOT REVERSIBLE.
Waiting 20s to continue; interrupt (Control-C) to cancel.
----------------------------
Installing to `dos` format partitions is not recommended
Initializing ostree layout
layers already present: 0; layers needed: 65 (673.1 MB)
Fetched layers: 641.92 MiB in 24 seconds (26.58 MiB/s)
Deploying container image...done
Injected: etc/tmpfiles.d/bootc-root-ssh.conf
running zipl to install bootloader on /dev/vda
> zipl --target /target/boot --image /target/boot/ostree/default-b4cc77944674566ca72921ec06ceed86d7d946d9ee55a6966d6388fa4a8a3366/vmlinuz-6.11.2-300.fc41.s390x --ramdisk /target/boot/ostree/default-b4cc77944674566ca72921ec06ceed86d7d946d9ee55a6966d6388fa4a8a3366/initramfs-6.11.2-300.fc41.s390x.img --parameters root=UUID=fa47874c-b599-4c86-be66-8199a9bd7efe rd.lvm.lv=fedora_s390x-kvm-005/root rw boot=UUID=76d436fb-0212-4bfe-bb65-f2084d535681 ostree=/ostree/boot.1/default/b4cc77944674566ca72921ec06ceed86d7d946d9ee55a6966d6388fa4a8a3366/0 --targetbase /dev/vda --targettype SCSI --targetblocksize 512 --targetoffset 2048 --add-files --verbose
Looking for components in '/lib/s390-tools'
Secure boot support: no
Target device information
  Device..........................: fc:00
  Device name.....................: vda *)
  Device driver name..............: virtblk
  Type............................: disk device
  Disk layout.....................: SCSI disk layout *)
  Geometry - start................: 2048 *)
  File system block size..........: 4096
  Physical block size.............: 512 *)
  Device size in physical blocks..: 2097152
  *) Data provided by user.
Building bootmap in '/target/boot' (files will be added to bootmap file)
Adding IPL section
  initial ramdisk...: /target/boot/ostree/default-b4cc77944674566ca72921ec06ceed86d7d946d9ee55a6966d6388fa4a8a3366/initramfs-6.11.2-300.fc41.s390x.img
  kernel image......: /target/boot/ostree/default-b4cc77944674566ca72921ec06ceed86d7d946d9ee55a6966d6388fa4a8a3366/vmlinuz-6.11.2-300.fc41.s390x
  kernel parmline...: 'root=UUID=fa47874c-b599-4c86-be66-8199a9bd7efe rd.lvm.lv=fedora_s390x-kvm-005/root rw boot=UUID=76d436fb-0212-4bfe-bb65-f2084d535681 ostree=/ostree/boot.1/default/b4cc77944674566ca72921ec06ceed86d7d946d9ee55a6966d6388fa4a8a3366/0'
  component address:
    heap area.......: 0x00002000-0x00005fff
    stack area......: 0x0000f000-0x0000ffff
    internal loader.: 0x0000a000-0x0000dfff
    parameters......: 0x00009000-0x000091ff
    kernel image....: 0x00010000-0x00d97fff
    parmline........: 0x00d98000-0x00d981ff
    initial ramdisk.: 0x00db0000-0x0401e1ff
    environment blk.: 0x00da8000-0x00da8fff
zIPL environment block content:
Preparing boot device for LD-IPL: vda.
Detected SCSI PCBIOS disk layout.
Writing SCSI master boot record.
Syncing disks...
Done.
Installation complete!

[cgwalters]

My best guess here is that we're writing to the wrong /boot or so...maybe in this job let's gather the output of lsblk before we reboot?

[henrywang]

Gathering some output (after bootc switch and before reboot):

[root@s390x-kvm-007 roothome]#  lsblk -a
NAME                            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
loop0                             7:0    0  7.7M  1 loop 
vda                             252:0    0  120G  0 disk 
├─vda1                          252:1    0    1G  0 part /boot
└─vda2                          252:2    0  119G  0 part 
  └─fedora_s390x--kvm--007-root 253:0    0   15G  0 lvm  /var
                                                         /sysroot/ostree/deploy/default/var
                                                         /etc
                                                         /sysroot

[root@s390x-kvm-007 roothome]# ll /sysroot/ostree/boot.1/default/b4cc77944674566ca72921ec06ceed86d7d946d9ee55a6966d6388fa4a8a3366/0
lrwxrwxrwx. 1 root root 97 Oct 15 00:43 /sysroot/ostree/boot.1/default/b4cc77944674566ca72921ec06ceed86d7d946d9ee55a6966d6388fa4a8a3366/0 -> ../../../deploy/default/deploy/0a1fc72c63c5352ae85807beefb6b957118e6bf8cc138ad2b14eea8caf965e91.0

[root@s390x-kvm-007 roothome]# ll /sysroot/ostree
total 0
lrwxrwxrwx. 1 root root   8 Oct 15 00:43 boot.1 -> boot.1.1
drwxr-xr-x. 3 root root  21 Oct 15 00:43 boot.1.1
drwxr-xr-x. 3 root root  21 Oct 15 00:42 bootc
drwxr-xr-x. 3 root root  21 Oct 15 00:48 deploy
drwxr-xr-x. 7 root root 102 Oct 15 00:48 repo

[root@s390x-kvm-007 roothome]# bootc status
apiVersion: org.containers.bootc/v1
kind: BootcHost
metadata:
  name: host
spec:
  image:
    image: quay.io/xiaofwan/fedora-bootc-os_replace:beaker
    transport: registry
  bootOrder: default
status:
  staged:
    image:
      image:
        image: quay.io/xiaofwan/fedora-bootc-os_replace:beaker
        transport: registry
      version: 41.20241011.0
      timestamp: null
      imageDigest: sha256:9459701dc95b071d7a4521cd9570787ca6ac1d2e449015cc3e52cc99a826d4c7
    cachedUpdate: null
    incompatible: false
    pinned: false
    store: ostreeContainer
    ostree:
      checksum: 34894ca857a5b2b8417d05615b14c5df678d2638185a9a5e0dda6ac36a3cdde4
      deploySerial: 0
  booted:
    image:
      image:
        image: quay.io/fedora/fedora-bootc:41
        transport: registry
      version: 41.20241011.0
      timestamp: null
      imageDigest: sha256:178476528c5795bfcb5773c7b79c16120cc43ab3c9e532fdcca107e8e3852c42
    cachedUpdate: null
    incompatible: false
    pinned: false
    store: ostreeContainer
    ostree:
      checksum: 0a1fc72c63c5352ae85807beefb6b957118e6bf8cc138ad2b14eea8caf965e91
      deploySerial: 0
  rollback: null
  rollbackQueued: false
  type: bootcHost

[root@s390x-kvm-007 roothome]# ll /sysroot/ostree/deploy/default/deploy/
total 16
drwxr-xr-x. 13 root root 4096 Oct 15 00:43 0a1fc72c63c5352ae85807beefb6b957118e6bf8cc138ad2b14eea8caf965e91.0
-rw-r--r--.  1 root root   93 Oct 15 00:43 0a1fc72c63c5352ae85807beefb6b957118e6bf8cc138ad2b14eea8caf965e91.0.origin
drwxr-xr-x. 13 root root 4096 Oct 15 00:48 34894ca857a5b2b8417d05615b14c5df678d2638185a9a5e0dda6ac36a3cdde4.0
-rw-r--r--.  1 root root  110 Oct 15 00:48 34894ca857a5b2b8417d05615b14c5df678d2638185a9a5e0dda6ac36a3cdde4.0.origin

[root@s390x-kvm-007 roothome]# cat /proc/cmdline 
root=UUID=6df90228-dab9-4a47-9f53-376fe5942fef rd.lvm.lv=fedora_s390x-kvm-007/root rw boot=UUID=c5e9d89e-16c9-4a76-ba4b-67d14726aa81 ostree=/ostree/boot.1/default/b4cc77944674566ca72921ec06ceed86d7d946d9ee55a6966d6388fa4a8a3366/0

[henrywang]

bootc upgrade has same issue. I updated the title and issue description.

[henrywang]

centos-bootc:stream9/10 and rhel-bootc:9.5/10.0-beta have the same issue