I rebuilt my Fedora bootc Containerfile, rebuilt and did a bootc switch, kernel panic on reboot

[ericcurtin]

initrd is missing which made ostree go down this no initrd code path and attempt to boot with initrd=ostree-prepare-root on the cmdline

FROM quay.io/fedora/fedora-kinoite:42

RUN curl -fsSLO https://ollama.com/install.sh && chmod a+x install.sh && \
      sed -i "s/set -eu/set -eux/g" install.sh && \
      sed -i "s# in /usr/local/bin# in#g" install.sh && \
      ./install.sh
RUN curl -fsSL https://get.docker.com | bash
RUN dnf install -y alacritty black clang cmake codespell distrobox dnf4 \
      fedora-workstation-repositories gcc hyperfine keepassxc libcurl-devel \
      make nvtop python3-tqdm qemu-kvm the_silver_searcher vim python3-flake8 \
      bats httpd-tools ninja meson python3-jinja2 SDL2-devel crun-krun podman \ 
      docker-model-plugin
RUN dnf remove -y nano
RUN curl -fsSL -o /usr/bin/build_llama_and_whisper.sh https://raw.githubusercontent.com/containers/ramalama/6d7effadc2f6cd8bf7b231af965e367e65153029/container-images/scripts/build_llama_and_whisper.sh
RUN chmod a+rx /usr/bin/build_llama_and_whisper.sh && build_llama_and_whisper.sh "rocm"

seems like the addition of RUN curl -fsSL https://get.docker.com | bash broke it.

[cgwalters]

FWIW the docs on initramfs for Fedora https://docs.fedoraproject.org/en-US/bootc/initramfs/

seems like the addition of RUN curl -fsSL https://get.docker.com | bash broke it.

Impressive, needs debugging. If it is regenerating the initramfs it's probably just running dracut which indeed right now will break things unfortunately and needs to be done per above doc.

I bet if you add RUN bootc container lint you'll get some warnings about stuff in /boot that shouldn't be there.

BTW, I strongly recommend heredocs.

and attempt to boot with initrd=ostree-prepare-root on the cmdline

Ah glancing at the code this will happen if you don't have an initramfs at all currently. That's really a bug, we shouldn't conflate a static prepareroot with no initramfs. However ultimately we'll hopefully move to UKIs by default which obviate this.

[ericcurtin]

I keep forgetting about:

RUN bootc container lint

must add

[ericcurtin]

Some warnings:

STEP 8/8: RUN bootc container lint
Lint warning: nonempty-boot: Found non-empty /boot:
  .vmlinuz-6.15.5-200.fc42.x86_64.hmac
  System.map-6.15.5-200.fc42.x86_64
  config-6.15.5-200.fc42.x86_64
  efi
  grub2
  ...and 4 more

Lint warning: sysusers: Found /etc/passwd entry without corresponding systemd sysusers.d:
  ollama
Found /etc/group entry without corresponding systemd sysusers.d:
  docker
  ollama
  qat

Lint warning: var-log: Found non-empty logfiles:
  /var/log/dnf5.log
  /var/log/dnf5.log.1
  /var/log/dnf5.log.2
  /var/log/dnf5.log.3
  /var/log/dnf5.log.4

Lint warning: var-tmpfiles: Found content in /var missing systemd tmpfiles.d entries:
  d /var/lib/containers 0755 root root - -
  d /var/lib/dnf 0755 root root - -
  d /var/lib/dnf/repos 0755 root root - -
  d /var/lib/dnf/repos/fedora-f8e7c8bda68a349e 0755 root root - -
  d /var/lib/dnf/repos/updates-79babcf8637033ce 0755 root root - -
  ...and 430 more
Found non-directory/non-symlink files in /var:
  var/lib/dnf/repos/fedora-f8e7c8bda68a349e/countme
  var/lib/dnf/repos/updates-79babcf8637033ce/countme
  var/lib/selinux/targeted/semanage.read.LOCK
  var/lib/selinux/targeted/semanage.trans.LOCK
  var/lib/selinux/targeted/active/modules/100/zabbix/cil
  ...and 1285 more

Checks passed: 8
Checks skipped: 1
Warnings: 4

It's not clear if any of these are causing the issue though...

[ericcurtin]

I think I see what is happening, there's a "dnf update" at some point, leaving us with two kernel versions because there's a new kernel version available. I run 3 random installation scripts here, any of them could be performing a "dnf update".

bootc container lint could be enhanced to check for this case.

This failed badly for me:

https://docs.fedoraproject.org/en-US/bootc/initramfs/

I tried to change the kver line to this:

kver=$(ls -1 /usr/lib/modules | sort -V | tail -n1)

and dracut was erroring all over the place, so not sure I want to go down that path...

The simple fix might be to just add:

RUN sudo dnf versionlock add kernel

early in the Containerfile.

[ericcurtin]

Removing

build_llama_and_whisper.sh

fixed it for me, I think that was doing the "dnf update"

[ericcurtin]

This check should have really failed though, not sure why it wasn't triggered:

#[distributed_slice(LINTS)]
static LINT_KERNEL: Lint = Lint::new_fatal(
    "kernel",
    indoc! { r#"
             Check for multiple kernels, i.e. multiple directories of the form /usr/lib/modules/$kver.
             Only one kernel is supported in an image.
     "# },
    check_kernel,
);

[ericcurtin]

Maybe this should be upgraded to error "nonempty-boot: Found non-empty /boot"