Stuck upgrading from 4.13.0-0.okd-2023-05-22-052007 to 4.14.0-0.okd-2023-12-01-225814

[donghuuhieu137]

Hi team,
I'm deploying OKD version 4.14.0-0.okd-2023-12-01-225814 with following info:

# --- PXE ---
pxe_initramfs_img: "fedora-coreos-38.20231027.3.2-live-initramfs.x86_64.img"
pxe_kernel: "fedora-coreos-38.20231027.3.2-live-kernel-x86_64"
pxe_rootfs_img: "fedora-coreos-38.20231027.3.2-live-rootfs.x86_64.img"
pxe_iso: "fedora-coreos-38.20231027.3.2-live.x86_64.iso"
#
# --- Openshift Client ---
openshift_client_file_tar: openshift-client-linux-4.14.0-0.okd-2023-12-01-225814.tar.gz
#
# --- Openshift Installer ---
openshift_installer_file_tar: openshift-install-linux-4.14.0-0.okd-2023-12-01-225814.tar.gz
openshift_installer_coreos_file_tar: fedora-coreos-38.20231027.3.2-metal.x86_64.raw.xz
openshift_installer_coreos_sig_file_tar: fedora-coreos-38.20231027.3.2-metal.x86_64.raw.xz.sig

We're using local quayio registry for bootstrap node to pull okd and okd-content image. We download image with following oc command:

export OCP_RELEASE=4.14.0-0.okd-2023-12-01-225814
export LOCAL_REGISTRY='local.quay.online:8443'
export LOCAL_REPOSITORY='ocp4/openshift4'
export PRODUCT_REPO='openshift'
export RELEASE_NAME="okd"
export LOCAL_SECRET_JSON='/user/pull-secret.json'

oc adm release mirror -a ${LOCAL_SECRET_JSON} --to-dir=./mirror quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}

And then we import okd image by following command:

oc image mirror -a ${LOCAL_SECRET_JSON} --from-dir=./mirror "file://openshift/release:${OCP_RELEASE}*" ${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}

We saw that both download and import images process complete with success logs. We also check on registry that quay registry has enough images from okd version.

But when we boot the bootstrap node it logs like bellow which image from okd-content registry.

We tried to downgrade the version to 4.13.0-0.okd-2023-05-22-052007 and it work success fully without error pull. The config 4.13 is bellow:

# --- PXE ---
pxe_initramfs_img: "fedora-coreos-38.20230414.3.0-live-initramfs.x86_64.img"
pxe_kernel: "fedora-coreos-38.20230414.3.0-live-kernel-x86_64"
pxe_rootfs_img: "fedora-coreos-38.20230414.3.0-live-rootfs.x86_64.img"
pxe_iso: "fedora-coreos-38.20230414.3.0-live.x86_64.iso"
#
# --- Openshift Client ---
openshift_client_file_tar: openshift-client-linux-4.13.0-0.okd-2023-05-22-052007.tar.gz
#
# --- Openshift Installer ---
openshift_installer_file_tar: openshift-install-linux-4.13.0-0.okd-2023-05-22-052007.tar.gz
openshift_installer_coreos_file_tar: fedora-coreos-38.20230414.3.0-metal.x86_64.raw.xz
openshift_installer_coreos_sig_file_tar: fedora-coreos-38.20230414.3.0-metal.x86_64.raw.xz.sig

Can any one support us ?
Thank you in advance !!!

[vrutkovs]

Check if setting oc adm release mirror --keep-manifest-list=true works?

Please upload log bundle to a public file sharing site so that we could have a look at more logs

[donghuuhieu137]

We tried oc adm release mirror --keep-manifest-list=true but it didn't work.
Please take a look our log bundle, thank you
log-bundle-20231220090239.tar.gz

[vrutkovs]

Wed 2023-12-20 08:59:31 UTC bootstrap release-image-pivot.service[19844]: time="2023-12-20T08:59:31Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://quay.io/openshift/okd-content@sha256:9d7531141cc297d75785cd3adc353ae98cd3738af46205de14c14ef2cd6039d0: (Mirrors also failed: [registry.az1.lab.xplat.online:8443/ocp4/openshift4@sha256:9d7531141cc297d75785cd3adc353ae98cd3738af46205de14c14ef2cd6039d0: reading manifest sha256:9d7531141cc297d75785cd3adc353ae98cd3738af46205de14c14ef2cd6039d0 in registry.az1.lab.xplat.online:8443/ocp4/openshift4: manifest unknown]): quay.io/openshift/okd-content@sha256:9d7531141cc297d75785cd3adc353ae98cd3738af46205de14c14ef2cd6039d0: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 54.156.243.51:443: connect: no route to host"

Which is fedora-coreos image. See #1815

[donghuuhieu137]

we try again and it work with option you mention above, thank you very much