Upgrade from 4.14.0-0.okd-2024-01-26-175629 to 4.15.0-0.okd-2024-03-10-010116 is failing

[mliker]

Hi okd users

has anybody experienced the following error?

MCO failing to update nodes

I tried setting selinux to premissive to no avail

I0416 11:58:34.231452    3581 daemon.go:1433] Previous boot ostree-finalize-staged.service appears successful
I0416 11:58:34.231462    3581 daemon.go:1557] Current config: rendered-worker-a5d22f0100b7a7607505a8a65a4089fa
I0416 11:58:34.231465    3581 daemon.go:1558] Desired config: rendered-worker-b3a086eecfe2cbaf3c08c719c56b758d
I0416 11:58:34.231468    3581 daemon.go:1566] state: Degraded
I0416 11:58:34.231478    3581 update.go:2326] Running: rpm-ostree cleanup -r
Deployments unchanged.
I0416 11:58:34.248890    3581 daemon.go:1966] Validating against current config rendered-worker-a5d22f0100b7a7607505a8a65a4089fa
I0416 11:58:34.248963    3581 daemon.go:1879] SSH key location ("/home/core/.ssh/authorized_keys.d/ignition") up-to-date!
I0416 11:58:35.671676    3581 rpm-ostree.go:308] Running captured: rpm-ostree kargs
I0416 11:58:35.692093    3581 update.go:2341] Validated on-disk state
I0416 11:58:35.693384    3581 update.go:2363] Adding SIGTERM protection
I0416 11:58:35.701430    3581 update.go:676] Checking Reconcilable for config rendered-worker-a5d22f0100b7a7607505a8a65a4089fa to rendered-worker-b3a086eecfe2cbaf3c08c719c56b758d
I0416 11:58:35.719391    3581 update.go:2341] Starting update from rendered-worker-a5d22f0100b7a7607505a8a65a4089fa to rendered-worker-b3a086eecfe2cbaf3c08c719c56b758d: &{osUpdate:true kargs:false fips:false passwd:false files:true units:true kernelType:false extensions:false}
I0416 11:58:35.720067    3581 helpers.go:1004] File diff: detected change to /usr/local/bin/configure-ovs.sh
I0416 11:58:35.720070    3581 helpers.go:1000] File diff: /etc/sysctl.d/gc-thresh.conf was added
I0416 11:58:35.720073    3581 helpers.go:1004] File diff: detected change to /etc/NetworkManager/dispatcher.d/pre-up.d/10-ofport-request.sh
I0416 11:58:35.720079    3581 helpers.go:1000] File diff: /etc/nmstate/nmstate.conf was added
I0416 11:58:35.720091    3581 helpers.go:1004] File diff: detected change to /etc/kubernetes/kubelet.conf
I0416 11:58:35.720103    3581 helpers.go:1004] File diff: detected change to /etc/crio/crio.conf.d/00-default
I0416 11:58:35.724266    3581 upgrade_monitor.go:226] MCN Featuregate is not enabled. Please enable the TechPreviewNoUpgrade featureset to use MachineConfigNodes
I0416 11:58:35.724273    3581 update.go:2341] drain is already completed on this node
I0416 11:58:35.725149    3581 update.go:1555] Updating files
I0416 11:58:35.725153    3581 file_writers.go:233] Writing file "/usr/local/bin/nm-clean-initrd-state.sh"
I0416 11:58:35.735812    3581 file_writers.go:233] Writing file "/etc/NetworkManager/conf.d/01-ipv6.conf"
I0416 11:58:35.746530    3581 file_writers.go:233] Writing file "/etc/NetworkManager/conf.d/20-keyfiles.conf"
I0416 11:58:35.756682    3581 file_writers.go:233] Writing file "/etc/pki/ca-trust/source/anchors/openshift-config-user-ca-bundle.crt"
I0416 11:58:35.766584    3581 file_writers.go:233] Writing file "/etc/audit/rules.d/mco-audit-quiet-containers.rules"
I0416 11:58:35.776245    3581 update.go:2371] Removing SIGTERM protection
E0416 11:58:35.776253    3581 writer.go:226] Marking Degraded due to: command "/usr/bin/rpm -qf /etc/audit/rules.d/mco-audit-quiet-containers.rules" returned with unexpected error: error: file /etc/audit/rules.d/mco-audit-quiet-containers.rules: Permission denied
: exit status 1

[mliker]

Got help in k8s slack. This is tracked in https://issues.redhat.com/browse/OCPBUGS-30149 and a workaround was to apply environment variable RPMOSTREE_CLIWRAP_SKIP=true to machine-config-daemon daemonset