OKD - Cannot update to 4.8.0-0.okd-2021-10-24-061736

[cronchiq]

Describe the bug
Cannot upgrade OKD cluster from 4.7.0-0.okd-2021-09-19-013247 to 4.8.0-0.okd-2021-10-24-061736. Progress is stuck at 97% and 2 nodes are in status Ready, SchedulingDisabled.

Update status:
$ oc adm upgrade
info: An upgrade is in progress. Unable to apply 4.8.0-0.okd-2021-10-24-061736: wait has exceeded 40 minutes for these operators: openshift-apiserver

Upgradeable=False

Reason: ClusterOperatorsNotUpgradeable
Message: Multiple cluster operators should not be upgraded between minor versions:

• Cluster operator kube-apiserver should not be upgraded between minor versions: KubeletMinorVersion_KubeletMinorVersionUnsupportedNextUpgrade: KubeletMinorVersionUpgradeable: Kubelet minor versions on 6 nodes will not be supported in the next OpenShift minor version upgrade.
• Cluster operator machine-config should not be upgraded between minor versions: DegradedPool: One or more machine config pools are degraded, please see oc get mcp for further details and resolve before upgrading

Updates:

VERSION IMAGE
4.8.0-0.okd-2021-11-14-052418 registry.ci.openshift.org/origin/release@sha256:9d977baaf76c5adf95ab5e4f1c0cfc94d5206ca2e437c423bdd633db124a1fe3

NAME STATUS ROLES AGE VERSION
master1.**** Ready,SchedulingDisabled master,worker 140d v1.20.7+bbbc079-1330
master2.**** Ready master,worker 140d v1.20.7+bbbc079-1330
master3.**** Ready master,worker 140d v1.20.7+bbbc079-1330
worker1.**** Ready,SchedulingDisabled app,worker 140d v1.20.7+bbbc079-1330
worker2.**** Ready app,worker 140d v1.20.7+bbbc079-1330
worker5.**** Ready app,worker 104d v1.20.7+bbbc079-1330

System alerts this errors:
Error detected in pivot logs on worker1.****
Error detected in pivot logs on master1.****

And describe from nodes:
$ oc get node/master1.**** -o yaml
apiVersion: v1
kind: Node
metadata:
annotations:
machineconfiguration.openshift.io/controlPlaneTopology: HighlyAvailable
machineconfiguration.openshift.io/currentConfig: rendered-master-4e6c45d5fed0ea672918b57489b1163a
machineconfiguration.openshift.io/desiredConfig: rendered-master-4192bb7beed3ee1b233a3732e5fa15ae
machineconfiguration.openshift.io/reason: |-
failed to update OS to quay.io/openshift/okd-content@sha256:a9c5b763321544965ea2b6cac5e7dce68db1beab0af74cdd0dbfec5ca5a5a472 : error running rpm-ostree rebase --experimental /run/mco-machine-os-content/os-content-165115937/srv/repo:496af093a51ce9c29f255d8b82f3249311418727ffdb1238a345cb72030cb62e --custom-origin-url pivot://quay.io/openshift/okd-content@sha256:a9c5b763321544965ea2b6cac5e7dce68db1beab0af74cdd0dbfec5ca5a5a472 --custom-origin-description Managed by machine-config-operator: error: Multiple subdirectories found in: usr/lib/modules
: exit status 1

Version
4.7.0-0.okd-2021-09-19-013247

How reproducible
Update to 4.8.0-0.okd-2021-10-24-061736

[vrutkovs]

Please attach (or upload to the public file sharing service) must-gather archive

[cronchiq]

I attached must-gather archive.

[cronchiq]

Do you have any update please?

[vrutkovs]

2022-01-05T16:48:59.812635658+01:00 Deployments:
2022-01-05T16:48:59.812635658+01:00 * pivot://quay.io/openshift/okd-content@sha256:2906cffef5c5aede29c46adfbd6f8bc2eb7605cbb011335b20f242415d76fbb0
2022-01-05T16:48:59.812635658+01:00               CustomOrigin: Managed by machine-config-operator
2022-01-05T16:48:59.812635658+01:00                    Version: 47.34.202109181327-0 (2021-09-18T13:31:04Z)
2022-01-05T16:48:59.812635658+01:00       ReplacedBasePackages: kernel kernel-core kernel-modules 5.13.12-200.fc34 -> 5.12.19-300.fc34
2022-01-05T16:48:59.812635658+01:00 
2022-01-05T16:48:59.812635658+01:00   pivot://quay.io/openshift/okd-content@sha256:6c83d7bb87002067ff8376053bb2e3a0b980e3cf8e5b8765b55bd0bb3d239a55
2022-01-05T16:48:59.812635658+01:00               CustomOrigin: Managed by machine-config-operator
2022-01-05T16:48:59.812635658+01:00                    Version: 47.34.202108121216-0 (2021-08-12T12:20:21Z)
2022-01-05T16:48:59.812635658+01:00       ReplacedBasePackages: kernel kernel-core kernel-modules 5.13.4-200.fc34 -> 5.12.19-300.fc34

See https://ask.fedoraproject.org/t/multiple-installed-kernels-on-silverblue/7486/4

[gialloguitar]

The same issue when update OKD 4.7 to 4.8
I have downgrade kernel at worker nodes due to following issue:
coreos/fedora-coreos-tracker#957

But now node has a next statement
` machineconfiguration.openshift.io/reason: >-
failed to update OS to
quay.io/openshift/okd-content@sha256:a9c5b763321544965ea2b6cac5e7dce68db1beab0af74cdd0dbfec5ca5a5a472
: error running rpm-ostree rebase --experimental
/run/mco-machine-os-content/os-content-013672188/srv/repo:496af093a51ce9c29f255d8b82f3249311418727ffdb1238a345cb72030cb62e
--custom-origin-url
pivot://quay.io/openshift/okd-content@sha256:a9c5b763321544965ea2b6cac5e7dce68db1beab0af74cdd0dbfec5ca5a5a472
--custom-origin-description Managed by machine-config-operator: error:
Multiple subdirectories found in: usr/lib/modules

  : exit status 1`

[cronchiq]

I reinstalled all nodes (one by one), start with worker nodes and than master nodes. After that, upgrade was done.

Be careful when you will reinstal master nodes, you have to resync ETCD on new (reinstal) node.

[gialloguitar]

Wow, Thanks! I will try