OKD 4.10 automatic update stuck - MCP in degraded state

[smanga00]

Hi all,
we have an issue with an automatic update blocked with an MCP in degraded state due to an error encountered during the update of Machine config on a worker node.

# oc get mcp
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-60944e81511f261a4a55785b7adc3552   True      False      False      3              3                   3                     0                      594d
worker   rendered-worker-5b14386fe7094ca64e6c30e90d6e8a90   False     True       True       18             5                   5                     1                      594d

The error below:

oc -n openshift-machine-config-operator logs machine-config-daemon-xq6jz -c machine-config-daemon -f --tail=0

E0731 12:46:58.236482    2150 daemon.go:506] Preflight config drift check failed: content mismatch for file "/etc/kubernetes/kubelet-ca.crt"
E0731 12:46:58.236495    2150 writer.go:135] Marking Degraded due to: content mismatch for file "/etc/kubernetes/kubelet-ca.crt"
E0731 12:47:58.301502    2150 on_disk_validation.go:209] content mismatch for file "/etc/kubernetes/kubelet-ca.crt" (-want +got):
  []uint8(
        """
        ... // 58 identical lines
        -----END CERTIFICATE-----
        -----BEGIN CERTIFICATE-----
+       MIIDlTCCAn2gAwIBAgIISb4OW9+FpqAwDQYJKoZIhvcNAQELBQAwWDFWMFQGA1UE
+       AwxNb3BlbnNoaWZ0LWt1YmUtYXBpc2VydmVyLW9wZXJhdG9yX2t1YmUtYXBpc2Vy
+       dmVyLXRvLWt1YmVsZXQtc2lnbmVyQDE2ODk5Mzk3OTQwHhcNMjMwNzIxMTE0MzEz
+       WhcNMjQwNzIwMTE0MzE0WjBYMVYwVAYDVQQDDE1vcGVuc2hpZnQta3ViZS1hcGlz
+       ZXJ2ZXItb3BlcmF0b3Jfa3ViZS1hcGlzZXJ2ZXItdG8ta3ViZWxldC1zaWduZXJA
+       MTY4OTkzOTc5NDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAM/ykxGP
+       PJfxiHbgTNf+mYQ/S3tgSuTHpNIo6/q9Z6/lq1X+BuENGPwo5NZocWm4x6eI4Wvu
+       BX+WTkxy9eX4a2IhXX+EEIAud3p3PJqSiL0UB3jSlj7j4nY9EcBUkgvtVi7mUeZI
+       b0YdYRpTz8/+HXbwRWryuLVJ1Kw9L77W8Xur0CDshoemIgYZBhWG8U7zJ0O0fS5C
+       +XZU8qkuBDgt0zHhuNZLfev4nqhzgNKKL9CH6ypbwfz4gQWzsd3x+wTevk23mzdF
+       hrZVJUmB6qoT3OwP81T2ZaJp4W8UiXSU4zE0Kg4iVbCtWFDskJL7up9nZ7pZSgFe
+       lAVcXOR82dHCo08CAwEAAaNjMGEwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB/wQF
+       MAMBAf8wHQYDVR0OBBYEFBxuCmqX99JRNBgxeKzVo3CtZalUMB8GA1UdIwQYMBaA
+       FBxuCmqX99JRNBgxeKzVo3CtZalUMA0GCSqGSIb3DQEBCwUAA4IBAQBA9fQpBOLA
+       XPiBpqTEMhVpz37GZ4tX3oQFvxpgjF6jTrmB/+Imrm/xXfq1UAwkRU3Gndb0+rpC
+       z599FLKypSOjk4EHV1oSb+jey8zQ3oXs7/9UQ5QSNH27cLGX2KEeMRI7b5OUZhwI
+       kpqBUbXssGsTr2UrPyZFtibGiuqxj8caKGDoPGG1L8ArqrrXqz4a0A8HfOEouk8i
+       s3rIz8UizDL/Id+OtV2a7Fvb07EUXiD1MMh+hPtNT3oUFTQGueEB3RrwqO9xccUT
+       FZbLfm2Rhn2EI2c/Q3uGjAIS01BzTIiwx26VZONXtc/9eMaEQBJMyJsPgVBR9tN0
+       l/3huSYKFRV+
+       -----END CERTIFICATE-----
+       -----BEGIN CERTIFICATE-----
        MIIDlTCCAn2gAwIBAgIIKwe9fejXY0EwDQYJKoZIhvcNAQELBQAwWDFWMFQGA1UE
        AwxNb3BlbnNoaWZ0LWt1YmUtYXBpc2VydmVyLW9wZXJhdG9yX2t1YmUtYXBpc2Vy
        ... // 40 identical lines
        """
  )

We applied the solution at https://access.redhat.com/solutions/5099331 but doesn't seem to solve the issue; after re-applying MC the error is still present.

Version

4.10.0-0.okd-2022-07-09-073606 - IPI mode on VMware Vsphere

How can we solve the issue so that the update procedure restarts and perform the update on the other nodes?

Thank you in advance
Regards
Stefano

[smanga00]

Deleting worker node in which the update is failing could be a solution or may I incur on other problems since the update is in progress?

[vrutkovs]

Please attach (or upload to the public file sharing service) must-gather archive

[smanga00]

Must gather too big to be attached here.
Here is the link (valid for two weeks): https://file.io/CIuv59m15fIW.

If can be useful for the analisys, I submit another info:
When we connect to the worker node in which the problem occurs, we are shown hundreds of messages like that:

Failed Units: 116136
  crio-conmon-000181324c46e77e383ace88d35fb2554734292e17f467fcb056a6477cdf3755.scope
  crio-conmon-00018a446b18c73dd35bdfbf8cc9063525c91b23e1c3f1daeafa48c72e58cdd7.scope
  crio-conmon-0002930b90bf01f54675f7a008a71edd5c3804d7b2eb489f0117f3e998f0643e.scope
  crio-conmon-0002ab773fef9dc302c75c06366dbcacca3f30cc704ed4672e8c03ebcfacf879.scope
  crio-conmon-0002eb9e9e6bf20652f9e6e45ee3c583ca886c3158cb8cd0fa15bcf2151f7cbc.scope
  crio-conmon-0003db78b9ba8938a012e567bb9eba61ca76638c0a70e40f3b365095d88fdd4a.scope
  crio-conmon-0003f91a8a962c3abf37e099e2f16a075a1b01ce5aa55a256d1a27b434612ea9.scope
  crio-conmon-00043dec54862c6b633f5d75e579e8232916135a25001383d9cc9325ee5c71a1.scope
  crio-conmon-0004410d9136c51dbd1eaf9be405a9e86a0b9bdd2075a68c0bc182b5aaf93d5b.scope

[smanga00]

Hi Vadim,
Did you get a chance to take a first look?
Thank you for your kind support.
Stefano

[melledouwsma]

Did you get a chance to take a first look?

Would you be able to provide a fresh (download for) the oc adm must-gather? The link supplied above is no longer valid. Thanks.

[smanga00]

Sorry for the inconvenience.
Issue fixed today by replacing the file /etc/kubernetes/kubelet-ca.crt with the content taken from current machine config.
Than the update process continued with the remaining nodes and finished successfully.

Thank you all for your time.