deployment of 4.16.0-0.okd-scos-2024-07-01-134315 on vSphere UPI almost finishes, but the cluster operator control-plane-machine-set is not available

[kai-uwe-rommel]

The installation on vSphere UPI almost finishes with
Unable to apply 4.16.0-0.okd-scos-2024-07-01-134315: the cluster operator control-plane-machine-set is not available

The operator logs:

I0702 16:27:23.196986       1 status.go:119] "msg"="Observed Machine Configuration" "controller"="controlplanemachineset" "name"="cluster" "namespace"="openshift-machine-api" "observedGeneration"=1 "readyReplicas"=0 "reconcileID"="abfb8203-7851-40fe-b7e9-0ba4cbc8d59b" "replicas"=0 "unavailableReplicas"=3 "updatedReplicas"=0
E0702 16:27:23.197306       1 controller.go:662] "msg"="Observed unmanaged control plane nodes" "error"="found unmanaged control plane nodes, the following node(s) do not have associated machines: master-01.kur-test.ars.de, master-02.kur-test.ars.de, master-03.kur-test.ars.de" "controller"="controlplanemachineset" "name"="cluster" "namespace"="openshift-machine-api" "reconcileID"="abfb8203-7851-40fe-b7e9-0ba4cbc8d59b" "unmanagedNodes"="master-01.kur-test.ars.de,master-02.kur-test.ars.de,master-03.kur-test.ars.de"
I0702 16:27:23.197663       1 controller.go:281] "msg"="Cluster state is degraded. The control plane machine set will not take any action until issues have been resolved." "controller"="controlplanemachineset" "name"="cluster" "namespace"="openshift-machine-api" "reconcileID"="abfb8203-7851-40fe-b7e9-0ba4cbc8d59b"
I0702 16:27:23.198280       1 controller.go:225] "msg"="Finished reconciling control plane machine set" "controller"="controlplanemachineset" "name"="cluster" "namespace"="openshift-machine-api" "reconcileID"="abfb8203-7851-40fe-b7e9-0ba4cbc8d59b"

On previous 4.15 OKD releases (also SCOS) this problem did not occur.

[kai-uwe-rommel]

Exactly the same happens when installing 4.17.0-0.okd-scos-2024-07-02 so that operator seems to have a bug now.

[bdlink]

Kai-Uwe, Since June 27, the scos 4.17 builds have been failing. That is why they never show as accepted (and green). The scos 4.16 builds have yet to work. If you check the build test results, you will see the failures. As Red Hat works through these, I expect to see working builds again. Bruce On Jul 2, 2024, at 9:13 PM, kai-uwe-rommel ***@***.***> wrote:  CAUTION: This email originated from outside of BCIT. Do not click links or open attachments unless you recognize the sender and know the content is safe. Exactly the same happens when installing 4.17.0-0.okd-scos-2024-07-02 so that operator seems to have a bug now. — Reply to this email directly, view it on GitHub<#1962 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAEBGE2KIWVZWLMKFDXFWHTZKN25TAVCNFSM6AAAAABKH5JVKOVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TSNBTGQ4TE>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[kai-uwe-rommel]

@bdlink no need to defend the OKD project. :-)
I know they are working hard ...
But actually we were asked to test early and report results exactly here as discussions.
And since I have found a specific problem I think it is necessary to document it here.
The question is, if that same operator is also failing in OCP pre-releases. In this case I could create a Jira issue for it ...
Otherwise, even though the CI builds apparently fail, I was surprised that my two deployments of a 4.16 and a 4.17 OKD went flawlessly on vSphere UPI except for this bug.

[jcpowermac]

Did you delete the files as specified in docs?
https://docs.openshift.com/container-platform/4.16/installing/installing_vsphere/upi/installing-vsphere.html#installation-user-infra-generate-k8s-manifest-ignition_installing-vsphere

rm -f openshift/99_openshift-cluster-api_master-machines-*.yaml openshift/99_openshift-cluster-api_worker-machineset-*.yaml openshift/99_openshift-machine-api_master-control-plane-machine-set.yaml

[kai-uwe-rommel]

Ah, there is now one more file to be deleted and its name sounds like it matches with my problem. :-)
This is there already in the docs for 4.15 and 4.14 but has escaped me so far.
Not deleting the 99_openshift-machine-api_master-control-plane-machine-set.yaml did not hurt in previous versions, though.
Of course, I will add this to my scripts and re-test. I assume this will fix the problem.
Thanks for the pointer!

[kai-uwe-rommel]

Deployment now worked fine for me on vSphere UPI.

[madpearl]

It is interessant to know (check), should it work also and an upgrade from 4.15.okd.scos to 4.16.okd.scos (all builds are "rejected") ????

[kai-uwe-rommel]

No idea ... did not try this (do not have an older SCOS based cluster).