Upgrade cli OKD FCOS Cluster to OKD SCOS cluster

[madpearl]

Hi to everybody,
I've just finished upgrading of OKD FCOS libvirt, bare metal 5 nodes Cluster to first Acceptable OKD SCOS version 4.15.0-0.okd-scos-2024-01-18-223523, which should be the start version of all upgrade path to version 4.17 OKD-SCOS.
All clusters which I'm upgrading are test and developer, the production should be the last one!
Next steps should be upgrade to 4.16.0-0.okd-scos-2024-07-21-033952 on my SNO cluster and then again back to my test developer 5 node libvirt, bare - metal cluster!

[root@terraform-runner okd-scos]# oc adm upgrade --force --allow-explicit-upgrade --to-image=quay.io/okd/scos-release:4.15.0-0.okd-scos-2024-01-18-223523
warning: Using by-tag pull specs is dangerous, and while we still allow it in combination with --force for backward compatibility, it would be much safer to pass a by-digest pull spec instead
warning: The requested upgrade image is not one of the available updates. You have used --allow-explicit-upgrade for the update to proceed anyway
warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures.
Requested update to release image quay.io/okd/scos-release:4.15.0-0.okd-scos-2024-01-18-223523
[root@terraform-runner okd-scos]# oc get co
NAME                                       VERSION                          AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.15.0-0.okd-2024-03-10-010116   True        False         False      5d1h
baremetal                                  4.15.0-0.okd-2024-03-10-010116   True        False         False      2y38d
cloud-controller-manager                   4.15.0-0.okd-2024-03-10-010116   True        False         False      2y38d
cloud-credential                           4.15.0-0.okd-2024-03-10-010116   True        False         False      2y38d
cluster-autoscaler                         4.15.0-0.okd-2024-03-10-010116   True        False         False      2y38d
config-operator                            4.15.0-0.okd-2024-03-10-010116   True        False         False      2y38d
console                                    4.15.0-0.okd-2024-03-10-010116   True        False         False      20d
control-plane-machine-set                  4.15.0-0.okd-2024-03-10-010116   True        False         False      540d
csi-snapshot-controller                    4.15.0-0.okd-2024-03-10-010116   True        False         False      256d
dns                                        4.15.0-0.okd-2024-03-10-010116   True        False         False      616d
etcd                                       4.15.0-0.okd-2024-03-10-010116   True        False         False      615d
image-registry                             4.15.0-0.okd-2024-03-10-010116   True        False         False      7d22h
ingress                                    4.15.0-0.okd-2024-03-10-010116   True        False         False      7d22h
insights                                   4.15.0-0.okd-2024-03-10-010116   True        False         False      540d
kube-apiserver                             4.15.0-0.okd-2024-03-10-010116   True        False         False      2y38d
kube-controller-manager                    4.15.0-0.okd-2024-03-10-010116   True        False         False      2y38d
kube-scheduler                             4.15.0-0.okd-2024-03-10-010116   True        False         False      2y38d
kube-storage-version-migrator              4.15.0-0.okd-2024-03-10-010116   True        False         False      35d
machine-api                                4.15.0-0.okd-2024-03-10-010116   True        False         False      2y38d
machine-approver                           4.15.0-0.okd-2024-03-10-010116   True        False         False      2y38d
machine-config                             4.15.0-0.okd-2024-03-10-010116   True        False         False      66d
marketplace                                4.15.0-0.okd-2024-03-10-010116   True        False         False      2y38d
monitoring                                 4.15.0-0.okd-2024-03-10-010116   True        False         False      174d
network                                    4.15.0-0.okd-2024-03-10-010116   True        False         False      2y38d
node-tuning                                4.15.0-0.okd-2024-03-10-010116   True        False         False      7d22h
openshift-apiserver                        4.15.0-0.okd-2024-03-10-010116   True        False         False      35d
openshift-controller-manager               4.15.0-0.okd-2024-03-10-010116   True        False         False      520d
openshift-samples                          4.15.0-0.okd-2024-03-10-010116   True        False         False      133d
operator-lifecycle-manager                 4.15.0-0.okd-2024-03-10-010116   True        False         False      2y38d
operator-lifecycle-manager-catalog         4.15.0-0.okd-2024-03-10-010116   True        False         False      2y38d
operator-lifecycle-manager-packageserver   4.15.0-0.okd-2024-03-10-010116   True        False         False      7d22h
service-ca                                 4.15.0-0.okd-2024-03-10-010116   True        False         False      2y38d
storage                                    4.15.0-0.okd-2024-03-10-010116   True        False         False      2y38d
[root@terraform-runner okd-scos]# oc adm upgrade
info: An upgrade is in progress. Working towards 4.15.0-0.okd-scos-2024-01-18-223523: 68 of 873 done (7% complete), waiting on config-operator

Upstream: https://amd64.origin.releases.ci.openshift.org/graph
Channel: stable-4
No updates available. You may still upgrade to a specific release image with --to-image or wait for new updates to be available.
[root@terraform-runner okd-scos]# oc get co
NAME                                       VERSION                               AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      5d1h
baremetal                                  4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      2y38d
cloud-controller-manager                   4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      2y38d
cloud-credential                           4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      2y38d
cluster-autoscaler                         4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      2y38d
config-operator                            4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      2y38d
console                                    4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      20d
control-plane-machine-set                  4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      540d
csi-snapshot-controller                    4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      256d
dns                                        4.15.0-0.okd-2024-03-10-010116        True        True          False      616d    DNS "default" reports Progressing=True: "Have 7 available DNS pods, want 8.\nHave 0 up-to-date DNS pods, want 8."...
etcd                                       4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      615d
image-registry                             4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      7d22h
ingress                                    4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      7d22h
insights                                   4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      540d
kube-apiserver                             4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      2y38d
kube-controller-manager                    4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      2y38d
kube-scheduler                             4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      2y38d
kube-storage-version-migrator              4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      35d
machine-api                                4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      2y38d
machine-approver                           4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      2y38d
machine-config                             4.15.0-0.okd-2024-03-10-010116        True        False         False      66d
marketplace                                4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      2y38d
monitoring                                 4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      174d
network                                    4.15.0-0.okd-2024-03-10-010116        True        True          False      2y38d   DaemonSet "/openshift-multus/multus" update is rolling out (2 out of 8 updated)...
node-tuning                                4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      7d22h
openshift-apiserver                        4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      35d
openshift-controller-manager               4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      520d
openshift-samples                          4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      9m46s
operator-lifecycle-manager                 4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      2y38d
operator-lifecycle-manager-catalog         4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      2y38d
operator-lifecycle-manager-packageserver   4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      7d22h
service-ca                                 4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      2y38d
storage                                    4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      2y38d
[root@terraform-runner okd-scos]# oc get nodes -owide
NAME                            STATUS                        ROLES    AGE     VERSION           INTERNAL-IP       EXTERNAL-IP   OS-IMAGE                                    KERNEL-VERSION          CONTAINER-RUNTIME
libvirt-cp-1.libvirt.alma.lan   Ready                         master   2y38d   v1.28.3+20a5764   192.168.122.101   <none>        CentOS Stream CoreOS 415.9.202401181453-0   5.14.0-408.el9.x86_64   cri-o://1.28.2
libvirt-cp-2.libvirt.alma.lan   Ready,SchedulingDisabled      master   2y38d   v1.28.7+6e2789b   192.168.122.102   <none>        Fedora CoreOS 39.20240210.3.0               6.7.4-200.fc39.x86_64   cri-o://1.28.2
libvirt-cp-3.libvirt.alma.lan   Ready                         master   2y38d   v1.28.7+6e2789b   192.168.122.103   <none>        Fedora CoreOS 39.20240210.3.0               6.7.4-200.fc39.x86_64   cri-o://1.28.2
libvirt-w-1.libvirt.alma.lan    Ready                         worker   2y38d   v1.28.3+20a5764   192.168.122.111   <none>        CentOS Stream CoreOS 415.9.202401181453-0   5.14.0-408.el9.x86_64   cri-o://1.28.2
libvirt-w-2.libvirt.alma.lan    NotReady,SchedulingDisabled   worker   2y38d   v1.28.3+20a5764   192.168.122.112   <none>        CentOS Stream CoreOS 415.9.202401181453-0   5.14.0-408.el9.x86_64   cri-o://1.28.2
libvirt-w-3.libvirt.alma.lan    Ready                         worker   2y38d   v1.28.7+6e2789b   192.168.122.113   <none>        Fedora CoreOS 39.20240210.3.0               6.7.4-200.fc39.x86_64   cri-o://1.28.2
libvirt-w-4.libvirt.alma.lan    Ready                         worker   581d    v1.28.7+6e2789b   192.168.122.114   <none>        Fedora CoreOS 39.20240210.3.0               6.7.4-200.fc39.x86_64   cri-o://1.28.2
libvirt-w-5.libvirt.alma.lan    Ready                         worker   581d    v1.28.7+6e2789b   192.168.122.215   <none>        Fedora CoreOS 39.20240210.3.0               6.7.4-200.fc39.x86_64   cri-o://1.28.2
[root@terraform-runner okd-scos]#
[root@terraform-runner okd-scos]# oc get nodes -owide
NAME                            STATUS   ROLES    AGE     VERSION           INTERNAL-IP       EXTERNAL-IP   OS-IMAGE                                    KERNEL-VERSION          CONTAINER-RUNTIME
libvirt-cp-1.libvirt.alma.lan   Ready    master   2y38d   v1.28.3+20a5764   192.168.122.101   <none>        CentOS Stream CoreOS 415.9.202401181453-0   5.14.0-408.el9.x86_64   cri-o://1.28.2
libvirt-cp-2.libvirt.alma.lan   Ready    master   2y38d   v1.28.3+20a5764   192.168.122.102   <none>        CentOS Stream CoreOS 415.9.202401181453-0   5.14.0-408.el9.x86_64   cri-o://1.28.2
libvirt-cp-3.libvirt.alma.lan   Ready    master   2y38d   v1.28.3+20a5764   192.168.122.103   <none>        CentOS Stream CoreOS 415.9.202401181453-0   5.14.0-408.el9.x86_64   cri-o://1.28.2
libvirt-w-1.libvirt.alma.lan    Ready    worker   2y38d   v1.28.3+20a5764   192.168.122.111   <none>        CentOS Stream CoreOS 415.9.202401181453-0   5.14.0-408.el9.x86_64   cri-o://1.28.2
libvirt-w-2.libvirt.alma.lan    Ready    worker   2y38d   v1.28.3+20a5764   192.168.122.112   <none>        CentOS Stream CoreOS 415.9.202401181453-0   5.14.0-408.el9.x86_64   cri-o://1.28.2
libvirt-w-3.libvirt.alma.lan    Ready    worker   2y38d   v1.28.3+20a5764   192.168.122.113   <none>        CentOS Stream CoreOS 415.9.202401181453-0   5.14.0-408.el9.x86_64   cri-o://1.28.2
libvirt-w-4.libvirt.alma.lan    Ready    worker   581d    v1.28.3+20a5764   192.168.122.114   <none>        CentOS Stream CoreOS 415.9.202401181453-0   5.14.0-408.el9.x86_64   cri-o://1.28.2
libvirt-w-5.libvirt.alma.lan    Ready    worker   581d    v1.28.3+20a5764   192.168.122.215   <none>        CentOS Stream CoreOS 415.9.202401181453-0   5.14.0-408.el9.x86_64   cri-o://1.28.2

[root@terraform-runner okd-scos]# oc adm upgrade
Cluster version is 4.15.0-0.okd-scos-2024-01-18-223523

Upstream: https://amd64.origin.releases.ci.openshift.org/graph
Channel: stable-4
No updates available. You may still upgrade to a specific release image with --to-image or wait for new updates to be available.
[root@terraform-runner okd-scos]#
[root@terraform-runner okd-scos]# oc get co
NAME                                       VERSION                               AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      5d2h
baremetal                                  4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      2y38d
cloud-controller-manager                   4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      2y38d
cloud-credential                           4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      2y38d
cluster-autoscaler                         4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      2y38d
config-operator                            4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      2y38d
console                                    4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      20d
control-plane-machine-set                  4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      540d
csi-snapshot-controller                    4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      31m
dns                                        4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      616d
etcd                                       4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      615d
image-registry                             4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      7d23h
ingress                                    4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      8m29s
insights                                   4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      540d
kube-apiserver                             4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      2y38d
kube-controller-manager                    4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      2y38d
kube-scheduler                             4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      2y38d
kube-storage-version-migrator              4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      8m41s
machine-api                                4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      2y38d
machine-approver                           4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      2y38d
machine-config                             4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      66d
marketplace                                4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      2y38d
monitoring                                 4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      174d
network                                    4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      2y38d
node-tuning                                4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      23m
openshift-apiserver                        4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      25m
openshift-controller-manager               4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      520d
openshift-samples                          4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      56m
operator-lifecycle-manager                 4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      2y38d
operator-lifecycle-manager-catalog         4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      2y38d
operator-lifecycle-manager-packageserver   4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      28m
service-ca                                 4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      2y38d
storage                                    4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      2y38d
[root@terraform-runner okd-scos]# oc get clusterversion -o json|jq ".items[0].status.history"
[
  {
    "acceptedRisks": "Target release version=\"\" image=\"quay.io/okd/scos-release:4.15.0-0.okd-scos-2024-01-18-223523\" cannot be verified, but continuing anyway because the update was forced: release images that are not accessed via digest cannot be verified\nPrecondition \"ClusterVersionRecommendedUpdate\" failed because of \"UnknownUpdate\": RetrievedUpdates=True (), so the update from 4.15.0-0.okd-2024-03-10-010116 to 4.15.0-0.okd-scos-2024-01-18-223523 is probably neither recommended nor supported.",
    "completionTime": "2024-07-23T11:49:46Z",
    "image": "quay.io/okd/scos-release:4.15.0-0.okd-scos-2024-01-18-223523",
    "startedTime": "2024-07-23T10:37:56Z",
    "state": "Completed",
    "verified": false,
    "version": "4.15.0-0.okd-scos-2024-01-18-223523"
  },
  {
    "completionTime": "2024-03-12T09:26:02Z",
    "image": "registry.ci.openshift.org/origin/release@sha256:46b462be1e4c15ce5ab5fba97e713e8824bbb9f614ac5abe1be41fda916920cc",
    "startedTime": "2024-03-12T08:14:18Z",
    "state": "Completed",
    "verified": true,
    "version": "4.15.0-0.okd-2024-03-10-010116"
  },
  {
    "completionTime": "2024-02-27T10:12:01Z",
    "image": "registry.ci.openshift.org/origin/release@sha256:75045c88a94bbfd4a390fdd4859bb58b1f222cf71e5b85c0c086526a734ea37d",
    "startedTime": "2024-02-27T09:02:31Z",
    "state": "Completed",
    "verified": true,
    "version": "4.15.0-0.okd-2024-02-23-163410"
  },
  {
    "completionTime": "2024-02-14T08:26:42Z",
    "image": "registry.ci.openshift.org/origin/release@sha256:e13d5f24fc0347b51f1144e7f5bb03b48a99b43d3de0a260af171bf7e39a30f7",
    "startedTime": "2024-02-14T07:20:16Z",
    "state": "Completed",
    "verified": true,
    "version": "4.15.0-0.okd-2024-02-10-035534"
  },
  {
    "completionTime": "2024-01-30T12:23:22Z",
    "image": "registry.ci.openshift.org/origin/release@sha256:5e5a19e1e9b880ad4692530571130dae3165058198bfef8711342a5b44f6659d",
    "startedTime": "2024-01-30T11:01:28Z",
    "state": "Completed",
    "verified": true,
    "version": "4.15.0-0.okd-2024-01-27-070424"
  },
  {
    "completionTime": "2024-01-29T13:02:56Z",
    "image": "registry.ci.openshift.org/origin/release@sha256:ae4f6a8bc6c5b2c8de3d9833d5addc2a69cbc73c8cbed5d9474dccdd5e5a700b",
    "startedTime": "2024-01-29T11:58:34Z",
    "state": "Completed",
    "verified": true,
    "version": "4.14.0-0.okd-2024-01-26-175629"
  },
  {
    "completionTime": "2024-01-09T08:35:03Z",
    "image": "registry.ci.openshift.org/origin/release@sha256:c4a6b6850701202f629c0e451de784b02f0de079650a1b9ccbf610448ebc9227",
    "startedTime": "2024-01-09T07:34:31Z",
    "state": "Completed",
    "verified": true,
    "version": "4.14.0-0.okd-2024-01-06-084517"
  },
  {
    "completionTime": "2023-12-05T09:24:29Z",
    "image": "registry.ci.openshift.org/origin/release@sha256:4510c50834c07e71521a68c021320b5e44978245c8031a858f74328e54e89e1c",
    "startedTime": "2023-12-05T08:30:54Z",
    "state": "Completed",
    "verified": true,
    "version": "4.14.0-0.okd-2023-12-01-225814"
  },
  {
    "completionTime": "2023-11-14T17:59:21Z",
    "image": "quay.io/openshift/okd:4.14.0-0.okd-2023-11-14-101924",
    "startedTime": "2023-11-14T16:09:45Z",
    "state": "Completed",
    "verified": false,
    "version": "4.14.0-0.okd-2023-11-14-101924"
  },
  {
    "completionTime": "2023-11-14T16:09:45Z",
    "image": "registry.ci.openshift.org/origin/release@sha256:2242d1df4e4cbcc0cd27191ab9ad5f55ac4f0c60c3cda2a186181a2435e3bd00",
    "startedTime": "2023-11-13T10:07:38Z",
    "state": "Partial",
    "verified": true,
    "version": "4.14.0-0.okd-2023-11-12-042703"
  },
  {
    "completionTime": "2023-11-03T09:12:17Z",
    "image": "registry.ci.openshift.org/origin/release@sha256:7a6200e347a1b857e47f2ab0735eb1303af7d796a847d79ef9706f217cd12f5c",
    "startedTime": "2023-11-03T07:32:47Z",
    "state": "Completed",
    "verified": true,
    "version": "4.14.0-0.okd-2023-10-28-073550"
  },
  {
    "completionTime": "2023-10-30T20:05:48Z",
    "image": "registry.ci.openshift.org/origin/release@sha256:29e8ad18b5d011c3fd8275ec9aad5964e4dcfd0138d229f2917acfcbefde930c",
    "startedTime": "2023-10-30T08:08:43Z",
    "state": "Completed",
    "verified": true,
    "version": "4.13.0-0.okd-2023-10-28-065448"
  },
  {
    "completionTime": "2023-10-02T14:10:19Z",
    "image": "registry.ci.openshift.org/origin/release@sha256:fc0dd43b87942f40d7eec16782ca2965199ced131906405127837884b5342b92",
    "startedTime": "2023-10-02T12:48:24Z",
    "state": "Completed",
    "verified": true,
    "version": "4.13.0-0.okd-2023-09-30-084937"
  },
  {
    "completionTime": "2023-09-04T14:35:27Z",
    "image": "registry.ci.openshift.org/origin/release@sha256:21589242ca379cd347bfe5ce1b0cc69fb6dcc2d3f2f536663e26632532f1ae97",
    "startedTime": "2023-09-04T13:24:38Z",
    "state": "Completed",
    "verified": true,
    "version": "4.13.0-0.okd-2023-09-03-082426"
  },
  {
    "completionTime": "2023-08-21T07:41:52Z",
    "image": "registry.ci.openshift.org/origin/release@sha256:59b1e486ed132fa084f654ca930d84a751c7956870f659ef522a9928bc200ec7",
    "startedTime": "2023-08-21T06:25:21Z",
    "state": "Completed",
    "verified": true,
    "version": "4.13.0-0.okd-2023-08-18-135805"
  },
  {
    "completionTime": "2023-08-08T08:06:03Z",
    "image": "registry.ci.openshift.org/origin/release@sha256:939918fd96101657d5763170916cac1031b4720d8d2eb65fa7878e63c5d7ff80",
    "startedTime": "2023-08-08T06:57:50Z",
    "state": "Completed",
    "verified": true,
    "version": "4.13.0-0.okd-2023-08-04-164726"
  },
  {
    "completionTime": "2023-07-24T10:40:39Z",
    "image": "registry.ci.openshift.org/origin/release@sha256:f0e95c44ff7307fc4b3927ab2f1a9d30c2688f4e4ad3e8d702f04b1171bf84c6",
    "startedTime": "2023-07-24T08:45:20Z",
    "state": "Completed",
    "verified": true,
    "version": "4.13.0-0.okd-2023-07-23-051208"
  },
  {
    "completionTime": "2023-07-10T11:58:32Z",
    "image": "registry.ci.openshift.org/origin/release@sha256:dd8eeccd6dfed69b47ff892c2d8c7e2db1d419d12c3f763f06b2eeb894d0f883",
    "startedTime": "2023-07-10T10:32:03Z",
    "state": "Completed",
    "verified": true,
    "version": "4.13.0-0.okd-2023-07-09-062029"
  },
  {
    "completionTime": "2023-06-05T07:49:03Z",
    "image": "registry.ci.openshift.org/origin/release@sha256:d696b82dbdac77fa98518ef1e8891104eb937d29c6461539407b301c79ea2177",
    "startedTime": "2023-06-05T06:28:43Z",
    "state": "Completed",
    "verified": false,
    "version": "4.13.0-0.okd-2023-06-04-080300"
  },
  {
    "completionTime": "2023-05-24T08:59:07Z",
    "image": "registry.ci.openshift.org/origin/release@sha256:16b7227d0c0de198d65c4478617e402d7f67ffc0dee312110a9d4c85877e5e4d",
    "startedTime": "2023-05-24T07:13:58Z",
    "state": "Completed",
    "verified": false,
    "version": "4.13.0-0.okd-2023-05-22-052007"
  },
  {
    "completionTime": "2023-04-18T08:23:13Z",
    "image": "registry.ci.openshift.org/origin/release@sha256:c51a70b2131b67a6f1140994de46a9ec42a9c07be051ca0883f96f5a2d0f4e4c",
    "startedTime": "2023-04-18T06:47:46Z",
    "state": "Completed",
    "verified": false,
    "version": "4.12.0-0.okd-2023-04-16-041331"
  },
  {
    "completionTime": "2023-04-17T09:18:21Z",
    "image": "registry.ci.openshift.org/origin/release@sha256:a8272e9992eee6b9c9dfa1b44e7348e5f979c0de96ad60d6235477a1aa0d7897",
    "startedTime": "2023-04-17T08:01:22Z",
    "state": "Completed",
    "verified": true,
    "version": "4.12.0-0.okd-2023-04-01-051724"
  },
  {
    "completionTime": "2023-03-21T08:54:47Z",
    "image": "registry.ci.openshift.org/origin/release@sha256:7153ed89133eeaca94b5fda702c5709b9ad199ce4ff9ad1a0f01678d6ecc720f",
    "startedTime": "2023-03-21T07:33:19Z",
    "state": "Completed",
    "verified": true,
    "version": "4.12.0-0.okd-2023-03-18-084815"
  },
  {
    "completionTime": "2023-03-08T08:42:16Z",
    "image": "registry.ci.openshift.org/origin/release@sha256:fef2d0803e6ee6ec36d1979a0b847441580fbd9ee3ad583c97edb7ff811ee3b6",
    "startedTime": "2023-03-08T07:24:58Z",
    "state": "Completed",
    "verified": true,
    "version": "4.12.0-0.okd-2023-03-05-022504"
  },
  {
    "completionTime": "2023-02-22T09:15:02Z",
    "image": "registry.ci.openshift.org/origin/release@sha256:fd08a1dae13a434729451cdb6edd969714a4329904e8d27eb45d94e96021dff4",
    "startedTime": "2023-02-22T08:12:57Z",
    "state": "Completed",
    "verified": true,
    "version": "4.12.0-0.okd-2023-02-18-033438"
  },
  {
    "completionTime": "2023-02-10T07:38:22Z",
    "image": "registry.ci.openshift.org/origin/release@sha256:3c14b3cc66310aede5086fabbbff81848956b690a695abeef2ba38bee5a03145",
    "startedTime": "2023-02-07T10:16:12Z",
    "state": "Completed",
    "verified": true,
    "version": "4.12.0-0.okd-2023-02-04-212953"
  },
  {
    "completionTime": "2023-01-30T09:29:39Z",
    "image": "registry.ci.openshift.org/origin/release@sha256:8c5e4d3a76aba995c005fa7f732d68658cc67d6f11da853360871012160b2ebf",
    "startedTime": "2023-01-30T07:52:13Z",
    "state": "Completed",
    "verified": true,
    "version": "4.12.0-0.okd-2023-01-21-055900"
  },
  {
    "completionTime": "2023-01-16T13:00:05Z",
    "image": "registry.ci.openshift.org/origin/release@sha256:27c99f5b7734f773a7981b9e527417edcda0eee217117632198d64aa33e95c46",
    "startedTime": "2023-01-16T11:49:05Z",
    "state": "Completed",
    "verified": true,
    "version": "4.11.0-0.okd-2023-01-14-152430"
  },
  {
    "completionTime": "2022-12-06T10:08:45Z",
    "image": "quay.io/openshift/okd@sha256:924958ed5ba93ea9d72e4b5a319d0d4df220edbb891816c106215cbf318a0a40",
    "startedTime": "2022-12-06T08:57:10Z",
    "state": "Completed",
    "verified": true,
    "version": "4.11.0-0.okd-2022-12-02-145640"
  },
  {
    "completionTime": "2022-12-05T11:09:06Z",
    "image": "registry.ci.openshift.org/origin/release@sha256:af7ba479dcbb7eab32869f0f96f01d90978f7a949be1e9781a4eaecf56bafa2b",
    "startedTime": "2022-12-05T09:55:47Z",
    "state": "Completed",
    "verified": true,
    "version": "4.11.0-0.okd-2022-12-03-100655"
  },
  {
    "completionTime": "2022-12-05T09:44:12Z",
    "image": "registry.ci.openshift.org/origin/release@sha256:924958ed5ba93ea9d72e4b5a319d0d4df220edbb891816c106215cbf318a0a40",
    "startedTime": "2022-12-05T08:48:42Z",
    "state": "Completed",
    "verified": true,
    "version": "4.11.0-0.okd-2022-12-02-145640"
  },
  {
    "completionTime": "2022-11-21T13:09:25Z",
    "image": "registry.ci.openshift.org/origin/release@sha256:b2e925d0178130baa99ebdcdf9bfe124a1fa496cae955af5543652c2166def63",
    "startedTime": "2022-11-21T11:58:28Z",
    "state": "Completed",
    "verified": true,
    "version": "4.11.0-0.okd-2022-11-19-050030"
  },
  {
    "completionTime": "2022-11-07T08:43:53Z",
    "image": "registry.ci.openshift.org/origin/release@sha256:d37ea780f755e8cda5ba6989f43060d1d2fccd1a98f9106bda70a307d1e04d40",
    "startedTime": "2022-11-07T07:46:09Z",
    "state": "Completed",
    "verified": true,
    "version": "4.11.0-0.okd-2022-11-05-030711"
  },
  {
    "completionTime": "2022-11-04T12:03:08Z",
    "image": "registry.ci.openshift.org/origin/release@sha256:13fceee2e4e919d942b8252e8a86b1bd62371cb88cfcb45746d6f232a7b9acba",
    "startedTime": "2022-11-04T08:20:47Z",
    "state": "Completed",
    "verified": true,
    "version": "4.11.0-0.okd-2022-10-28-153352"
  },
  {
    "completionTime": "2022-10-17T07:21:45Z",
    "image": "registry.ci.openshift.org/origin/release@sha256:1f72fb4fc1452da5c1300f7f7ba376599d9cea41e00a141ba738d2620b51d542",
    "startedTime": "2022-10-17T06:24:31Z",
    "state": "Completed",
    "verified": true,
    "version": "4.11.0-0.okd-2022-10-15-073651"
  },
  {
    "completionTime": "2022-08-22T08:17:30Z",
    "image": "registry.ci.openshift.org/origin/release@sha256:7a99c7af2f3e6107107a6cf92669fef74ebbd0cc1780fbd187be2410b003a429",
    "startedTime": "2022-08-22T06:47:13Z",
    "state": "Completed",
    "verified": true,
    "version": "4.11.0-0.okd-2022-08-20-022919"
  },
  {
    "completionTime": "2022-08-05T08:09:39Z",
    "image": "registry.ci.openshift.org/origin/release@sha256:a506d80873f7c51201c614be9c8defa6455f3b593faa54d11ded7b4d20c0fc4e",
    "startedTime": "2022-08-05T06:53:23Z",
    "state": "Completed",
    "verified": true,
    "version": "4.11.0-0.okd-2022-07-29-154152"
  },
  {
    "completionTime": "2022-07-29T12:02:08Z",
    "image": "registry.ci.openshift.org/origin/release@sha256:5a748561c829c23c7413b0fe0ec8f97b4eabf5cfd1934b9e93a8a5df3f59738d",
    "startedTime": "2022-07-29T10:59:55Z",
    "state": "Completed",
    "verified": true,
    "version": "4.10.0-0.okd-2022-07-09-073606"
  },
  {
    "completionTime": "2022-06-27T11:10:23Z",
    "image": "registry.ci.openshift.org/origin/release@sha256:ed37a14e619f59dc75c3beab12ebf0f577b3edd07fe126624ac295163b59faaa",
    "startedTime": "2022-06-27T09:16:48Z",
    "state": "Completed",
    "verified": true,
    "version": "4.10.0-0.okd-2022-06-24-212905"
  },
  {
    "completionTime": "2022-06-15T17:28:50Z",
    "image": "quay.io/openshift/okd@sha256:9c65c674c012f5f4c514b3945cff12d4a4253ab91756c1563957126afd4ae3f4",
    "startedTime": "2022-06-15T15:01:34Z",
    "state": "Completed",
    "verified": false,
    "version": "4.10.0-0.okd-2022-06-10-131327"
  }
]
[root@terraform-runner okd-scos]#

[madpearl]

Upgrade from v4.15.0-0.okd-scos-2024-01-18-223523 to v4.16.0-0.okd-scos-2024-07-21-033952 following #1971 (comment)

[root@okd-svc okd-scos]# oc get mcp
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-606e8b85d433bcad1a55bbf2e13f7ef2   True      False      False      1              1                   1                     0                      48d
worker   rendered-worker-c49346b86bc4f1a300b1d0a8090dcf78   True      False      False      0              0                   0                     0                      48d
[root@okd-svc okd-scos]# oc get co
NAME                                       VERSION                               AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      22m
baremetal                                  4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      48d
cloud-controller-manager                   4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      48d
cloud-credential                           4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      48d
cluster-autoscaler                         4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      48d
config-operator                            4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      48d
console                                    4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      23m
control-plane-machine-set                  4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      48d
csi-snapshot-controller                    4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      25m
dns                                        4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      4m52s
etcd                                       4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      48d
image-registry                             4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      25m
ingress                                    4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      48d
insights                                   4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      48d
kube-apiserver                             4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      48d
kube-controller-manager                    4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      48d
kube-scheduler                             4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      48d
kube-storage-version-migrator              4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      48d
machine-api                                4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      48d
machine-approver                           4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      48d
machine-config                             4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      42d
marketplace                                4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      48d
monitoring                                 4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      18d
network                                    4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      48d
node-tuning                                4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      23m
openshift-apiserver                        4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      23m
openshift-controller-manager               4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      23m
openshift-samples                          4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      23m
operator-lifecycle-manager                 4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      48d
operator-lifecycle-manager-catalog         4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      48d
operator-lifecycle-manager-packageserver   4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      5m23s
service-ca                                 4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      48d
storage                                    4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      48d
[root@okd-svc okd-scos]# oc adm upgrade
info: An upgrade is in progress. Working towards 4.16.0-0.okd-scos-2024-07-21-033952: 321 of 895 done (35% complete)

Upstream: https://amd64.origin.releases.ci.openshift.org/graph
Channel: stable-4

Recommended updates:

  VERSION                             IMAGE
  4.16.0-0.okd-scos-2024-07-22-113533 registry.ci.openshift.org/origin/release-scos@sha256:a3bc316194a20a37a9a3999496a9f3e0bca57295b572b553d86939a8811088f6
[root@okd-svc okd-scos]# oc adm upgrade
Cluster version is 4.16.0-0.okd-scos-2024-07-21-033952

Upstream: https://amd64.origin.releases.ci.openshift.org/graph
Channel: stable-4

Recommended updates:

  VERSION                             IMAGE
  4.16.0-0.okd-scos-2024-07-22-113533 registry.ci.openshift.org/origin/release-scos@sha256:a3bc316194a20a37a9a3999496a9f3e0bca57295b572b553d86939a8811088f6
[root@okd-svc okd-scos]# oc get nodes -owide
NAME               STATUS   ROLES                         AGE   VERSION           INTERNAL-IP       EXTERNAL-IP   OS-IMAGE                                    KERNEL-VERSION          CONTAINER-RUNTIME
scos.lab.okd.lan   Ready    control-plane,master,worker   48d   v1.29.6+aba1e8d   192.168.122.111   <none>        CentOS Stream CoreOS 417.9.202407210037-0   5.14.0-480.el9.x86_64   cri-o://1.29.5
[root@okd-svc okd-scos]# oc get clusterversion -o json|jq ".items[0].status.history"
[
  {
    "acceptedRisks": "Target release version=\"\" image=\"registry.ci.openshift.org/origin/release-scos:4.16.0-0.okd-scos-2024-07-21-033952\" cannot be verified, but continuing anyway because the update was forced: release images that are not accessed via digest cannot be verified\nPrecondition \"ClusterVersionRecommendedUpdate\" failed because of \"UnknownUpdate\": RetrievedUpdates=True (), so the update from 4.15.0-0.okd-scos-2024-01-18-223523 to 4.16.0-0.okd-scos-2024-07-21-033952 is probably neither recommended nor supported.",
    "completionTime": "2024-07-23T13:32:21Z",
    "image": "registry.ci.openshift.org/origin/release-scos:4.16.0-0.okd-scos-2024-07-21-033952",
    "startedTime": "2024-07-23T12:24:24Z",
    "state": "Completed",
    "verified": false,
    "version": "4.16.0-0.okd-scos-2024-07-21-033952"
  },
  {
    "completionTime": "2024-07-05T09:17:50Z",
    "image": "quay.io/okd/scos-release:4.15.0-0.okd-scos-2024-01-18-223523",
    "startedTime": "2024-07-05T08:17:50Z",
    "state": "Completed",
    "verified": false,
    "version": "4.15.0-0.okd-scos-2024-01-18-223523"
  },
  {
    "completionTime": "2024-07-05T08:17:50Z",
    "image": "registry.ci.openshift.org/origin/release-scos:4.16.0-0.okd-scos-2024-07-04-010935",
    "startedTime": "2024-07-05T07:59:31Z",
    "state": "Partial",
    "verified": false,
    "version": "4.16.0-0.okd-scos-2024-07-04-010935"
  },
  {
    "acceptedRisks": "Target release version=\"\" image=\"registry.ci.openshift.org/origin/release-scos:4.15.0-0.okd-scos-2024-05-13-223609\" cannot be verified, but continuing anyway because the update was forced: release images that are not accessed via digest cannot be verified\nPrecondition \"ClusterVersionRecommendedUpdate\" failed because of \"UnknownUpdate\": RetrievedUpdates=True (), so the update from 4.15.0-0.okd-scos-2024-01-18-223523 to 4.15.0-0.okd-scos-2024-05-13-223609 is probably neither recommended nor supported.",
    "completionTime": "2024-07-05T07:59:31Z",
    "image": "registry.ci.openshift.org/origin/release-scos:4.15.0-0.okd-scos-2024-05-13-223609",
    "startedTime": "2024-07-05T07:08:19Z",
    "state": "Partial",
    "verified": false,
    "version": "4.15.0-0.okd-scos-2024-05-13-223609"
  },
  {
    "completionTime": "2024-06-19T11:34:31Z",
    "image": "quay.io/okd/scos-release:4.15.0-0.okd-scos-2024-01-18-223523",
    "startedTime": "2024-06-19T10:34:31Z",
    "state": "Completed",
    "verified": false,
    "version": "4.15.0-0.okd-scos-2024-01-18-223523"
  },
  {
    "completionTime": "2024-06-19T10:34:31Z",
    "image": "registry.ci.openshift.org/origin/release-scos:4.17.0-0.okd-scos-2024-06-18-175620",
    "startedTime": "2024-06-19T10:18:56Z",
    "state": "Partial",
    "verified": false,
    "version": "4.17.0-0.okd-scos-2024-06-18-175620"
  },
  {
    "completionTime": "2024-06-10T15:07:44Z",
    "image": "quay.io/okd/scos-release:4.15.0-0.okd-scos-2024-01-18-223523",
    "startedTime": "2024-06-10T14:21:44Z",
    "state": "Completed",
    "verified": false,
    "version": "4.15.0-0.okd-scos-2024-01-18-223523"
  },
  {
    "completionTime": "2024-06-10T14:21:44Z",
    "image": "quay.io/okd/scos-release:4.15.0-0.okd-scos-2024-01-11-032357",
    "startedTime": "2024-06-10T11:50:54Z",
    "state": "Partial",
    "verified": false,
    "version": "4.15.0-0.okd-scos-2024-01-11-032357"
  },
  {
    "completionTime": "2024-06-04T14:53:47Z",
    "image": "quay.io/openshift/okd@sha256:46b462be1e4c15ce5ab5fba97e713e8824bbb9f614ac5abe1be41fda916920cc",
    "startedTime": "2024-06-04T14:28:38Z",
    "state": "Completed",
    "verified": false,
    "version": "4.15.0-0.okd-2024-03-10-010116"
  }
]

[madpearl]

I've finished with 5 nodes libvirt bare-metal cluster following #1971 (comment)

[root@terraform-runner ~]# oc get nodes -owide
NAME                            STATUS   ROLES    AGE     VERSION           INTERNAL-IP       EXTERNAL-IP   OS-IMAGE                                    KERNEL-VERSION          CONTAINER-RUNTIME
libvirt-cp-1.libvirt.alma.lan   Ready    master   2y39d   v1.29.6+aba1e8d   192.168.122.101   <none>        CentOS Stream CoreOS 417.9.202407210037-0   5.14.0-480.el9.x86_64   cri-o://1.29.5
libvirt-cp-2.libvirt.alma.lan   Ready    master   2y39d   v1.29.6+aba1e8d   192.168.122.102   <none>        CentOS Stream CoreOS 417.9.202407210037-0   5.14.0-480.el9.x86_64   cri-o://1.29.5
libvirt-cp-3.libvirt.alma.lan   Ready    master   2y39d   v1.29.6+aba1e8d   192.168.122.103   <none>        CentOS Stream CoreOS 417.9.202407210037-0   5.14.0-480.el9.x86_64   cri-o://1.29.5
libvirt-w-1.libvirt.alma.lan    Ready    worker   2y39d   v1.29.6+aba1e8d   192.168.122.111   <none>        CentOS Stream CoreOS 417.9.202407210037-0   5.14.0-480.el9.x86_64   cri-o://1.29.5
libvirt-w-2.libvirt.alma.lan    Ready    worker   2y39d   v1.29.6+aba1e8d   192.168.122.112   <none>        CentOS Stream CoreOS 417.9.202407210037-0   5.14.0-480.el9.x86_64   cri-o://1.29.5
libvirt-w-3.libvirt.alma.lan    Ready    worker   2y39d   v1.29.6+aba1e8d   192.168.122.113   <none>        CentOS Stream CoreOS 417.9.202407210037-0   5.14.0-480.el9.x86_64   cri-o://1.29.5
libvirt-w-4.libvirt.alma.lan    Ready    worker   582d    v1.29.6+aba1e8d   192.168.122.114   <none>        CentOS Stream CoreOS 417.9.202407210037-0   5.14.0-480.el9.x86_64   cri-o://1.29.5
libvirt-w-5.libvirt.alma.lan    Ready    worker   582d    v1.29.6+aba1e8d   192.168.122.215   <none>        CentOS Stream CoreOS 417.9.202407210037-0   5.14.0-480.el9.x86_64   cri-o://1.29.5
[root@terraform-runner ~]# oc adm upgrade
Cluster version is 4.16.0-0.okd-scos-2024-07-24-025748

Upstream: https://amd64.origin.releases.ci.openshift.org/graph
Channel: stable-4
No updates available. You may still upgrade to a specific release image with --to-image or wait for new updates to be available.
[root@terraform-runner ~]# oc get clusterversion
NAME      VERSION                               AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.16.0-0.okd-scos-2024-07-24-025748   True        False         52m     Cluster version is 4.16.0-0.okd-scos-2024-07-24-025748
[root@terraform-runner ~]# oc get co
NAME                                       VERSION                               AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.16.0-0.okd-scos-2024-07-24-025748   True        False         False      6d2h
baremetal                                  4.16.0-0.okd-scos-2024-07-24-025748   True        False         False      2y39d
cloud-controller-manager                   4.16.0-0.okd-scos-2024-07-24-025748   True        False         False      2y39d
cloud-credential                           4.16.0-0.okd-scos-2024-07-24-025748   True        False         False      2y39d
cluster-autoscaler                         4.16.0-0.okd-scos-2024-07-24-025748   True        False         False      2y39d
config-operator                            4.16.0-0.okd-scos-2024-07-24-025748   True        False         False      2y39d
console                                    4.16.0-0.okd-scos-2024-07-24-025748   True        False         False      21d
control-plane-machine-set                  4.16.0-0.okd-scos-2024-07-24-025748   True        False         False      541d
csi-snapshot-controller                    4.16.0-0.okd-scos-2024-07-24-025748   True        False         False      25h
dns                                        4.16.0-0.okd-scos-2024-07-24-025748   True        False         False      617d
etcd                                       4.16.0-0.okd-scos-2024-07-24-025748   True        False         False      616d
image-registry                             4.16.0-0.okd-scos-2024-07-24-025748   True        False         False      9d
ingress                                    4.16.0-0.okd-scos-2024-07-24-025748   True        False         False      18m
insights                                   4.16.0-0.okd-scos-2024-07-24-025748   True        False         False      541d
kube-apiserver                             4.16.0-0.okd-scos-2024-07-24-025748   True        False         False      2y39d
kube-controller-manager                    4.16.0-0.okd-scos-2024-07-24-025748   True        False         False      2y39d
kube-scheduler                             4.16.0-0.okd-scos-2024-07-24-025748   True        False         False      2y39d
kube-storage-version-migrator              4.16.0-0.okd-scos-2024-07-24-025748   True        False         False      18m
machine-api                                4.16.0-0.okd-scos-2024-07-24-025748   True        False         False      2y39d
machine-approver                           4.16.0-0.okd-scos-2024-07-24-025748   True        False         False      2y39d
machine-config                             4.16.0-0.okd-scos-2024-07-24-025748   True        False         False      67d
marketplace                                4.16.0-0.okd-scos-2024-07-24-025748   True        False         False      2y39d
monitoring                                 4.16.0-0.okd-scos-2024-07-24-025748   True        False         False      176d
network                                    4.16.0-0.okd-scos-2024-07-24-025748   True        False         False      2y39d
node-tuning                                4.16.0-0.okd-scos-2024-07-24-025748   True        False         False      78m
openshift-apiserver                        4.16.0-0.okd-scos-2024-07-24-025748   True        False         False      24h
openshift-controller-manager               4.16.0-0.okd-scos-2024-07-24-025748   True        False         False      521d
openshift-samples                          4.16.0-0.okd-scos-2024-07-24-025748   True        False         False      103m
operator-lifecycle-manager                 4.16.0-0.okd-scos-2024-07-24-025748   True        False         False      2y39d
operator-lifecycle-manager-catalog         4.16.0-0.okd-scos-2024-07-24-025748   True        False         False      2y39d
operator-lifecycle-manager-packageserver   4.16.0-0.okd-scos-2024-07-24-025748   True        False         False      24h
service-ca                                 4.16.0-0.okd-scos-2024-07-24-025748   True        False         False      2y39d
storage                                    4.16.0-0.okd-scos-2024-07-24-025748   True        False         False      2y39d
[root@terraform-runner ~]#