update from 4.15.0-0.okd-scos-2024-01-18-223523 to 4.16.0-0.okd-scos-2024-08-21-155613 APIServicesAvailable: PreconditionNotReady

[JOECHONG0420]

I have try to upgrade from 4.15.0-0.okd-scos-2024-01-18-223523 to 4.16.0-0.okd-scos-2024-08-21-155613, the error show APIServicesAvailable: PreconditionNotReady. How can I fix this error to continues the upgrade. I also try the version 4.16.0-0.okd-scos-2024-09-24-151747, it also appear with the same error. I am following the scos-stable steam. If my upgrade path is wrong, can you suggest how can I upgrade to 4.16.0-0.okd-scos-2024-09-24-151747.

[GingerGeek]

Hey, thanks for getting in touch.

Are you able to get logs from openshift-apiserver pods and see what error states there are getting?

Could you let us know what method you have taken for the update?

We have some in progress documentation for coming from 4.15 FCOS into later streams in okd-project/okd-web!22

I don't think we've done much testing from the 4.15 SCOS (which confusingly is an entirely different build pipeline to 4.16+ SCOS) to 4.16 migration but using some of the workarounds in the above docs may still work.

Alternatively, you may need to pivot your cluster first to the latest version of FCOS 4.15 before then proceeding back into SCOS 4.16+

[JOECHONG0420]

openshift-apiserver-operator log have:
connectivity_check_controller.go:169] ConnectivityCheckController is waiting for transition to desired version ( 4.16.0-0.okd-scos-2024-08-21-155613) to be completed.

I have run the command to upgrade
oc adm upgrade --force --allow-explicit-upgrade --to-image=quay.io/okd/scos-release: 4.16.0-0.okd-scos-2024-08-21-155613

I also try to upgrade from FCOS 4.15.0-0.okd-2024-03-10-010116 to scos-release: 4.16.0-0.okd-scos-2024-08-21-155613 and 4.16.0-0.okd-scos-2024-09-24-151747. The same error appear.

I think the proper way is upgrade from FCOS 4.15 -> SCOS 4.15 -> SCOS 4.16

I am using the method in #1971 to upgrade my OKD.
My case is similar as #2036

[JaimeMagiera]

Hello. There isn't an OKD SCOS 4.15 release. There were some test builds very early this year, but nothing ever made it to production. Please note that while 4.16 SCOS had a few builds make it to the stable channel, they were not production-ready releases. We have not cut a final 4.16 SCOS release. You can read more here...

https://okd.io/docs/project/scos-migration-faq/

[devzeronull]

Hello. There isn't an OKD SCOS 4.15 release. There were some test builds very early this year, but nothing ever made it to production. Please note that while 4.16 SCOS had a few builds make it to the stable channel, they were not production-ready releases. We have not cut a final 4.16 SCOS release. You can read more here...

https://okd.io/docs/project/scos-migration-faq/

Hi and thank you for your engagement and explanation @JaimeMagiera, sadly it caused a bit of confusion :(

From what you pointed out, it seems that none of the OKD SCOS releases is yet "stable" and all as of now available SCOS releases are NOT intended for production use - neither pre nor post SCOS 4.15.0-0.okd-scos-2024-01-18-223523 which appeared in "4-scos-stable" as an accepted 4.15.z SCOS Release.

How does it align with the SCOS Migration FAQ question when OKD SCOS will be released, which is answered as:

.. We expect to release 4.16 in November, with 4.17 to follow shortly thereafter.

We are in November and we have accepted 4.16.z releases appearing in "4-scos-stable" graph which are NOT marked as "pre-release" on the releases page.

Sorry for the questions:

1. Is the latest 4.16 release (4.16.0-okd-scos.0) not "stable" and useable in production?
2. If there are accepted releases in "4-scos-stable" that are NOT stable and NOT for production use, how could someone ever identify a "stable" production-ready OKD SCOS release?

Not only now, also in the future, because in FCOS we were relying on the accepted releases in "4-stable" and published/marked on the corresponding releases page.
Also the July Blog post was to "encourage the community to test nightlies" and not picks in the stable, which led to the conclusion that "testing" will be done aside "stable".

It would be very kind if someone can explain which source to rely on and which can be a valid decision tree for choosing an OKD SCOS release for production use and how far we can trust accepted releases appearing in "4-scos-stable".

[JaimeMagiera]

Correct. There are no official releases of OKD SCOS 4.16. As mentioned, some builds have ended up in stable because they passed the limited CI testing, but should not be considered an actual release. This is due to our transition to a new process. We will post something in our various spaces when there is an official release. We're looking at the end of November.

In terms of all releases, we will likely update the releases in Github. Likewise, we will update the website with the release info. This is all getting fleshed out now. Thanks for your patience. This has been a monumental task for the handful of volunteers that make up OKD.

[devzeronull]

Thank you so much for taking your time to explain and summarize the situation and your efforts you are bringing to the community!

We really hope that someday the "stable" in "4-scos-stable" has a substantial meaning in terms of production-readyness we can rely on besides our own canary Installation/Upgrade tests!