OpenShift - error validating existing CRs against new CRD's schema

[unknownsolo]

hello,

We had HPE CSI Driver 2.1.0 installed and we were showing an update for 2.2.0 in the Installed Operators page, after clicking update, I am seeing this error:

"error validating existing CRs against new CRD's schema for "hpecsidrivers.storage.hpe.com": error validating custom resource against new schema for HPECSIDriver hpe-csi-driver/csi-driver: [[].spec.csp: Required value, [].spec.controller: Required value, [].spec.node: Required value]"

[pavansshanbhag]

Hi,

It looks like the older operator exists and there is an attempt to install a newer version. Can you tell us how did you upgrade the operator ?

[pavansshanbhag]

The upgrade requires you to follow the instructions at the Scod Page :
https://scod.hpedev.io/partners/redhat_openshift/index.html#upgrading

After all "uninstall" steps in the instructions are completed and before installing the new operator, there is one additional step (not yet mentioned in scod) that you would need to do.
oc delete crd hpecsidrivers.storage.hpe.com

After this, you can proceed to install the latest v2.2.1 operator.

Let us know how it goes...thanks

[unknownsolo]

Can you tell us how did you upgrade the operator ?

Through the GUI in the Operators page, I simply clicked the upgrade button and got the errors above. The old operator is still there, I now have the old and the new one and both are errored out, but no impact to PVCs.

I can go ahead and uninstall the operator to solve this issue, but just to clarify, does that not impact my current running PODs with persistent storage accounts through this operator? Can I safely do that without ruining anything? The thought of "uninstall" is a little concerning.

[unknownsolo]

Just wanted to verify if uninstalling the operator is going to have any impact on my currently running pods.

[datamattsson]

Just wanted to verify if uninstalling the operator is going to have any impact on my currently running pods.

No. PVs, PVCs and StorageClasses remain unaffected while uninstalling and deleting the HPECSIDriver resources and Operator. In-flight CSI operations and new requests will be retried once the new CSI driver is instantiated.

[unknownsolo]

Thank you so much for the confirmation! One last question, how can I successfully get rid of HPECSIDriver? The contenxt menu in the GUI is grayed out

[unknownsolo]

I could really use some help here! This "simple" upgrade process is slowly turning into a nightmare! I tried reaching out to Red Hat too and they said to reach out to you.

I am not sure how to get rid of HPECSIDriver before uninstalling the operator! Everything is grayed out. The only thing I can do is to uninstall the operator itself.

[unknownsolo]

I finally got it! (What's a few heart attacks here and there?)

The HPECSIDriver CRD was not properly terminating so when I uninstalled the operator and installed again, the same HPECSIDriver was present. I manually removed the finalizers from HPECSIDriver and then it was deleted, uninstalled the operator, installed the operator again, created a new HPECSIDrive instance and everything worked! All while not impacting my current bound PVCs/PVs.

Thank you all for the help as I get more familiar with this upgrade process!

[datamattsson]

Thank you all for the help as I get more familiar with this upgrade process!

I apologize you had to go through this. We have not officially shipped yet and we have documentation that more clearly outlines the steps to "upgrade". It will be published next week and the page you're bookmarking is this one: https://scod.hpedev.io/partners/redhat_openshift/index.html

[unknownsolo]

I am just glad it's working now!

Honestly, the upgrade documentation could use some clarification. Maybe a note indicating that it is OK to "remove the driver and uninstall the operator, it will not impact any running pods with persistent storage. Clarify what is OK to be deleted and what is not! I identify all the important resources and how to get to them.

When I had a chat with Red Hat, they wanted me to delete the whole project and re-create - I said nope! Not knowing whether it was OK or not, the whole thought of "delete/uninstall" and recreate is concerning in general.

That said, I do have a supported OpenShift cluster and I'm happy to contribute if needed.

[joeltraber]

I'm still a little puzzled about how the official upgrade procedure works.
Why aren't the CRDs simply versioned cleanly so that an in-place upgrade works?

[datamattsson]

Why aren't the CRDs simply versioned cleanly so that an in-place upgrade works?

Because it's not a real "Operator". It's a Helm chart. A Helm chart packaged in an Operator as per Red Hat's guidelines on how to provide certified CSI drivers to OpenShift. Helm charts does not manage CRDs as you may know.

[joeltraber]

Yes got it - thanks for clarification! Good to know!
We may switch to the helm installation directly than instead of the "operator catalog" installation.

[datamattsson]

We may switch to the helm installation directly than instead of the "operator catalog" installation.

It's not supported by Red Hat. HPE supports it if you come to us with a CSI driver issues but you can't log cases with Red Hat. Here are the instructions: https://scod.hpedev.io/partners/redhat_openshift/index.html#unsupported_helm_chart_install