MCAD and InstaScale CRs are not deleted after removal of DataScienceCluster CR

[sutaakar]

Describe the bug
If you create DataScienceCluster CR with codeflare enabled and then delete it, the MCAD and InstaScale CRs are not deleted. I would expect them to be deleted too as DataScienceCluster should own them.

To Reproduce
Steps to reproduce the behavior:

1. Install OpenDataHub operator fast channel
2. Install CodeFlare operator
3. Create DataScienceCluster:

kind: DataScienceCluster
apiVersion: datasciencecluster.opendatahub.io/v1alpha1
metadata:
  name: default
  labels:
    app.kubernetes.io/name: datasciencecluster
    app.kubernetes.io/instance: default
    app.kubernetes.io/part-of: opendatahub-operator
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/created-by: opendatahub-operator
spec:
  components:
    codeflare:
      enabled: true
    dashboard:
      enabled: true
    datasciencepipelines:
      enabled: true
    kserve:
      enabled: false
    modelmeshserving:
      enabled: true
    ray:
      enabled: true
    workbenches:
      enabled: true

4. Wait until all pods in opendatahub namespace starts
5. Delete DataScienceCluster

Actual behavior
InstaScale and MCAD pods remain running. InstaScale and MCAD CRs in CodeFlare operator (created by DataScienceCluster) still exist.

Expected behavior
A clear and concise description of what you expected to happen.
All pods from opendatahub namespace are deleted, namespace is empty. InstaScale and MCAD CRs are deleted.

Screenshots

Additional context
Based on preliminary investigations the MCAD and InstaScale CRs are missing OwnerReference to DataScienceCluster.

[sutaakar]

@zdtsw FYI

[zdtsw]

To be clear for the scope of this issue, for the current code baseline, InstaScale only exists in ODH, not downstream.

[lugi0]

in the 1.33 RC1 build for RHODS, using the 1.33 RC1 build for Codeflare, I was not able to get the MCAD pod to be terminated after deleting the DSC CR (all other RHODS pods have been deleted).
Uninstalling the Codeflare operator or setting Codeflare to Removed in the DSC CR did not terminate the pod either.

As @zdtsw said, I was not able to find an InstaScale CR/Pod created after installing Codeflare and enabling the component.

[zdtsw]

re-open it and i will need to reproduce this tomorrow to see if it is related to the #550

[sutaakar]

Actually, this issue becomes obsolete. With the latest redesign (project-codeflare/codeflare-operator#216) the MCAD and InstaScale CRs are removed from CodeFlare.

[zdtsw]

Actually, this issue becomes obsolete. With the latest redesign (project-codeflare/codeflare-operator#216) the MCAD and InstaScale CRs are removed from CodeFlare.

shoud we close this issue and skip verification in ODH operator? @lugi0

[lugi0]

As long as we don't care about what behaviour it has in 1.33?
I seem to understand Codeflare won't be "officially" released until 1.34, but would like some confirmation of this.

[astefanutti]

It may be the logic introduced in #511 fail with the new version of the CodeFlare operator. I'd need to look at the exact error the controller-runtime cached client returns in case the CRD / API endpoint is not present. In any case #511 can be reverted.

[zdtsw]

if we all agree that we will not continue this issue, i can have it closed, and revert the change in #511
but then we have to sync down the "revert" to 1.33

[astefanutti]

@zdtsw sounds good to me.