Cannot set blockOwnerDeletion on OpenShift 4.x

[rtsisyk]

Describe the bug

The operator doesn't work on OpenShift/OKD 4.x at all.

To Reproduce

oc apply -f examples/generic/operator.yaml
oc apply -f examples/generic/cluster.yaml

Logs

{"L":"INFO","T":"2020-06-18T04:59:02.534Z","M":"Operator started","version":"0.1.8","build_date":"2020-06-09T10:00:21Z","commit":"123be99a110ae03031cf580259c8cdf941740cad","built_by":"goreleaser","go_version":"go version go1.14.4 linux/amd64","options":{"Name":"scylla-operator-controller-manager-0","Namespace":"scylla-operator-system","LogLevel":"info","Image":"","EnableAdmissionWebhook":true},"_trace_id":"T8frN3XgSUiwodQ99k6fpQ"}
{"L":"INFO","T":"2020-06-18T04:59:03.836Z","M":"Registering Components.","_trace_id":"T8frN3XgSUiwodQ99k6fpQ"}
{"L":"INFO","T":"2020-06-18T04:59:05.336Z","M":"Starting the operator...","_trace_id":"T8frN3XgSUiwodQ99k6fpQ"}
{"L":"INFO","T":"2020-06-18T04:59:06.731Z","N":"cluster-controller","M":"Starting reconciliation...","cluster":"scylla/simple-cluster","resourceVersion":"7465258","_trace_id":"sEREqxMKTKCKngm-qb4DyQ"}
{"L":"ERROR","T":"2020-06-18T04:59:07.332Z","N":"cluster-controller","M":"An error occurred during cluster reconciliation","cluster":"scylla/simple-cluster","resourceVersion":"7465258","error":"failed to sync headless service: error syncing headless service simple-cluster-client: services \"simple-cluster-client\" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>","errorVerbose":"services \"simple-cluster-client\" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>\nerror syncing headless service simple-cluster-client\ngithub.com/scylladb/scylla-operator/pkg/controller/cluster.(*ClusterController).syncClusterHeadlessService\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/pkg/controller/cluster/service.go:24\ngithub.com/scylladb/scylla-operator/pkg/controller/cluster.(*ClusterController).sync\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/pkg/controller/cluster/sync.go:49\ngithub.com/scylladb/scylla-operator/pkg/controller/cluster.(*ClusterController).Reconcile\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/pkg/controller/cluster/cluster_controller.go:203\ngithub.com/scylladb/scylla-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215\ngithub.com/scylladb/scylla-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/scylladb/scylla-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/scylladb/scylla-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/scylladb/scylla-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/bin/deps/go/src/runtime/asm_amd64.s:1373\nfailed to sync headless service\ngithub.com/scylladb/scylla-operator/pkg/controller/cluster.(*ClusterController).sync\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/pkg/controller/cluster/sync.go:51\ngithub.com/scylladb/scylla-operator/pkg/controller/cluster.(*ClusterController).Reconcile\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/pkg/controller/cluster/cluster_controller.go:203\ngithub.com/scylladb/scylla-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215\ngithub.com/scylladb/scylla-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/scylladb/scylla-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/scylladb/scylla-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/scylladb/scylla-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/bin/deps/go/src/runtime/asm_amd64.s:1373","_trace_id":"lqlcl61ETMqfMeGjWswzXA","S":"github.com/scylladb/scylla-operator/vendor/github.com/scylladb/go-log.Logger.log\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/vendor/github.com/scylladb/go-log/logger.go:92\ngithub.com/scylladb/scylla-operator/vendor/github.com/scylladb/go-log.Logger.Error\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/vendor/github.com/scylladb/go-log/logger.go:75\ngithub.com/scylladb/scylla-operator/pkg/controller/cluster.(*ClusterController).Reconcile\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/pkg/controller/cluster/cluster_controller.go:204\ngithub.com/scylladb/scylla-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215\ngithub.com/scylladb/scylla-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/scylladb/scylla-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/scylladb/scylla-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/scylladb/scylla-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"L":"INFO","T":"2020-06-18T04:59:08.831Z","N":"cluster-controller","M":"Starting reconciliation...","cluster":"scylla/simple-cluster","resourceVersion":"7465258","_trace_id":"naw3L3MxQQubIBZrHO-iCA"}

Environment:

• Platform: OKD 4.4.0-0.okd-2020-05-23-055148-beta5
• Kubernetes version: v1.17.1
• Scylla version: 4.0.0
• Scylla-operator version: master ba89337

Additional context
Add any other context about the problem here.

[dahankzter]

Can you try with this change to the cluster yaml?

  - apiGroups:
       - "apps"
     resources:
       - statefulsets
+      - deployments/finalizers
     verbs:
       - get
       - list
       - patch
       - watch
+      - update
   - apiGroups:
       - scylla.scylladb.com

[rtsisyk]

No, it doesn't work:

diff --git a/examples/generic/operator.yaml b/examples/generic/operator.yaml
index cca9238..3453768 100644
--- a/examples/generic/operator.yaml
+++ b/examples/generic/operator.yaml
@@ -214,6 +214,7 @@ rules:
   - apps
   resources:
   - statefulsets
+  - deployments/finalizers
   verbs:
   - get
   - list
@@ -222,6 +223,7 @@ rules:
   - update
   - patch
   - delete
+  - update
 - apiGroups:
   - scylla.scylladb.com
   resources:

{"L":"INFO","T":"2020-06-18T12:31:54.987Z","M":"Operator started","version":"0.1.8","build_date":"2020-06-09T10:00:21Z","commit":"123be99a110ae03031cf580259c8cdf941740cad","built_by":"goreleaser","go_version":"go version go1.14.4 linux/amd64","options":{"Name":"scylla-operator-controller-manager-0","Namespace":"scylla-operator-system","LogLevel":"info","Image":"","EnableAdmissionWebhook":true},"_trace_id":"i18s51gSR_GnLKNHvSJAFA"}
{"L":"INFO","T":"2020-06-18T12:31:56.386Z","M":"Registering Components.","_trace_id":"i18s51gSR_GnLKNHvSJAFA"}
{"L":"INFO","T":"2020-06-18T12:31:57.888Z","M":"Starting the operator...","_trace_id":"i18s51gSR_GnLKNHvSJAFA"}
{"L":"INFO","T":"2020-06-18T12:31:58.497Z","N":"cluster-controller","M":"Starting reconciliation...","cluster":"scylla/simple-cluster","resourceVersion":"7465258","_trace_id":"BQtmUmgsSby4IWU7dqP_RA"}
{"L":"ERROR","T":"2020-06-18T12:31:58.985Z","N":"cluster-controller","M":"An error occurred during cluster reconciliation","cluster":"scylla/simple-cluster","resourceVersion":"7465258","error":"failed to sync headless service: error syncing headless service simple-cluster-client: services \"simple-cluster-client\" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>","errorVerbose":"services \"simple-cluster-client\" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>\nerror syncing headless service simple-cluster-client\ngithub.com/scylladb/scylla-operator/pkg/controller/cluster.(*ClusterController).syncClusterHeadlessService\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/pkg/controller/cluster/service.go:24\ngithub.com/scylladb/scylla-operator/pkg/controller/cluster.(*ClusterController).sync\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/pkg/controller/cluster/sync.go:49\ngithub.com/scylladb/scylla-operator/pkg/controller/cluster.(*ClusterController).Reconcile\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/pkg/controller/cluster/cluster_controller.go:203\ngithub.com/scylladb/scylla-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215\ngithub.com/scylladb/scylla-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/scylladb/scylla-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/scylladb/scylla-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/scylladb/scylla-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/bin/deps/go/src/runtime/asm_amd64.s:1373\nfailed to sync headless service\ngithub.com/scylladb/scylla-operator/pkg/controller/cluster.(*ClusterController).sync\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/pkg/controller/cluster/sync.go:51\ngithub.com/scylladb/scylla-operator/pkg/controller/cluster.(*ClusterController).Reconcile\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/pkg/controller/cluster/cluster_controller.go:203\ngithub.com/scylladb/scylla-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215\ngithub.com/scylladb/scylla-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/scylladb/scylla-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/scylladb/scylla-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/scylladb/scylla-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/bin/deps/go/src/runtime/asm_amd64.s:1373","_trace_id":"CIBGIY4zQ7aFhr9NXwxKhQ","S":"github.com/scylladb/scylla-operator/vendor/github.com/scylladb/go-log.Logger.log\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/vendor/github.com/scylladb/go-log/logger.go:92\ngithub.com/scylladb/scylla-operator/vendor/github.com/scylladb/go-log.Logger.Error\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/vendor/github.com/scylladb/go-log/logger.go:75\ngithub.com/scylladb/scylla-operator/pkg/controller/cluster.(*ClusterController).Reconcile\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/pkg/controller/cluster/cluster_controller.go:204\ngithub.com/scylladb/scylla-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215\ngithub.com/scylladb/scylla-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/scylladb/scylla-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/scylladb/scylla-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/scylladb/scylla-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}

[rtsisyk]

You can try CodeReady virtual machine (https://developers.redhat.com/products/codeready-workspaces/download) or OpenShift Online Trial (https://manage.openshift.com) for debug. Please ping mt if you need help with openshift/okd.

[rtsisyk]

Any ideas how to fix it?

[ezbz]

Continue tracking this issue in #424

[cgruver]

Need to set scyllaclusters/finalizers with update privilege on the cluster member ROLE.

More notes added to: #424