Cluster instanciation failure when trying to activate SSL and OIDC on openshift

[looping-aba]

Type of question

Implementation Assistance

Support question

Hello,
I succeed to instanciate a 2 node cluster on openshift, but when I tried to activate authentication and SSL :)
I'm looking for a workaround... and better a solution ;)

Context

• OCP
  I'm running on an Openshift Cluster
  Client Version: 4.13.25
  Kustomize Version: v4.5.7
  Server Version: 4.14.15
  Kubernetes Version: v1.27.10+c79e5e2

• Operator
  Operator is initiated using the following command:
  helm install nifikop . --version 1.10.0 --set image.tag=v1.10.0-release --set resources.requests.memory=256Mi --set resources.requests.cpu=250m --set resources.limits.memory=256Mi --set resources.limits.cpu=250m --debug --set namespaces={"dev01-nifi"} --namespace=dev01-nifi --set runAsUser=$uid

• Nifi Cluster
  The nifi image has been customized to include keycloak certificate in jvm cacert.
  Cluster is instanciated in the same namespace as the operator => namespace dev01-nifi
  I tried to find solution in previous post and I changed the nodeUserIdentityTemplate.

---
apiVersion: nifi.konpyutaika.com/v1
kind: NifiCluster
metadata:
  name: nifi
spec:
  service:
    headlessEnabled: true
    labels:
      cluster-name: nifi
  zkAddress: "<ZOOKEEPER_URL>:2181"
  zkPath: /nifi
  clusterImage: "<HARBOR_URL>/data-fabrique/ucn-nifi:v0.7"
  initContainerImage: 'bash:5.2.2'
  oneNifiNodePerNode: true
  nodeUserIdentityTemplate: "n-%d"
  readOnlyConfig:
    nifiProperties:
      overrideConfigs: |
        nifi.sensitive.props.key=thisIsABadSensitiveKeyPassword
        nifi.security.user.oidc.discovery.url=https://<URL_KEYCLOAK>/.well-known/openid-configuration
        nifi.security.user.oidc.client.id=nifi
        nifi.security.user.oidc.client.secret=<CLIENT_SECRET>
        nifi.security.identity.mapping.pattern.dn=CN=([^,]*)(?:, (?:O|OU)=.*)?
        nifi.security.identity.mapping.value.dn=$1
        nifi.security.identity.mapping.transform.dn=NONE
    bootstrapProperties:
      overrideConfigs: |
        java.arg.2=-Xms2g
        java.arg.3=-Xmx6g
  pod:
    labels:
      cluster-name: nifi
  nodeConfigGroups:
    default_group:
      runAsUser: 1000810000 # set an uid in your namespace range
      fsGroup: 1000810000 # set an gid in your namespace range
      imagePullPolicy: IfNotPresent
      isNode: true
      serviceAccountName: default
      externalVolumeConfigs:
        - name: krb5-config
          mountPath: "/opt/nifi/nifi-current/kerberos"
          configMap:
            name: "krb5-config"
        - name: nifi-keytab
          mountPath: "/opt/nifi/nifi-current/keytabs"
          secret:
            secretName: "nifi-keytab"
      storageConfigs:
        - mountPath: "/opt/nifi/nifi-current/logs"
          name: logs
          pvcSpec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 1Gi
      resourcesRequirements:
        limits:
          cpu: "6"
          memory: 16Gi
        requests:
          cpu: "6"
          memory: 16Gi
  nodes:
    - id: 1
      nodeConfigGroup: "default_group"
    - id: 2
      nodeConfigGroup: "default_group"
  propagateLabels: true
  nifiClusterTaskSpec:
    retryDurationMinutes: 10
  listenersConfig:
    internalListeners:
      - containerPort: 8443
        type: https
        name: https
      - containerPort: 6007
        type: cluster
        name: cluster
      - containerPort: 10000
        type: s2s
        name: s2s
      - containerPort: 9090
        type: prometheus
        name: prometheus
    sslSecrets:
      tlsSecretName: "nifi-secure-test"
      create: true

Behaviour
The cluster instanciation fail and no nifi pod is created.

In the certmanager namespace i can see that logs and it seems that cert and issuer are correctly created.

I can see cert in dev01-nifi namespace
nifikop-webhook-cert
nifi-ca-certificate

I can found issuer in dev01-nifi namespace
nifi-issuer
nifi-self-signer
selfsigned-issuer

I can find secrets in dev01-nifi namespace
nifi-1-server-certificate
nifi-2-server-certificate
nifi-ca-certificate
nifi-controller

Logs and Error

• Certmanager
  Everything songs good ...

I0907 18:00:25.751260       1 controller.go:162] "cert-manager/certificates-readiness: re-queuing item due to optimistic locking on resource" key="dev01-nifi/nifi-ca-certificate" error="Operation cannot be fulfilled on certificates.cert-manager.io \"nifi-ca-certificate\": the object has been modified; please apply your changes to the latest version and try again"
I0907 18:00:25.752202       1 conditions.go:192] Found status change for Certificate "nifi-ca-certificate" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2024-09-07 18:00:25.752192024 +0000 UTC m=+696851.285804354
I0907 18:00:25.778058       1 controller.go:162] "cert-manager/certificates-issuing: re-queuing item due to optimistic locking on resource" key="dev01-nifi/nifi-ca-certificate" error="Operation cannot be fulfilled on certificates.cert-manager.io \"nifi-ca-certificate\": the object has been modified; please apply your changes to the latest version and try again"
I0907 18:00:25.797666       1 controller.go:162] "cert-manager/certificates-key-manager: re-queuing item due to optimistic locking on resource" key="dev01-nifi/nifi-ca-certificate" error="Operation cannot be fulfilled on certificates.cert-manager.io \"nifi-ca-certificate\": the object has been modified; please apply your changes to the latest version and try again"
I0907 18:00:30.257663       1 conditions.go:85] Found status change for Issuer "nifi-issuer" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2024-09-07 18:00:30.257649778 +0000 UTC m=+696855.791262085

• Operator

I have several error regarding the fact client certificate cannont be generated

{"level":"info","time":"2024-09-07T18:00:24.964Z","logger":"controller.NifiCluster","caller":"certmanagerpki/certmanager_pki.go:83","msg":"Reconciling cert-manager PKI","clusterName":"nifi"}
{"level":"info","time":"2024-09-07T18:00:25.285Z","logger":"controller.NifiUser","caller":"controller/controller_common.go:35","msg":"failed to reconcile secret for user nifi-controller"}
{"level":"error","time":"2024-09-07T18:00:25.285Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nifiuser","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiUser","nifiUser":{"name":"nifi-controller","namespace":"dev01-nifi"},"namespace":"dev01-nifi","name":"nifi-controller","reconcileID":"8ed91546-8fe3-4013-90bc-49e1cf197af0","error":"**_could not create user certificate: certificates.cert-manager.io \"nifi-controller\" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on:_** , <nil>","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"info","time":"2024-09-07T18:00:25.289Z","logger":"controller.NifiCluster","caller":"k8sutil/resource.go:51","msg":"resource created","name":"nifi-headless","namespace":"dev01-nifi","kind":""}
{"level":"info","time":"2024-09-07T18:00:25.289Z","logger":"controller.NifiCluster","caller":"controller/nificluster_controller.go:156","msg":"A new resource was not found or may not be ready","reason":"server secret not ready: Secret \"nifi-1-server-certificate\" not found"}
{"level":"info","time":"2024-09-07T18:00:25.290Z","logger":"controller.NifiCluster","caller":"controller/nificluster_controller.go:134","msg":"NifiCluster starting reconciliation","clusterName":"nifi"}
{"level":"info","time":"2024-09-07T18:00:25.290Z","logger":"controller.NifiCluster","caller":"certmanagerpki/certmanager_pki.go:83","msg":"Reconciling cert-manager PKI","clusterName":"nifi"}
{"level":"info","time":"2024-09-07T18:00:25.348Z","logger":"controller.NifiUser","caller":"controller/controller_common.go:35","msg":"failed to reconcile secret for user n-1"}
{"level":"error","time":"2024-09-07T18:00:25.348Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nifiuser","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiUser","nifiUser":{"name":"n-1","namespace":"dev01-nifi"},"namespace":"dev01-nifi","name":"n-1","reconcileID":"283ff513-0221-43c0-926f-2f73c70cb546","error":"could not create user certificate: certificates.cert-manager.io \"n-1\" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"info","time":"2024-09-07T18:00:25.449Z","logger":"controller.NifiUser","caller":"controller/controller_common.go:35","msg":"failed to reconcile secret for user n-2"}
{"level":"error","time":"2024-09-07T18:00:25.449Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nifiuser","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiUser","nifiUser":{"name":"n-2","namespace":"dev01-nifi"},"namespace":"dev01-nifi","name":"n-2","reconcileID":"e21d688f-30a7-4bc0-b225-0047e891a871","error":"could not create user certificate: certificates.cert-manager.io \"n-2\" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}

And After I can see certificate decoding error

{"level":"error","time":"2024-09-07T18:00:30.649Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nificluster","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiCluster","nifiCluster":{"name":"nifi","namespace":"dev01-nifi"},"namespace":"dev01-nifi","name":"nifi","reconcileID":"cf2c365e-4f29-4942-8139-06b8fa3936ef","error":"**_failed to decode certificate: failed to decode x509 certificate_** from PEM","errorVerbose":"failed to decode x509 certificate from PEM\nfailed to decode certificate\ngithub.com/konpyutaika/nifikop/pkg/resources/nifi.(*Reconciler).getServerAndClientDetails\n\t/workspace/pkg/resources/nifi/nifi.go:483\ngithub.com/konpyutaika/nifikop/pkg/resources/nifi.(*Reconciler).Reconcile\n\t/workspace/pkg/resources/nifi/nifi.go:139\ngithub.com/konpyutaika/nifikop/internal/controller.(*NifiClusterReconciler).Reconcile\n\t/workspace/internal/controller/nificluster_controller.go:146\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"info","time":"2024-09-07T18:00:30.755Z","logger":"controller.NifiUser","caller":"controller/controller_common.go:35","msg":"failed to reconcile secret for user nifi-controller"}
{"level":"error","time":"2024-09-07T18:00:30.755Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nifiuser","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiUser","nifiUser":{"name":"nifi-controller","namespace":"dev01-nifi"},"namespace":"dev01-nifi","name":"nifi-controller","reconcileID":"c594b1f8-fbe1-42d5-9b12-6f7291450248","error":"could not create secret with jks password: secrets \"nifi-controller\" already exists","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"info","time":"2024-09-07T18:00:30.796Z","logger":"controller.NifiUser","caller":"controller/controller_common.go:35","msg":"failed to reconcile secret for user n-1"}
{"level":"error","time":"2024-09-07T18:00:30.796Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nifiuser","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiUser","nifiUser":{"name":"n-1","namespace":"dev01-nifi"},"namespace":"dev01-nifi","name":"n-1","reconcileID":"1aa1d201-a684-4385-bfa5-1c9ef3cd5af8","error":"could not create secret with jks password: secrets \"nifi-1-server-certificate\" already exists","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"info","time":"2024-09-07T18:00:30.805Z","logger":"controller.NifiUser","caller":"controller/controller_common.go:35","msg":"failed to reconcile secret for user n-2"}
{"level":"error","time":"2024-09-07T18:00:30.805Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nifiuser","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiUser","nifiUser":{"name":"n-2","namespace":"dev01-nifi"},"namespace":"dev01-nifi","name":"n-2","reconcileID":"d9b14cdf-581b-49ce-b635-0ea6699ea018","error":"could not create secret with jks password: secrets \"nifi-2-server-certificate\" already exists","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"info","time":"2024-09-07T18:00:35.770Z","logger":"controller.NifiCluster","caller":"controller/nificluster_controller.go:134","msg":"NifiCluster starting reconciliation","clusterName":"nifi"}
{"level":"info","time":"2024-09-07T18:00:35.770Z","logger":"controller.NifiCluster","caller":"certmanagerpki/certmanager_pki.go:83","msg":"Reconciling cert-manager PKI","clusterName":"nifi"}
{"level":"info","time":"2024-09-07T18:00:35.773Z","logger":"controller.NifiCluster","caller":"controller/controller_common.go:35","msg":"failed to decode certificate: failed to decode x509 certificate from PEM"}
{"level":"error","time":"2024-09-07T18:00:35.773Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nificluster","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiCluster","nifiCluster":{"name":"nifi","namespace":"dev01-nifi"},"namespace":"dev01-nifi","name":"nifi","reconcileID":"b832964d-1306-4685-9711-c39a3872a14f","error":"failed to decode certificate: failed to decode x509 certificate from PEM","errorVerbose":"failed to decode x509 certificate from PEM\nfailed to decode certificate\ngithub.com/konpyutaika/nifikop/pkg/resources/nifi.(*Reconciler).getServerAndClientDetails\n\t/workspace/pkg/resources/nifi/nifi.go:483\ngithub.com/konpyutaika/nifikop/pkg/resources/nifi.(*Reconciler).Reconcile\n\t/workspace/pkg/resources/nifi/nifi.go:139\ngithub.com/konpyutaika/nifikop/internal/controller.(*NifiClusterReconciler).Reconcile\n\t/workspace/internal/controller/nificluster_controller.go:146\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"info","time":"2024-09-07T18:00:35.894Z","logger":"controller.NifiUser","caller":"controller/controller_common.go:35","msg":"failed to reconcile secret for user nifi-controller"}

NiFiKop version

No response

Golang version

No response

Kubernetes version

Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.1", GitCommit:"c7c6eb21da5c5b9f813ea09a21aa3e7226206993", GitTreeState:"clean", BuildDate:"2023-11-21T17:49:49Z", GoVersion:"go1.19.13 X:strictfipsruntime", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.10+c79e5e2", GitCommit:"c725f2ce5164bf4165b22d6c28dd0ace4b3b7e9b", GitTreeState:"clean", BuildDate:"2024-02-21T18:19:42Z", GoVersion:"go1.20.12 X:strictfipsruntime", Compiler:"gc", Platform:"linux/amd64"}

NiFi version

1.21 => Image de base de l'image utilisée

[forzamehlano]

We had to add the finalizers permission to the role to get nifikop to work on openshift (well OKD 4.15) when deploying via helm.

Specifically adding additional lines for each of the objects beneath

nifikop/helm/nifikop/templates/role.yaml

Line 82 in 963e301

resources:

(nifiusers/finalizers, nificlusters/finalizers and so on)