feat: add Kerberos authentication for Kafka

[adwk67]

Fixes #655
Pending decision: https://github.com/stackabletech/decisions/issues/28 ( ✔️ closed/done)

Openshift/OKD tests 🟢 :

--- PASS: kuttl (520.16s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/configuration_kafka-latest-3.7.1_zookeeper-latest-3.9.2_openshift-true (41.05s)
        --- PASS: kuttl/harness/kerberos_kafka-3.8.0_zookeeper-latest-3.9.2_openshift-true_krb5-1.21.1_kerberos-realm-PROD.MYCORP_kerberos-backend-mit (100.10s)
        --- PASS: kuttl/harness/delete-rolegroup_kafka-3.8.0_zookeeper-latest-3.9.2_openshift-true (71.71s)
        --- PASS: kuttl/harness/smoke_kafka-3.8.0_zookeeper-3.9.2_use-client-tls-true_openshift-true (107.86s)
        --- PASS: kuttl/harness/logging_kafka-3.8.0_zookeeper-latest-3.9.2_openshift-true (117.43s)
        --- PASS: kuttl/harness/cluster-operation_kafka-latest-3.7.1_zookeeper-latest-3.9.2_openshift-true (130.04s)
        --- PASS: kuttl/harness/tls_kafka-3.8.0_zookeeper-latest-3.9.2_use-client-tls-true_use-client-auth-tls-true_openshift-true (164.68s)
        --- PASS: kuttl/harness/upgrade_zookeeper-3.9.2_upgrade_old-3.7.1_upgrade_new-3.8.0_use-client-tls-true_use-client-auth-tls-true_openshift-true (151.38s)
PASS

Definition of Done Checklist

• Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
• Please make sure all these things are done and tick the boxes

Author

• Changes are OpenShift compatible
• CRD changes approved. Awaiting feat: add kerberos authentication provider operator-rs#880.
• CRD documentation for all fields, following the style guide.
• Helm chart can be installed and deployed operator works
• Integration tests passed (for non trivial changes)
• Changes need to be "offline" compatible

Reviewer

• Code contains useful comments
• Code contains useful logging statements
• (Integration-)Test cases added
• Documentation added or updated. Follows the style guide.
• Changelog updated
• Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

• Feature Tracker has been updated
• Proper release label has been added
• Roadmap has been updated

[adwk67]

@sbernauer I tried to use one principal for all brokers, as we discussed, but could not quite get it to work.
This variation works with the brokers (they all come up):
principal=\"kafka/test-kafka-broker-default.kuttl-test-glorious-airedale.svc.cluster.local@$KERBEROS_REALM\"
but I was not able to do the same for the client job. I set
BROKER=test-kafka-broker-default.$NAMESPACE.svc.cluster.local:9093
which allows me to query the topics but it fails with authentication errors when in producer/consumer mode.

[sbernauer]

Only did a quick review of the code. I can go into more detail when we merged the changes from #443 and I find the proper time

[adwk67]

Ran nightly suite again locally following latest changes 🟢

--- PASS: kuttl (884.94s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/delete-rolegroup_kafka-3.8.0_zookeeper-latest-3.9.2_openshift-false (73.35s)
        --- PASS: kuttl/harness/tls_kafka-3.8.0_zookeeper-latest-3.9.2_use-client-tls-true_use-client-auth-tls-false_openshift-false (177.87s)
        --- PASS: kuttl/harness/upgrade_zookeeper-3.9.2_upgrade_old-3.7.1_upgrade_new-3.8.0_use-client-tls-true_use-client-auth-tls-false_openshift-false (178.71s)
        --- PASS: kuttl/harness/kerberos_kafka-3.8.0_zookeeper-latest-3.9.2_openshift-false_krb5-1.21.1_kerberos-realm-PROD.MYCORP_kerberos-backend-mit (95.47s)
        --- PASS: kuttl/harness/kerberos_kafka-3.8.0_zookeeper-latest-3.9.2_openshift-false_krb5-1.21.1_kerberos-realm-CLUSTER.LOCAL_kerberos-backend-mit (105.52s)
        --- PASS: kuttl/harness/upgrade_zookeeper-3.9.2_upgrade_old-3.7.1_upgrade_new-3.8.0_use-client-tls-true_use-client-auth-tls-true_openshift-false (125.30s)
        --- PASS: kuttl/harness/smoke_kafka-3.8.0_zookeeper-3.9.2_use-client-tls-true_openshift-false (141.44s)
        --- PASS: kuttl/harness/tls_kafka-3.8.0_zookeeper-latest-3.9.2_use-client-tls-false_use-client-auth-tls-true_openshift-false (148.59s)
        --- PASS: kuttl/harness/tls_kafka-3.8.0_zookeeper-latest-3.9.2_use-client-tls-false_use-client-auth-tls-false_openshift-false (65.18s)
        --- PASS: kuttl/harness/logging_kafka-3.8.0_zookeeper-latest-3.9.2_openshift-false (91.36s)
        --- PASS: kuttl/harness/cluster-operation_kafka-latest-3.7.1_zookeeper-latest-3.9.2_openshift-false (88.81s)
        --- PASS: kuttl/harness/configuration_kafka-latest-3.7.1_zookeeper-latest-3.9.2_openshift-false (28.44s)
        --- PASS: kuttl/harness/smoke_kafka-3.8.0_zookeeper-3.9.2_use-client-tls-false_openshift-false (58.76s)
        --- PASS: kuttl/harness/upgrade_zookeeper-3.9.2_upgrade_old-3.7.1_upgrade_new-3.8.0_use-client-tls-false_use-client-auth-tls-false_openshift-false (120.38s)
        --- PASS: kuttl/harness/tls_kafka-3.8.0_zookeeper-latest-3.9.2_use-client-tls-true_use-client-auth-tls-true_openshift-false (160.60s)
        --- PASS: kuttl/harness/upgrade_zookeeper-3.9.2_upgrade_old-3.7.1_upgrade_new-3.8.0_use-client-tls-false_use-client-auth-tls-true_openshift-false (82.92s)

[adwk67]

Re-tested 🟢

[siegfriedweber]

LGTM

[siegfriedweber]

Good explanation!

All changes implemented and/or discussed in planning (will merge as experimental).

[adwk67]

Re-testing before merge:

• AKS 🟢 https://testing.stackable.tech/view/02%20Operator%20Tests%20(custom)/job/kafka-operator-it-custom/12/
• OKD 🟢 https://testing.stackable.tech/view/02%20Operator%20Tests%20(custom)/job/kafka-operator-it-custom/13/