Migrate kafka-operator to use listener-operator for exposing brokers

[nightkr]

Description

Fixes #714.

Blockers:

• ClusterResources::delete_orphaned_resources also deletes objects that are not owned directly by it operator-rs#861

Followup issues/regressions:

• Stop NodePort services from flapping when a replica comes on- or offline listener-operator#220
• Allow listenerclasses to specify whether to prefer IP or Hostname addresses listener-operator#139

Review Checklist

• Code contains useful comments
• CRD change approved (or not applicable)
• (Integration-)Test cases added (or not applicable)
• Documentation added (or not applicable)
• Changelog updated (or not applicable)
• Cargo.toml only contains references to git tags (not specific commits or branches)
• Helm chart can be installed and deployed operator works (or not applicable)

Once the review is done, comment bors r+ (or bors merge) to merge. Further information

[nightkr]

Blocked on stackabletech/listener-operator#1

[maltesander]

Tested and works!

[lfrancke]

I've moved this back into "Ready for Development" because the Listener operator is now merged. AFAIU this should now be ready to be worked on again?

[siegfriedweber]

I've moved this back into "Ready for Development" because the Listener operator is now merged. AFAIU this should now be ready to be worked on again?

This pull request is updated as requested by stackabletech/listener-operator#5. When stackabletech/operator-rs#496 is merged and operator-rs and the listener-operator are released then this pull request can be made ready for review.

[nightkr]

Tests currently fail due to stackabletech/listener-operator#110.

[nightkr]

The tests now pass!

[sbernauer]

@nightkr do you plan to merge this or add the WIP functionality of secret-op beforehand, so that the kafka broker certificates will include the listener addresses?
It'd be awesome if we get it right in the first run :)

[nightkr]

Preferably not, since we're still working out a few API kinks there (truststore passwords), and I'd rather keep this PR focused. ^^

[sbernauer]

@nightkr given you opened stackabletech/operator-rs#858 (reviewed it), I would be very in favor of putting the listenerVolume scope on the secret-op mount, so that we have valid certificates.
This would be great for a customer of us ;)

[nightkr]

Yes, that was why I opened that PR ^^

[nightkr]

Current (known) issues:

1. Need to cut an op-rs release
2. Only online brokers are listed in discovery ConfigMap when using a nodeport listenerclass
   • Brokers that are down will be missed by clients that don't update their ConfigMap regularly
   • Clients that restart on configmap change (like our CM restarter service) will be restarted whenever a broker goes down or up
3. Discovery ConfigMap when using a ClusterIP listenerclass only exposes K8s-managed LB endpoint.. means more resilience to cluster rescaling but clients are stuck using K8s' native load balancing rather than client side LB/failover.
4. Bootstrap listener is currently global for the whole cluster, so can't apply rolegroup specific config.

2 and 3 come from currently using a single shared bootstrap listener.. simpler for kafka-op and arguably where we want the UX to go, but means we'll probably want to fix it.

Arguably 3 is desirable, since this only affects initial bootstrap anyway.

[nightkr]

stackabletech/operator-rs#861 is also a pretty blocking issue for this.

[nightkr]

This isn't ready to merge yet (there are still open blockers), but the gist should be reviewable now.