Configure bootstrapServers for KafkaSink from ConfigMap field

[eshepelyuk]

Problem
Frequently kafka bootstrap servers are stored in ConfigMap field in every namespace, for various app to be able to access the same brokers uniformely. Then applications could refer the ConfigMap either mounting it via volumeMount or setup env variable via Downward API

Also a lot of applications supports bootstrap servers as a string separated by commas, like:

broker1:9092,broker2:9092,broker3:9092.

So the ConfigMap may look like

apiVersion: v1
data:
  kafka.brokers: broker1:9092,broker2:9092,broker2:9092
kind: ConfigMap
metadata:
  name: common-config

It could be great if KafkaSink could refer such ConfigMap, so the same settings can be leveraged between KafkaSink and other applications and there would be no need to configure bootstrap servers separately for KafkaSink instances.

New API may look like

apiVersion: eventing.knative.dev/v1alpha1
kind: KafkaSink
metadata:
  name: my-sink
spec:
  topic: my-sink
  bootstrapServers:
    ref:
      kind: ConfigMap
      apiVersion: v1
      name: common-config
      data: kafka.brokers

Persona:
KafkaSink users

Exit Criteria

1. Create ConfigMap with bootstrap servers
2. Create KafkaSink referring that ConfigMap
3. Make sure messages are produced to kafka topic

Time Estimate (optional):
Have no idea :(

[Cali0707]

Hey @eshepelyuk I like this idea, it does seem to make things easier for users.

Maybe for @matzew or @pierDipi or @creydr - if we decide to support this API change on KafkaSinks, does it make sense to move towards supporting it across all of our Knative Kafka components? I could see this having a few benefits:

1. Easier for end users - make the CM once, reference it from all your Knative Kafka components
2. Probably will let us simplify some code, and reuse logic to parse the kafka configs

I understand we currently support different secret formats and config stuff across different components, and we may need to continue to support those, so this may not replace those existing configs and instead add a newer, more consistent way to configure the Kafka pieces across all Knative Kafka components.

WDYT?

[pierDipi]

Would supporting the same Kafka broker config format (with fixed keys) work for you @eshepelyuk? https://knative.dev/docs/eventing/brokers/broker-types/kafka-broker/#create-a-kafka-broker

[eshepelyuk]

Would supporting the same Kafka broker config format (with fixed keys) work for you @eshepelyuk? https://knative.dev/docs/eventing/brokers/broker-types/kafka-broker/#create-a-kafka-broker

Can you provide an example how KafkaSink will look like ?

[pierDipi]

Like this @eshepelyuk

apiVersion: eventing.knative.dev/v1alpha1
kind: KafkaSink
metadata:
  name: my-sink
spec:
  topic: my-sink
  config:
    apiVersion: v1
    kind: ConfigMap
    name: kafka-sink-config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: kafka-sink-config
data:
  bootstrap.servers: "my-cluster-kafka-bootstrap.kafka:9092"

[eshepelyuk]

Yes, that would work.

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[eshepelyuk]

go away bot

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[pierDipi]

/triage accepted