Is protobuf supported?

[danifullview]

Hi,

We want to consume a kafka topic where the messages are serialized using protobuf.

When running the workers (in k8s) we did set the following env vars:

  - name: CONNECT_KEY_CONVERTER
    value: org.apache.kafka.connect.storage.StringConverter
  - name: CONNECT_VALUE_CONVERTER
    value: io.confluent.connect.protobuf.ProtobufConverter
  - name: CONNECT_CONSUMER_VALUE_CONVERTER
    value: io.confluent.connect.protobuf.ProtobufConverter
  - name: CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL
    value: http://localhost:8081
  - name: CONNECT_CONSUMER_VALUE_CONVERTER_SCHEMA_REGISTRY_URL
    value: http://localhost:8081

but we're getting this error:

ERROR Error encountered in task ClickHouseProtoSinkConnector-0. Executing stage 'VALUE_CONVERTER' with class 'org.apache.kafka.connect.json.JsonConverter'. (org.apache.kafka.connect.runtime.errors.LogReporter)
org.apache.kafka.connect.errors.DataException: Converting byte[] to Kafka Connect data failed due to serialization error: 
at org.apache.kafka.connect.json.JsonConverter.toConnectData(JsonConverter.java:326)

Thank you.

[AndreiLucaci]

It seems the right configuration, as this is specified here
https://www.confluent.io/blog/kafka-connect-deep-dive-converters-serialization-explained/#finding-kafka-connect-configuration-file

[Paultagoras]

Hi @danifullview - are you folks using Confluent Cloud (and a local instance of Kafka Connect)?

If you're using their cloud environment, CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL should be set to the actual instance of Schema Registry (if you go to "CLI and Tools" -> "Kafka Connect" they have a section about Schema registry)

You would also need to specify CONNECT_CONSUMER_BASIC_AUTH_CREDENTIALS_SOURCE and CONNECT_CONSUMER_BASIC_AUTH_USER_INFO (it should give you all that info on the page)

[danifullview]

Hi @Paultagoras,

We're using Confluent Cloud an we're running the Kafka Connect workers on our k8s cluster.

We realised that we need to pass value.converter, value.converter.schemas.enable value.converter.basic.auth.credentials.source, value.converter.schema.registry.basic.auth.user.info and value.converter.schema.registry.url configs when creating the ClickHouse sink Connector.

Thank you.