Skip to content
This repository has been archived by the owner on Mar 26, 2024. It is now read-only.

Latest commit

 

History

History
36 lines (23 loc) · 1.53 KB

README.md

File metadata and controls

36 lines (23 loc) · 1.53 KB
Raw

common-kafka-connect

This project contains Kafka Connect component implementations.

The primary classes available for use are listed below.

Sources

No Kafka Connect Sources are currently provided.

Sinks

KafkaSinkConnector

  • A simple Kafka Connect Sink Connector that sends data to another Kafka cluster without any transformation or filtering.
  • Requires the use of the org.apache.kafka.connect.converters.ByteArrayConverter converter.
  • To run this connector in kafka-connect add the common-kafka and commmon-kafka-connect jars to the kafka-connect classpath.

Configuration

  • bootstrap.servers: The list of broker host/port for the kafka cluster to send data to (Required)
  • producer.*: Kafka producer configuration. To add a particular producer config prefix it with producer. (i.e. producer.acks)

By default the following is set to ensure everything is delivered,

  • acks: all
  • max.in.flight.requests.per.connection: 1
  • retries: 2147483647 (i.e. Integer.MAX_VALUE)

KafkaSinkTask

  • Sink Task implementation used by the KafkaSinkConnector.
  • Does not do any deserialization or serialization of data.
  • Sets producer configuration to ensure strong consistency.
  • Writes data to the same topic and partition in the destination Kafka cluster that it was read from the source Kafka cluster.