Technical Documentation

Overview

The connector is a custom connector used to write Kafka topics to Elasticsearch indexes.

The connector reads messages from different Kafka topic and, for every message, creates a Map representing the object that has to be written on Elasticsearch. The original connector reads a stream of documents from Couchbase and publish them on Kafka using a normal producer.

Javadoc

TODO JAVADOC

Class ElasticsearchSinkConnector

The class inherits from SinkConnector and it's the responsible of creating the connector Tasks using the supplied properties.

The start(Map<String, String> props) method is called when the connector is started. The method simply parses and validates the supplied properties.

The method taskConfigs(int maxTasks) returns a set of configurationss for Tasks using the properties parsed by the start(...) method.f

Class ElasticsearchSinkTask

The class is an implementation of the SinkTask class and its job is to actually write documents on Elasticsearch.

The start(Map<String, String> props) method is called when a Task is started and its job is to create the connection to the Elasticsearch cluster.

The put(Collection<SinkRecord> sinkRecords) writes the documents on Elasticsearch. For every record in the list passed to the method, it converts the record into a Map representing the document that has to be written on Elasticsearch and then adds the document to a bulk request object. When the desired size to use for the bulk requests is reached, the request is executed. If the bulk request fails, the connector logs the errors.