Skip to content

A Lenses.io Kafka Connect Single Message Transform(SMT) collection

Notifications You must be signed in to change notification settings

davidsloan/kafka-connect-smt

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

53 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

A collection of Kafka Connect Single Message Transforms (SMTs)

These SMTs are designed to be used with the Kafka Connect framework. The SMTs create record headers. The advantage of using headers is that they reduce the memory and CPU cycles required to change the payload. See for example the Kafka Connect TimestampConverter. Furthermore, they support Stream-Reactor S3 sink partitioner, for scenarios like:

  • Partitioning by system clock (e.g. using the system clock as a partition key with a yyyy-MM-dd-HH format)
  • Partitioning by a rolling window (e.g. every 15 minutes, or one hour)
  • Partitioning by a custom timestamp (e.g. a timestamp field in the payload, record Key or Value)
  • Partitioning by a custom timestamp with a rolling window (e.g. a timestamp field in the payload, every 15 minutes, or one hour)

SMTs

  • InsertWallclock - Inserts the system clock as a message header.
  • InsertRollingWallclock - Inserts the system clock as a message header based on a rolling window boundary.
  • InsertRollingRecordTimestampHeaders - Inserts date, year, month, day, hour, minute, and second headers using the record timestamp and a rolling time window configuration.
  • InsertRollingWallclockHeaders - Inserts date, year, month, day, hour, minute, and second headers using the system timestamp and a rolling time window configuration.
  • InsertRecordTimestampHeaders - Inserts date, year, month, day, hour, minute, and second headers using the record timestamp.
  • InsertFieldTimestampHeaders - Inserts date, year, month, day, hour, minute, and second headers using a field in the payload, record Key or Value.
  • InsertRollingFieldTimestampHeaders - Inserts date, year, month, day, hour, minute, and second headers using a field in the payload, record Key or Value and a rolling window boundary.
  • InsertWallclockHeaders - Inserts date, year, month, day, hour, minute, and second headers using the system clock.
  • TimestampConverter - Converts a timestamp field in the payload, record Key or Value to a different format, and optionally applies a rolling window boundary. An adapted version of the one packed in the Kafka Connect framework.
  • InsertWallclockDateTimePart - Inserts the system clock year, month, day, minute, or seconds as a message header, with a value of type STRING.

Installation

The build jar can be found in the releases. To install the jar, copy it to the plugin.path directory of your Kafka Connect worker.

Build

To build the project run:

mvn clean package

Checkstyle

To check the code style run:

mvn checkstyle:check

To format the code run:

  mvn com.coveo:fmt-maven-plugin:format

To add license header, run:

mvn license:format

About

A Lenses.io Kafka Connect Single Message Transform(SMT) collection

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Java 100.0%