Skip to content

pablocasares/normalizer

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Normalizer

CircleCI Docs GitHub release wizzie-io

Normalizer is a stream processing engine based on Kafka Streams. Normalizer does maps (stateless and statefull), flatmaps and filters by us. You only need to define a JSON stream where you specify the process logic and how the message are transformed. It allows you to normalize data streams from different sources to convert them to the same data schema! Normalizer offers us: scalability, fault tolerance, back-pressure, KV states (RocksDB) and full Kafka integration ... Try it now!!

It reads json messages from Apache Kafka and writes them back normalized to Kafka.

Some of its abilities are:

  • Add fields.
  • Select or simplify fields.
  • Replace values.
  • Join two or more values.
  • Select max/min from an array of values.
  • Classify numeric values
  • Split one value into multiple dimensions
  • Convert time formats.
  • Convert field types.
  • Calculate arithmetic values
  • Rename fields.
  • Filtering
  • Transform one message into several
  • Easy scalable (automatic discovering and load balancing).
  • Check docs for more!

Documentation

You can find the docs on the Docs Website

Getting Started

You can get started with Normalizer with this tutorial.

Compiling Sources

To build this project you can use maven tool.

If you want to build the JAR of the project you can do:

mvn clean package

If you want to check if it passes all the test:

mvn test

If you want to build the distribution tar.gz:

mvn clean package -P dist

If you want to build the docker image, make sure that you have the docker service running:

mvn clean package -P docker

Contributing

  1. Fork it
  2. Create your feature branch: git checkout -b my-new-feature
  3. Commit your changes: git commit -am 'Add some feature'
  4. Push to the branch: git push origin my-new-feature
  5. Create a new Pull Request

About

Normalizer is a stream processing engine based of Kafka Streams.

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Java 99.5%
  • Other 0.5%