Skip to content

Latest commit

 

History

History
97 lines (60 loc) · 2.95 KB

README.md

File metadata and controls

97 lines (60 loc) · 2.95 KB
Raw

Metamorphosis

A super dope, pluggable NodeJS Kafka Framework for building Kafka workers like consumers, producers and stream processors.

Requirements

  • Kafka cluster either BYO or run the install scripts below for a test cluster
  • Node 12.x
  • Yarn or NPM package managers (this project defaults to Yarn)
  • Docker (if testing without Kafka cluster)
  • Docker Compose

Install

yarn install

Config

Copy example .env and replace values with your own

cp .env.example .env
Option Description Default
KAFKA_BROKERS Comma-delimited string of brokers e.g. 10.10.10.1:9092,10.10.10.2:9092,... localhost:9092
KAFKA_CLIENT_ID Unique ID used by consumers and producers in this app metamorphosis
KAFKA_TOPIC_CONSUMER Topic to read from for a single-topic consumer app metamorphosis.test
KAFKA_TOPIC_PRODUCER Topic to write to for a single-topic producer app metamorphosis.test
CONSUMER_GROUP_ID Group name responsible for tracking offsets on a consumer. This should be unique for every consumer app. metamorphosis-consumer-group

Usage

See /examples folder for specific sample applications

Services

Allow for consistent setup processing of Kafka topics using common tools for repeated tasks.

Default Consumer

Pass a callback and perform work on each message in a single topic

Default Producer

Generate a client connection with the ability to write messages into a Kafka topic.

Webhook Producer

Define an endpoint to consume payloads and write to a Kafka topic.

Database Adapter

Pass a database adapter into any service for reading and writing to a database.

Testing

Install either the Kitchen Sink package or pick and choose from the items below.

A) Kitchen Sink

Install everything needed to test Kafka and all database adapters

yarn setup:all

B) Choose your components

You can use Kafka or database environments you are already running by setting them in the .env or the config/{environment}.js file

Kafka Cluster

Setup local Kafka cluster with Zookeper and SSL/SASL connection

yarn kafka:start

MySQL Database

Setup MySQL 8 database with sample schema and table

yarn mysql:start

Run Mocha Tests

Run all tests from root of project. You can adjust the timeout and other settings in .mocharc.json and test/config.js as well as the standard application config.

yarn test