Skip to content

Latest commit

 

History

History
125 lines (90 loc) · 5.76 KB

README.md

File metadata and controls

125 lines (90 loc) · 5.76 KB
Raw

warp-pipe

NOTE: NOT FOR PRODUCTION USE, THIS IS A WORK IN PROGRESS

warp-pipe is a simple CDC tool for Postgres.

How does it work

warp-pipe supports two modes, depending on the version and configuration of Postgres that you are running:

Logical Replication (LR)

Requirements

  • Postgres >= 9.5
  • wal2json
  • Logical replication enabled (via postgresql.conf)

You need to set up at least two parameters at postgresql.conf:

// postgresql.conf (changes require restart)
wal_level = logical
max_replication_slots = 1

In LR mode, warp-pipe will connect to a replication slot on your database using the wal2json output plugin, and emit Changesets via channel.

NOTE: You must set the appropriate REPLICA IDENTITY on your tables if you wish to expose old values in changesets. To learn more, see replica identity.

Audit

Requirements

  • Postgres >= 9.4

In audit mode, warp-pipe creates a new schema (warp_pipe) with a changesets tables in your database to track modifications on your schema's tables. A trigger is registered with all configured tables to notify (via NOTIFY/LISTEN) when there are new changes to be read.

Installation

Install the warp-pipe library with:

go get github.com/perangel/warp-pipe

Install the library and daemon with:

go get github.com/perangel/warp-pipe/...

Demo

The demo shows how data in a Source DB can be accessed as a stream and used to sync to a Target DB.

  1. Creates two databases in Docker.
  2. Sets up a schema on the Source DB.
  3. Sets up a schema on the Target DB.
  4. Runs warp-pipe setup-db on the Source DB.
  5. Pauses to allow the user to load a front-end web application which loads Warp Pipe Changesets via websockets.
  6. Adds data to the Source DB.
  7. Runs axon to sync data from the Source to Target.

Run:

make demo
make demo-clean

Usage

Run a warp-pipe and stream changes from a Postgres database.

Usage:
  warp-pipe [flags]
  warp-pipe [command]

Available Commands:
  help        Help about any command
  setup-db    Setup the source database
  teardown-db Teardown the `warp_pipe` schema

Flags:
      --start-from-lsn int         stream all changes starting from the provided LSN (default -1)
      --start-from-id int          stream all changes starting from the provided changeset ID (default -1)
      --start-from-ts int          stream all changes starting from the provided timestamp (default -1)
  -M, --replication-mode string    replication mode (default "lr")
  -i, --ignore-tables strings      tables to ignore during replication
  -w, --whitelist-tables strings   tables to include during replication
  -H, --db-host string             database host
  -d, --db-name string             database name
  -P, --db-pass string             database password
  -p, --db-port int                database port
  -U, --db-user string             database user
  -L, --log-level string           log level (default "info")
  -h, --help                       help for warp-pipe

Use "warp-pipe [command] --help" for more information about a command.

Configuration

Flag Environment Variable Description Mode
--start-from-lsn START_FROM_LSN Sets the logical sequence number from which to start logical replication lr
--start-from-id START_FROM_ID Sets the changeset ID from which to start relaying changesets audit
--start-from-ts START_FROM_TIMESTAMP Sets the timestamp from which to start replaying changesets audit
-M, --replication-mode REPLICATION_MODE Sets the replication mode to one of audit or lr (logical replication) (see: requirements) *
-i, --ignore-tables IGNORE_TABLES Specify tables to exclude from replication. *
-w, --whitelist-tables WHITELIST_TABLES Specify tables to include during replication. *
-H, --db-host DB_HOST The database host. *
-d, --db-name DB_NAME The database name. *
-P, --db-pass DB_PASS The database password. *
-p, --db-port DB_PORT The database port. *
-U, --db-user DB_USER The database user. *
-L, --log-level LOG_LEVEL Sets the logging level *

Additional Reading