Chain Replication

Development Status: Experimental

Chain replication is a replication protocol designed for high throughput, fault tolerant systems to achieve both fast replication as well as, in some implementations, strong consistency. A chain is able to recover quickly from failure by bypassing failed nodes in the chain, and new nodes can be added with minimal disruption.

This implementation is an experiment to build chain replication with Rust and Tokio. The goal is to build an underlying primitive on which specific systems can be built (e.g. key value store, search engine, state machine replication).

Building/Running

Must be using Rust nightly.

cargo build --release

# start the management server
./target/release/management-server config/management.toml &

# start a head node
./target/release/storage-server config/head.toml &

# start a replica node
./target/release/storage-server config/middle.toml &

# start a tail node
./target/release/storage-serer config/tail.toml &

# start the CLI
./target/release/cli

The benchit tool allows a quick way to benchmark log insertion using multiple clients.

# Start 2 clients, 50 requests in flight per client
./target/release/benchit -c 2 -r 50

Progress

• Chain Replication
  • Replicated log
  • Chained replication
  • Multiplexed command protocol
  • Replication protocol
  • Tail replies
  • Tail queries
• Reconfiguration
  • Modes
    • Head node failure
    • Tail node failure
    • Middle node failure
  • Master Node
    • Failure detector
    • Reconfiguration Protocol
    • Partitioning
    • Chain reconfiguration
    • Backup Master Nodes (Requires consensus protocol)
• Framework Elements
  • Custom Commands
  • Tail node queries
• Optimizations
  • Zero-Copy Transfer
• Other ideas
  • Kubernetes Operator
  • Complex chains
    • Replex
    • HyperDex
    • Multiple listeners
  • Multi-Data Center topology
  • Integration with non-replicated systems or poorly replicated systems
  • IoT offline mode (coalescing chains)