Skip to content

superfly/corrosion

This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.

Folders and files

NameName
Last commit message
Last commit date

Latest commit

404feed · Nov 30, 2023
Sep 18, 2023
May 27, 2023
Aug 21, 2023
Oct 19, 2023
Nov 30, 2023
Oct 27, 2023
Oct 25, 2023
Sep 7, 2023
May 30, 2023
Aug 23, 2023
Oct 27, 2023
Nov 30, 2023
Nov 20, 2023
Sep 21, 2023
Sep 22, 2023
Oct 19, 2023
Aug 16, 2023
Sep 18, 2023
Jul 17, 2023
Jul 15, 2023

Repository files navigation

Corrosion

Gossip-based service discovery (and more) for large distributed systems.

Why we built Corrosion

We built Corrosion specifically for service discovery across a large global network, replacing Consul’s central state database with eventually consistent state distributed across our hosts.

Our new tool needed to deliver the following:

Fast reads and writes

Getting state (data) from a central remote source can be incredibly expensive (at least 300ms for a round-trip to something on the other side of the world), but usually takes less than 1ms from a local source.

Fast, eventual, consistency

Many use cases can cope with eventual consistency, especially if a consistent state is attained sooner than later. Raft fell short for some use cases at Fly.io where round-trips to a centralized location are too expensive.

Flexibility

Global state for a distributed system isn't one-size-fits-all. Flexible schemas and queries are essential.

How Corrosion works

In a nutshell, Corrosion:

  • Maintains a SQLite database on each node
  • Gossips local changes throughout the cluster
  • Uses CR-SQLite for conflict resolution with CRDTs
  • Uses Foca to manage cluster membership using a SWIM protocol
  • Periodically synchronizes with a subset of other cluster nodes, to ensure consistency

Features

  • A flexible API to read from and write to Corrosion's store using SQL statements
  • File-based schemas with on-the-fly updates
  • HTTP streaming subscriptions based on SQL queries
  • Live population of configuration files from Corrosion state with user-defined Rhai templates
  • Storage and propagation of state from locally registered Consul services, replacing the central database with Corrosion's distributed state
  • Secure peer-to-peer communication with the QUIC transport protocol (using Quinn)

Usage overview

Run the Corrosion agent on every node/host in the cluster. Other programs running on the node use Corrosion's HTTP API to query the local Corrosion SQLite database, add and update data, and subscribe to change notifications.

The Corrosion CLI provides commands for administration and access to database and features.

Quick start

See the WIP Corrosion documentation for more details.

Building Corrosion

Clone https://github.com/superfly/corrosion.git.

From within the repo directory:

cargo build --release && mv target/release/corrosion ./