n2s is a storage coordination service. It connects version-based and incrementally updating data sources with any storage backend. It discovers changed files through clients (dsg repositories, ZFS/btrfs snapshots, or filesystem scanning), encrypts the content with path-aware blob creation, then pushes it to configured backends (S3-compatible, IPFS, rclone-supported services, or local archives) with upload tracking and restart capability. A database tracks all operations, enabling fast queries about what's stored where and when the upload was confirmed.
All content is hashed using BLAKE3, ensuring that data integrity can be verified at every point in the system. Clients push data to backends in units called changesets, which represent the state of the client's data at a specific time. The simplified architecture prioritizes operational reliability and disaster recovery over storage deduplication.
The architecture separates concerns: clients handle file discovery, n2s handles storage logistics, and backends handle the actual bytes. This design supports everything from single-user laptop backups to organization-wide data preservation workflows.
Clients - File Discovery Layer:
- dsg: Store versioned data repositories
- ZFS or btrfs snapshots: backup incremental filesystem changes between snapshots
- Find-based scanning: Schedule regular backups of recently modified files in any directory tree
n2s - Storage Coordination Layer:
- API operations: Provides push, pull, list, and search operations for clients
- Data management: Handles encryption and path-aware content-addressable storage
- State tracking: Manages database records of what's stored, where, and when
Backends - Storage Layer:
- Cloud storage: S3-compatible for reliable object storage
- Distributed storage: IPFS for content-addressed peer-to-peer storage
- Flexible destinations: Any rclone-supported backend (Google Drive, Dropbox, etc.) or unix filesystems (local or SSH)
n2s uses a simplified 2-table design optimizing for operational simplicity:
- Path-aware blob creation: Same content at different paths creates different blobs, enabling disaster recovery without database dependency
- Deterministic operations: Identical content at same path always produces same blob ID (same file hash + same path = same blob)
- Single backend per database: Eliminates complex coordination overhead
- Resume capability: Simple retry logic for failed uploads
This design trades storage efficiency for operational reliability - you can always recover your data with minimal tooling.
- Path-aware content-addressable storage: Files identified by BLAKE3(path:file_hash)
- Deterministic blob creation: Identical content at same path always produces same blob ID
- Multiple backends: Store to S3, IPFS, rclone-supported services, or local storage
- ChaCha20-Poly1305 encryption: All content encrypted before storage with deterministic key derivation
- Disaster recovery: Complete system rebuild from storage and encryption keys alone
The system uses a simplified 2-table design:
- changesets: Groups of files pushed together with status tracking
- files: File records with upload status and blob references
For detailed technical documentation:
- Architecture - Complete system design including components, database schema, API, and design decisions
- Blob Creation Performance Analysis - ChaCha20-Poly1305 implementation and performance testing
- Disaster Recovery - Complete disaster recovery guide with working tools
# Requires Python >= 3.13 and uv
git clone [email protected]:HRDAG/n2s.git
cd n2s
uv venv
source .venv/bin/activate
uv pip install -e .n2s --help
Usage: n2s [OPTIONS] COMMAND [ARGS]...
Storage coordination service with path-aware blob creation and disaster
recovery
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --verbose -v Enable verbose logging │
│ --config -c TEXT Path to config file [default: None] │
│ --install-completion Install completion for the current │
│ shell. │
│ --show-completion Show completion for the current shell, │
│ to copy it or customize the │
│ installation. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ───────────────────────────────────────────────────────────────────╮
│ push Store files in a new changeset │
│ pull Restore files from storage │
│ status Query changeset and upload status │
╰──────────────────────────────────────────────────────────────────────────────╯
Example Usage:
# Store files in a changeset
n2s push file1.txt file2.txt --changeset-name "daily-backup"
# Restore files from storage
n2s pull --changeset-id abc123 --target-path /restore/here
# Check upload status
n2s statusNote: n2s is primarily a library for external clients. The CLI is provided for development and debugging.
GPL-2 or newer, (c) HRDAG 2025