Prototype

.github/workflows/test.yml

@@ -17,6 +17,7 @@ jobs:
   run:
 
     runs-on: ubuntu-latest
+    timeout-minutes: 5
     strategy:
       fail-fast: false
       matrix:
@@ -32,6 +33,10 @@ jobs:
         run: |
           python -m pip install --upgrade pip
           pip install hatch
+      - name: Pull images
+        run: |
+          docker pull mrcide/privateer-server:prototype
+          docker pull mrcide/privateer-client:prototype
       - name: Test
         run: |
           hatch run cov-ci

.gitignore

@@ -0,0 +1,9 @@
+*.pyc
+__pycache__
+dist/
+.coverage
+coverage.xml
+*.tar
+.privateer_identity
+tmp/
+.coverage.*

README.md

@@ -5,10 +5,94 @@
 
 -----
 
-**Table of Contents**
+## The idea
 
-- [Installation](#installation)
-- [License](#license)
+We need a way of synchronising some docker volumes from a machine to some backup server, incrementally, using `rsync`. We previously used [`offen/docker-volume-backup`](https://github.com/offen/docker-volume-backup) to backup volumes in their entirety to another machine as a tar file but the space and time requirements made this hard to use in practice.
+
+### The setup
+
+We assume some number of **server** machines -- these will receive data, and some number of **client** machines -- these will send data to the server(s).  A client can back any number of volumes to any number of servers, and a server can receive and serve any number of volumes to any number of clients.
+
+A typical framework for us would be that we would have a "production" machine which is backing up to one or more servers, and then some additional set of "staging" machines that receive data from the servers, which in practice never send any data.
+
+Because we are going to use ssh for transport, we assume existence of [HashiCorp Vault](https://www.vaultproject.io/) to store secrets.
+
+### Configuration
+
+The system is configured via a single `json` document, `privateer.json` which contains information about all the moving parts: servers, clients, volumes and the vault configuration. See [`example/`](example/) for some examples.
+
+We imagine that your configuration will exist in some repo, and that that repo will be checked out on all involved machines. Please add `.privateer_identity` to your `.gitignore` for this repo.
+
+### Setup
+
+After writing a configuration, on any machine run
+
+```
+privateer2 keygen --all
+```
+
+which will generate ssh keypairs for all machines and put them in the vault. This only needs to be done once, but you might need to run it again if
+
+* you add more machines to your system
+* you want to rotate keys
+
+Once keys are written to the vault, on each machine run
+
+```
+privateer2 configure <name>
+```
+
+replacing `<name>` with the name of the machine within either the `servers` or `clients` section of your configuration.  This sets up a special docker volume that will persist ssh keys and configurations so that communication between clients and servers is straightforward and secure.  It also leaves a file `.privateer_identity` at the same location as the configuration file, which is used as the default identity for subsequent commands. Typically this is what you want.
+
+### Manual backup
+
+```
+privateer backup <volume>
+```
+
+Add `--dry-run` to see the commands to run it yourself
+
+### Restore
+
+Restoration is always manual
+
+```
+privateer2 restore <volume> [--server=NAME] [--source=NAME]
+```
+
+where `--server` controls the server you are pulling from (useful if you have more than one configured) and `--source` controls the original machine that backed the data up (if more than one machine is pushing backups).
+
+For example, if you are on a "staging" machine, connecting to the "backup" server and want to pull the "user_data" volume that was backed up from "production" machine called  you would type
+
+```
+privateer2 restore user_data --server=backup --source=production
+```
+
+
+## What's the problem anyway?
+
+[Docker volumes](https://docs.docker.com/storage/volumes/) are useful for abstracting away some persistent storage for an application. They're much nicer to use than bind mounts because they don't pollute the host system with immovable files (docker containers often running as root or with a uid different to the user running docker).  The docker [docs](https://docs.docker.com/storage/volumes/#back-up-restore-or-migrate-data-volumes) describe some approaches to backup and restore but in practice this ignores many practical issues, especially when the volumes are large or off-site backup is important.
+
+We want to be able to synchronise a volume to another volume on a different machine; our setup looks like this:
+
+```
+bob                            alice
++-------------------+          +-----------------------+
+|                   |          |                       |
+| application       |          |                       |
+|  |                |          |                       |
+| volume1           |          |     volume2           |
+|  |                |   ssh/   |      |                |
+| privateer-client--=----------=---> privateer-server  |
+|  |                |  rsync   |      |                |
+| keys              |          |     keys              |
+|                   |          |                       |
++-------------------+          +-----------------------+
+```
+
+so in this case `bob` runs a privateer client which sends data over ssh+rsync to a server running on `alice`, eventually meaning that the data in `volume1` on `bob` is replicated to `volume2` on `alice`.  This process uses a set of ssh keys that each client and server will hold in a `keys` volume.  This means that they do not interact with any ssh systems on the host.  Note that if `alice` is also running sshd, this backup process will use a *second* ssh connection.
+
+In addition, we will support point-in-time backups on `alice`, creating `tar` files of the volume onto disk that can be easily restored onto any host.
 
 ## Installation
 

buildkite/pipeline.yml

@@ -0,0 +1,3 @@
+steps:
+  - label: ":whale: Build and push"
+    command: docker/build

development.md

@@ -0,0 +1,115 @@
+# Development notes
+
+Because this uses docker, vault and requires work with hostnames, this is going to be hard to test properly without a lot of mocking.  We'll update this as our strategy improves.
+
+## Vault server for testing
+
+We use [`vault-dev`](https://github.com/vimc/vault-dev) to bring up vault in testing mode.  You can also do this manually (e.g., to match the configuration in [`example/simple.json`](example/simple.json) by running
+
+```
+vault server -dev -dev-kv-v1
+```
+
+If you need to interact with this on the command line, use:
+
+```
+export VAULT_ADDR='http://127.0.0.1:8200'
+export VAULT_TOKEN=$(cat ~/.vault-token)
+```
+
+within the hatch environment before running any commands.
+
+## Worked example
+
+We need to swap in the globally-findable address for alice (`alice.example.com`) for the value of the machine this is tested on:
+
+```
+mkdir -p tmp
+sed "s/alice.example.com/$(hostname)/" example/local.json > tmp/privateer.json
+```
+
+Create a set of keys
+
+```
+privateer2 --path tmp keygen --all
+```
+
+You could also do this individually like
+
+```
+privateer2 --path tmp keygen alice
+```
+
+Set up the key volumes
+
+```
+privateer2 --path tmp configure alice
+privateer2 --path tmp configure bob
+```
+
+Start the server, as a background process (note that if these were on different machine the `privateer2 configure <name>` step would generate the `.privateer_identity` automatically so the `--as` argument is not needed)
+
+```
+privateer2 --path tmp --as=alice server start
+```
+
+Once `alice` is running, we can test this connection from `bob`:
+
+```
+privateer2 --path tmp --as=bob check --connection
+```
+
+This command would be simpler to run if we are in the `tmp` directory, which would be the usual situation in a multi-machine setup
+
+```
+privateer2 check --connection
+```
+
+For all other commands below, you can drop the `--path` and `--as` arguments if you change directory.
+
+Create some random data within the `data` volume (this is the one that we want to send from `bob` to `alice`)
+
+```
+docker volume create data
+docker run -it --rm -v data:/data ubuntu bash -c "base64 /dev/urandom | head -c 100000 > /data/file1.txt"
+```
+
+We can now backup from `bob` to `alice` as:
+
+```
+privateer2 --path tmp --as=bob backup data
+```
+
+or see what commands you would need in order to try this yourself:
+
+```
+privateer2 --path tmp --as=bob backup data --dry-run
+```
+
+Delete the volume
+
+```
+docker volume rm data
+```
+
+We can now restore it:
+
+```
+privateer2 --path tmp --as=bob restore data
+```
+
+or see the commands to do this ourselves:
+
+```
+privateer2 --path tmp --as=bob restore data --dry-run
+```
+
+Tear down the server with
+
+```
+privateer2 --path tmp --as=alice server stop
+```
+
+## Writing tests
+
+We use a lot of global resources, so it's easy to leave behind volumes and containers (often exited) after running tests. At best this is lazy and messy, but at worst it creates hard-to-diagnose dependencies between tests. Try and create names for auto-cleaned volumes and containers using the `managed_docker` fixture (see [`tests/conftest.py`](tests/conftest.py) for details).

docker/Dockerfile.client

@@ -0,0 +1,10 @@
+FROM ubuntu
+
+RUN apt-get update && \
+        apt-get install -y --no-install-recommends \
+        openssh-client \
+        rsync && \
+        mkdir -p /root/.ssh
+
+COPY ssh_config /etc/ssh/ssh_config
+VOLUME /privateer/keys

docker/Dockerfile.server

@@ -0,0 +1,17 @@
+FROM ubuntu
+
+RUN apt-get update && \
+        apt-get install -y --no-install-recommends \
+        openssh-client \
+        openssh-server \
+        rsync && \
+        mkdir -p /var/run/sshd && \
+        mkdir -p /root/.ssh
+
+COPY sshd_config /etc/ssh/sshd_config
+
+VOLUME /privateer/keys
+VOLUME /privateer/volumes
+EXPOSE 22
+
+ENTRYPOINT ["/usr/sbin/sshd", "-D", "-E", "/dev/stderr"]

docker/build

@@ -0,0 +1,30 @@
+#!/usr/bin/env bash
+set -exu
+
+HERE=$(dirname $0)
+. $HERE/common
+
+docker build --pull \
+       --tag $TAG_SERVER_SHA \
+       -f $HERE/Dockerfile.server \
+       $HERE
+
+docker build --pull \
+       --tag $TAG_CLIENT_SHA \
+       -f $HERE/Dockerfile.client \
+       $HERE
+
+docker push $TAG_SERVER_SHA
+docker push $TAG_CLIENT_SHA
+
+docker tag $TAG_SERVER_SHA $TAG_SERVER_BRANCH
+docker push $TAG_SERVER_BRANCH
+docker tag $TAG_CLIENT_SHA $TAG_CLIENT_BRANCH
+docker push $TAG_CLIENT_BRANCH
+
+if [ $GIT_BRANCH == "main" ]; then
+   docker tag $TAG_SERVER_SHA $TAG_SERVER_LATEST
+   docker push $TAG_SERVER_LATEST
+   docker tag $TAG_CLIENT_SHA $TAG_CLIENT_LATEST
+   docker push $TAG_CLIENT_LATEST
+fi

docker/common

@@ -0,0 +1,24 @@
+# -*-sh-*-
+DOCKER_ROOT=$(realpath $HERE/..)
+PACKAGE_ORG=mrcide
+CLIENT_NAME=privateer-client
+SERVER_NAME=privateer-server
+
+# Buildkite doesn't check out a full history from the remote (just the
+# single commit) so you end up with a detached head and git rev-parse
+# doesn't work
+if [ false && "$BUILDKITE" = "true" ]; then
+    GIT_SHA=${BUILDKITE_COMMIT:0:7}
+    GIT_BRANCH=$BUILDKITE_BRANCH
+else
+    GIT_SHA=$(git -C "$DOCKER_ROOT" rev-parse --short=7 HEAD)
+    GIT_BRANCH=$(git -C "$DOCKER_ROOT" symbolic-ref --short HEAD)
+fi
+
+TAG_CLIENT_SHA="${PACKAGE_ORG}/${CLIENT_NAME}:${GIT_SHA}"
+TAG_CLIENT_BRANCH="${PACKAGE_ORG}/${CLIENT_NAME}:${GIT_BRANCH}"
+TAG_CLIENT_LATEST="${PACKAGE_ORG}/${CLIENT_NAME}:latest"
+
+TAG_SERVER_SHA="${PACKAGE_ORG}/${SERVER_NAME}:${GIT_SHA}"
+TAG_SERVER_BRANCH="${PACKAGE_ORG}/${SERVER_NAME}:${GIT_BRANCH}"
+TAG_SERVER_LATEST="${PACKAGE_ORG}/${SERVER_NAME}:latest"

docker/ssh_config

@@ -0,0 +1,6 @@
+PasswordAuthentication no
+IdentityFile /privateer/keys/id_rsa
+SendEnv LANG LC_*
+HashKnownHosts no
+UserKnownHostsFile /privateer/keys/known_hosts
+Include /privateer/keys/config

docker/sshd_config

@@ -0,0 +1,28 @@
+Include /etc/ssh/sshd_config.d/*.conf
+
+PermitRootLogin prohibit-password
+#StrictModes yes
+#MaxAuthTries 1
+#MaxSessions 10
+
+PubkeyAuthentication yes
+
+AuthorizedKeysFile	/privateer/keys/authorized_keys
+HostKey /privateer/keys/id_rsa
+
+PasswordAuthentication no
+ChallengeResponseAuthentication no
+UsePAM no
+
+#AllowAgentForwarding yes
+#AllowTcpForwarding yes
+#GatewayPorts no
+X11Forwarding no
+# PermitTTY no
+PrintMotd no
+
+# Allow client to pass locale environment variables
+AcceptEnv LANG LC_*
+
+# override default of no subsystems
+# Subsystem	sftp	/usr/lib/openssh/sftp-server