Optimize magic-wormhole mailbox and relay images

[btlogy]

Following up on #20 (and #21), we may want to reduce the size of the mailbox and relay image using slim-bullseye images (from 1.5GB to 929MB).
Alternatively, we may want to switch to Alpine (see #22).

Although, we should assess the possible impact on our own back end which consumes those images (extensive test on staging preferable).

In addition, there is an issue in the current Dockerfiles.
The following RUN commands should be written in a single line:

RUN apt update
RUN apt install rustc -y

Also, moving this RUN command a few line above would improve the caching (at least when building locally).

[meejah]

I would be very hesitant to use a 'smaller' / less well-known (and tested) distribution like Alpine for a long-running, SPoF, high-security Internet server. This needs to be secure, and Debian seems to be like a great choice for that.

Why do we care how big the image is?

If we're considering changing how these are deployed, I would also ask why we would continue to use Docker at all given its dubious security track-record.

[piegamesde]

If we're considering changing how these are deployed, I would also ask why we would continue to use Docker at all given its dubious security track-record.

If we start that discussion, I'd like to throw NixOS into the ring again. Its package set is one of the most up-to-date, and it is stable enough to be left unattended with auto-update IME. If our main motivator for currently using Docker is process isolation, then systemd service hardening can already do all what we need.

[meejah]

Yeah, in my mind NixOS would be obvious "other" choice since a bunch of our other services are deployed using NixOS.

[btlogy]

Why do we care how big the image is?

The size (and the "broken" cache) is mostly annoying when developing and testing, not really a problem when deploying indeed (unless you're deploying a lot).

But smaller image also tends to improve security (less software = less vulnerabilities).
For instance (base image name: compressed size - Critical / High / Medium / Low vulnerabilities):

• pypy:3.9-7.3.9-bullseye: 343 MB - 5 C / 8 H / 57 M / 85 L
• pypy:3.9-7.3.9-slim-bullseye: 69 MB - 3 H / 3 M / 19 L

In general, the smaller, the better.

IMHO, adding -slim to the Dockerfiles is a very small change that is worth trying.

[hacklschorsch]

If our main motivator for currently using Docker is process isolation, then systemd service hardening can already do all what we need.

@piegamesde I believe our main motivator of using docker is ease of deployment. (And yes, Docker conflates too many things IMHO)

IIRC for good isolation one should also not run the thing in the docker container as root - since that makes escaping the container / namespace / etc easier. Maybe running all of Docker in rootless mode is feasible/simpler?

[btlogy]

I would be very hesitant to use a 'smaller' / less well-known (and tested) distribution like Alpine for a long-running, SPoF, high-security Internet server. This needs to be secure, and Debian seems to be like a great choice for that.

I agree: moving from Debian to Alpine might be a step too far AND do not seem to be an obvious option in the case of the mailbox and relay as Alpine base images are not available for pypy!

... I'd like to throw NixOS into the ring again ...

... NixOS would be obvious "other" choice ...

NixOS is indeed a path we should have a good look at.
IMHO, considering a migration to NixOS is not what I was scoping in "optimizing" the images.

We had a discussion about Docker vs NixOS in an open pairing session yesterday.
I've took some notes I'll share elsewhere later, likely in an other issue (and possibly an org-wide project).

[btlogy]

IIRC for good isolation one should also not run the thing in the docker container as root - since that makes escaping the container / namespace / etc easier. Maybe running all of Docker in rootless mode is feasible/simpler?

Good point @hacklschorsch, I would be surprised if we could not run these services rootless.
I think this would be a good thing regardless of which path we choose later on.
I'll make a quick test.

[btlogy]

Good point @hacklschorsch, I would be surprised if we could not run these services rootless. I think this would be a good thing regardless of which path we choose later on. I'll make a quick test.

It seems to work just fine. But I'll create a separate issue because the change will require some work elsewhere.

[exarkun]

It might be worth clarifying the "NixOS" solution. It can be broken into a few pieces:

• Use Nix to build Docker images. Leave everything else alone. As demonstrated on the call yesterday, with fairly little effort (~1 line of Nix expression) we can build a Docker image with magic-wormhole (though meejah pointed I picked the wrong Python package to install). The resulting image is less than 100MB and is much closer to containing only the real dependencies than any of the Debian, Alpine, etc base images get us. These Docker images also have better caching properties than you typically get from images built using Dockerfile (eg, if zlib updates but glibc doesn't, you will get a whole new base Debian image of hundreds of megabytes but the Nix image will only have one layer update that includes only the zlib changes). These Docker images can be deployed just the same as a Docker image built using a Dockerfile. Apart from this, reproducibility is better and there's a simpler story around how you do testing to make sure all of your dependencies are included and properly configured and such.
• Use a NixOS host to run Docker containers. Not all of the usual motivations apply when running a Docker container on a NixOS host - but you can certainly do it. It doesn't matter how the Docker image is built so this can be considered entirely independently of any decision about how the Docker images are built. The advantage here is better system management tools than you get with Ansible: reproducibility extends to the whole Linux system, you get rollbacks for free, you can easily build VMs of production systems for local testing, you can compute diffs between two versions of a production system, etc.
• Use Nix natively from top to bottom. Use NixOS on the host and Nix expressions to define the service and skip the Docker layer. This gives all the benefits of both of the above, plus some more since Docker is no longer involved (so you no longer even have images you have to build and manage). You can do as much or as little process isolation as you want with systemd (including systemd-nspawn, which is roughly the same thing as Docker, which you can configure natively with Nix expressions too).

Another nice thing to note here is that the first two options are each incremental steps towards the last option.

[btlogy]

Thank you @exarkun : it is exactly what I'd like to integrate in this general document we've talked about.

[meejah]

I do understand that any one change to the existing deployments is "small" -- the reason I bring it up now is because we never "actually chose" Docker (it was an accident) and continuing to do a lot of "small" tweaks (and then subsequent downstream updates, fixes etc) is that in the end "a lot" of time has then been sunk into a thing.

So: do we want that thing? (Or more accurately here: do we want more of the thing, or less of it?)

[btlogy]

I'm not sure how to accurately measure the time Docker has cost for the whole magic-wormhole project so far.
But looking at those contributors and the code-frequency graphs, it does not seem like this repository is weighting "a lot", right?

Maybe there was more time spend downstream in LeastAuthority/mw-projects? I'm not sure I can tell.

Though, even if I find it hard to picture how we're going to save time, I'll be happy to migrate to anything else (as long as it FOSS).
But let's discuss this matter in LeastAuthority/mw-projects#113

Back to this issue, I propose to consider the optimizations introduced by #28 to be good enough once staging will be tested with the new images.

[btlogy]

I still need to deploy these smaller images, but I was waiting to have better tag upstream and dependabot enabled to verify the automation has been improved:

• Publish tags that can be consumed by dependabot #33
• LeastAuthority/mw-projects#114

I'm not so sure I should wait that long...

[btlogy]

Working smoothly on staging since yesterday:

REPOSITORY                              TAG       IMAGE ID       CREATED        SIZE
leastauthority/magic-wormhole-mailbox   23.5.0    b6e53b0ad28e   9 days ago     973MB
local/magic-wormhole-mailbox            latest    b6e53b0ad28e   9 days ago     973MB
leastauthority/magic-wormhole-relay     23.5.0    a904a1d7536d   9 days ago     971MB
local/magic-wormhole-relay              latest    a904a1d7536d   9 days ago     971MB