Make controller network optional in distributed mode

[lucaslorentz]

This diff reduces friction in adopting distributed mode by making it work with a single network, without a controller network.

[lucaslorentz]

Still thinking about this, it's less secure, but might be useful for some people.

It's also going in the direction I want for the future. I would like to have a simpler way of securing the controller and server relationship. Maybe just a shared secret sent via HTTP Authorization header. Since caddy admin doesn't support it, maybe we should use this clever approach to expose admin with basicauth or some authorization header check.

[francislavoie]

@mholt you might have some thoughts on securing admin access here

[mholt]

The recommended way of securing an open admin endpoint is to use TLS mutual auth. Use the remote admin endpoint for this, rather than opening the plaintext one up for everyone to access: https://caddyserver.com/docs/json/admin/remote/

[francislavoie]

Is there a way to fully automate that though? Distributing/sharing a cert is a lot more complicated than just a shared key.

[mholt]

The way I see it, it's the same: you're sharing a key in both cases. Instead of sharing a secret key, you share a public key. And sharing the public key is better because then you're not sharing secrets 😃

[francislavoie]

But automating it is difficult (impossible?) because the "secret" in that case can't be known ahead of time before starting the server(s). From a UX standpoint for this project, that's a no-go, I think. It's not viable to ask users of CDP to understand how to set up and copy a cert to establish trust. Especially difficult in Docker because of filesystem isolation (which is an advantage).

[mholt]

the "secret" in that case can't be known ahead of time before starting the server(s).

Why's that? The public key you trust doesn't depend on itself, does it?

It's not viable to ask users of CDP to understand how to set up and copy a cert to establish trust. Especially difficult in Docker because of filesystem isolation (which is an advantage).

Well, I don't know anything about how Docker works, but I'll just say that in general, this can be automated because Caddy can operate a self-managing CA, then you just add its public key to your configuration to trust it. Instead of adding a secret string you add a public one.

[francislavoie]

"Distributed mode" implies that you have a Caddy instance that acts as the ingress (server), and another that reads the Docker cluster to see what services exist (controller), and generates a config to push it to the ingress instance. See https://github.com/lucaslorentz/caddy-docker-proxy#execution-modes

In that case, since the controller wants to push a config to the server on any service changes. But doing that securely is difficult currently. Those two instances are isolated from eachother at the filesystem level, but can communicate with eachother over the network. Assume they could be on different machines entirely (e.g. Docker Swarm).

Setting up mutual auth with certs is difficult because users are expected to not need to reach into containers to grab files (root CA cert etc) or write files to other containers.

A shared secret is much easier because it can be passed into both the server and controller containers as environment variables (for example) with no consideration of storage, no manual steps to copy things, no need to bootstrap the server's CA by running it once first before the controller can do anything.

[lucaslorentz]

I understand that ideally, we should use MTLS. But indeed automating that will be a big effort.

I'm closing this PR in favor of #431
The new PR simplifies configuration while maintaining security by exposing Caddy admin only to a "controller network".

[francislavoie]

Ah crap - I didn't mean to click to delete the branch, GitHub mobile app makes it a big fat button, accidentally tapped it while scrolling. I hope that's not a problem for you 🙈

[lucaslorentz]

No worries. I would delete it as well. Have it locally as well

[mholt]

Setting up mutual auth with certs is difficult because users are expected to not need to reach into containers to grab files (root CA cert etc) or write files to other containers.

I would have each of the instances configured to use ACME to get their certs from a known, trusted Caddy instance running its own acme_server. Then simply trust the one root cert of that ACME server.

Anyway, sounds like the solution you ended up with is fine @lucaslorentz -- thanks for working on that. :) And thanks for the awesome plugin, many users appreciate this!

[francislavoie]

Then simply trust the one root cert of that ACME server.

But that still requires copying the root cert from that instance to establish trust. Which is the hard part for most users, and a blocker for making this automated.

[mholt]

Isn't it a string just like a shared secret? 🤔

[lucaslorentz]

I would have each of the instances configured to use ACME to get their certs from a known, trusted Caddy instance running its own acme_server. Then simply trust the one root cert of that ACME server.

Yeah, I've considered doing that with https://github.com/smallstep/certificates.
But I didn't know caddy had it built in, I think that will be a better option. Will do some tests on another PR.

[mholt]

I talked to Francis off-line about this a bit, and I feel like I understand the problem a little more.

We do have an API endpoint to get the CA's certificates: https://caddyserver.com/docs/api#get-pkicaltidgt

It could be reverse-proxied to over HTTP, as it reveals only public information. (At least, for now. I don't plan on revealing anything sensitive on that endpoint, but https://caddyserver.com/docs/api#get-pkicaltidgtcertificates is probably a safer bet as it gets only the certificate chain.)