Disaster Recovery

[jrudolph]

Hi,

I've used hetzner-k3s for a few months and it works great, thanks a lot.

In those 4 months I have had two instances of ~1 day of downtime because of failures inside the Hetzner infrastructure and very slow support times (in one case mounting/unmounting volumes hang and locked the server, so you couldn't run any more operations, so all other mounted volumes were also trapped in a powered-off server. In the current instance, one cloud node, unfortunately the single k8s master, has severe networking problems).

Of course, I can rebuild the whole cluster quickly with hetzner-k3s but it would still take me some time to get to the same state as before (e.g. hcloud pvcs need to either be copied to new instances or you need to figure out how to let the hetzner csi pick up the existing volumes).

I have backups (and some kind of access) to the existing k3s etcd snapshots, so in theory it should be possible to recover k8s state from that snapshot. In practice it is not so easy to set k3s up on a new node and have hetzner-k3s fill out the remaining bits for everything to work together.

Any hints about how to go about that? (One useful feature would be to have a kind of --dry-run feature that just prints out the k3s commands necessary to start k3s, so that you could adapt them how you need them to be.)

[vitobotta]

Hi, the problematic part is PVCs. Even if you restore etcd snapshots this won't work in an entirely new cluster. I use Velero to back up everything, including volumes. Have you tried that?

[jrudolph]

Sorry for dropping the ball here. No, after all the problem was solved phoning in to support.

I agree dealing with PVCs is somewhat of a hassle. I think I read somewhere that it is possible to reconfigure the hcloud CSI to use existing volumes but I cannot seem to find the information right now.

[vitobotta]

I don't think it's possible to somehow "reconnect" a pvc to an existing volume but it needs investigation.