How to fix broken machine due to bad patch?

[justyns]

I've been testing various things with omni and a few times I've added a "bad" patch that causes nodes to get stuck in a reboot loop.

E.g., I tried adding this patch:

      machine:
        files:
          - content: ""
             op: overwrite
             path: /etc/cni/net.d/05-cilium.conf

It failed due to the file not existing, which is fine, but even after removing that part of the patch and saving it in omni - the affected nodes were still stuck in a reboot loop. They'd boot into Talos, have an error about that file not existing, and then reboot again. Rebooting the machine into maintenance mode seems to break it out of the loop and fix it, but that doesn't seem like the right way to do it.

Is there another way to fix nodes that are in a state like this?

Or is there a way to run talosctl apply-config using omnictl?

edit: For context, the log says this (different error for create vs overwrite, but otherwise the same), but I wasn't able to run either of these talosctl commands:

[talos] task writeUserFiles (1/1): writeUserFiles failed, rebooting in 35 minutes. You can use talosctl apply-config or talosctl edit mc to fix the issues, error:
create operation not allowed outside of /var: "/etc/cni/net.d/05-cilium.conf"

The node also reboots almost immediately - not 35 minutes later like it says.