impossible to troubleshoot ignition failure on vsphere

[mritalian]

The bane of my existence is when ignition fails on vsphere.

In short

1. I create a VM with ignition 3.3, and something fails
2. Opening the console, the scrollback buffer contains only the ignition JSON (truncated) and "press enter to enter emergency shell"
3. Pressing enter does not actually open any emergency shell. Again, the scrollback history is limited

So, how am I supposed to diagnose these failures? Is this a known issue? It has been this way for years and years wih the latest ova.

On AWS, the ami there actually opens the shell when prompted. But I've never seen it actually work on vsphere.

TIA

[tormath1]

Hello @mritalian,

What happens when you're pressing enter to open the emergency shell?

EDIT: I just tried to boot a Flatcar instance with a wrong Ignition configuration on vpshere, I landed into the emergency shell as expected by pressing enter (then systemctl --failed to the failed Ignition units)

[pothos]

What we do in the automated testing is to fetch the serial console log as text: https://www.suse.com/support/kb/doc/?id=000018805 (Default name is "serial.out")

[mritalian]

Right when the shell does work the cause is easy to find. In my case pressing enter scrolls lots of text and then hangs with an error failed to open shell or similar.

The serial log is a great idea I will use that and force a failure to see what happens and will follow up here.

Thanks both of you

[pothos]

Can you also check if the emergency console works for you on the latest Alpha release? I think something related got fixed.

[mritalian]

An upate - things are conspiring to make this harder to troubleshoot

1. The problem is transient: vm deployment that fails to come up and chokes as described above, will run firstboot successfully including ignition if I simply send ctrl+alt+del

2. vsphere_virtual_machine resource of terraform doesn't support serial_port. Serial ports defined in an ova are ignored by vsphere. Because of point 1, by the time the problem happens, is too late to add the serial port

3. In case where problem is not transient, i.e. trying to mount a non-existent disk, I can enter the shell without problem

I'll try a more complex ignition failure that I can trigger, like a duplicate file.

[pothos]

Did you test it locally with the qemu script? Also, if you can share a reproduce that would also be good.

[mritalian]

@pothos So this is obviously a heisenbug, now that I want to collect logs of the emergency shell failure, I can't. But I found the cause of the sporadic random ignition failures. 😩 I will comment there #222

[pothos]

Ah, good, yeah, the sdb, sdc, etc. ordering can't be relied on, the solution is to set up a partition (or filesystem) label and use that in mount units or elsewhere.