IPI Ovirt

[mathgonzlez]

Describe the bug
After installing OKD (version 4.13.0-0.okd-2023-06-04-080300) using the IPI method within RHV 4.4SP1, the cluster was successfully installed.
I conducted a test by gracefully shutting down the cluster as per the documentation: Graceful Cluster Shutdown, and everything worked without any issues.
However, when attempting to shut down the nodes forcibly, by forcefully stopping the VMs to simulate an unexpected failure, the cluster could not recover.
While one of the workers could successfully obtain both an IP and the VIPs for Ingress and API, the other nodes displayed the following messages:

Version
OKD version: 4.13.0-0.okd-2023-06-04-080300
Install method: IPI
Cloud platform: RHV 4.4SP1

How reproducible
Always

Log bundle
I cant share must-gather because my cluster is unhealty

Thanks

[vrutkovs]

the cluster could not recover.

Do masters boot?

While one of the workers could successfully obtain both an IP and the VIPs for Ingress and API,

Why would a worker get an IP for API?

[mathgonzlez]

Sorry, it was a mistake, I meant the control planes. If the nodes boot.

EDIT: after a couple of minutes this is my IP asignation:

I access to master-2 and see the afterburn-service in failed state:

× [email protected] - Afterburn (SSH Keys)
Loaded: loaded (/usr/lib/systemd/system/[email protected]; enabled; preset: disabled)
Drop-In: /usr/lib/systemd/system/service.d
└─10-timeout-abort.conf
Active: failed (Result: exit-code) since Mon 2023-11-13 20:48:06 UTC; 21min ago
Process: 852 ExecStart=/usr/bin/afterburn ${AFTERBURN_OPT_PROVIDER} --ssh-keys=core (code=exited, status=1/FAILURE)
Main PID: 852 (code=exited, status=1/FAILURE)
CPU: 106ms
Nov 13 20:48:06 okd-ovirt-cdgkj-master-2 afterburn[852]: 0: writing ssh keys
Nov 13 20:48:06 okd-ovirt-cdgkj-master-2 afterburn[852]: 1: maximum number of retries (10) reached
Nov 13 20:48:06 okd-ovirt-cdgkj-master-2 afterburn[852]: 2: failed to fetch
Nov 13 20:48:06 okd-ovirt-cdgkj-master-2 afterburn[852]: 3: error sending request for url (http://169.254.169.254/latest/meta-data/public-keys): error trying to connect: tcp connect error: Network is unreachable (os error 101)
Nov 13 20:48:06 okd-ovirt-cdgkj-master-2 afterburn[852]: 4: error trying to connect: tcp connect error: Network is unreachable (os error 101)
Nov 13 20:48:06 okd-ovirt-cdgkj-master-2 afterburn[852]: 5: tcp connect error: Network is unreachable (os error 101)
Nov 13 20:48:06 okd-ovirt-cdgkj-master-2 afterburn[852]: 6: Network is unreachable (os error 101)
Nov 13 20:48:06 okd-ovirt-cdgkj-master-2 systemd[1]: [email protected]: Main process exited, code=exited, status=1/FAILURE
Nov 13 20:48:06 okd-ovirt-cdgkj-master-2 systemd[1]: [email protected]: Failed with result 'exit-code'.
Nov 13 20:48:06 okd-ovirt-cdgkj-master-2 systemd[1]: Failed to start [email protected] - Afterburn (SSH Keys).

[vrutkovs]

Nov 13 20:48:06 okd-ovirt-cdgkj-master-2 afterburn[852]: 3: error sending request for url (http://169.254.169.254/latest/meta-data/public-keys): error trying to connect: tcp connect error: Network is unreachable (os error 101)

afterburn is unable to reach metadata service.

[mathgonzlez]

Nov 13 20:48:06 okd-ovirt-cdgkj-master-2 afterburn[852]: 3: error sending request for url (http://169.254.169.254/latest/meta-data/public-keys): error trying to connect: tcp connect error: Network is unreachable (os error 101)

afterburn is unable to reach metadata service.

Yes i see that, But why this only happend when i shutdown my nodes in a not gracefully?

[vrutkovs]

Not much I can help with without a must-gather

[mathgonzlez]

Could I redeploy the cluster and share a must-gather before shutdown the cluster, do you think it could help?