Randao and Builder Proposer Failures

[boulder225]

🎯 Problem to be solved

Three Lido clusters (Duck, Elk, and Kinkajou) on mainnet are experiencing problems with randao duties and builder proposer failures. One cluster is slightly different from the other two, where it marks the randao duty as failed, while the others simply state that the builder proposer failed.

https://grafana.monitoring.gcp.obol.tech/d/d6qujIJVk/cluster-overview-mainnet?from=now-2d&orgId=8&to=now&var-cluster_hash=0131dd1&var-cluster_name=Lido+x+Obol%3A+Knightly+Kinkajou&var-cluster_network=mainnet&var-cluster_peer=alive-fish&var-duty=All&var-interval=%24__auto_interval_interval&var-job=charon&var-peer=All

🛠️ Proposed solution

• Investigate the logs and metrics to identify the root cause of the randao and builder proposer failures.
• Check for any recent updates or configurations that might be causing the issue.
• Verify the connectivity and health of the clusters' nodes
• Review the randao and proposer settings and configurations

Update

The issue seems related to produce block v3, as this endpoint needs to be turned off until v1 is released.

[boulder225]

ObolNetwork#63

[pinebit]

A quick investigation done by @OisinKyne @KaloyanTanev and @pinebit led to the conclusion that due to the version bump for lodestar (1.16 => 1.17), we have to enforce --useProduceBlockV3=false flag, otherwise in according to the code - it will be true (for deneb+).
Because we don't receive logs for the affected clusters, we could not prove this 100%.
We all agreed the fix proposed by @OisinKyne ObolNetwork#63 is legit and should help resolving the issue.