CosmosFullNode: Detect height drift and take action

[DavidNix]

This is probably an extension of SelfHeal.

There are some chains (primarily Evmos) that do not properly report the comet /status of catching up. They report as if they are in sync, but they fall behind.

With some manual testing, deleting and restarting the pod seems to correct the issue, at least with Evmos. There could be other actions like increasing the outbound peers but that will come with extra egress costs.

I feel deleting the pod (and the controller will recreate it) is a good first pass.

Tasks

Beta Give feedback

1. do the thing
2. release in prod (injective? we're not in the active set)

[DavidNix]

I've also seen this occasionally with Injective and other chains, but Evmos is the largest offender.

[jonathanpberger]

Is there a vrt angle here around testing drift?