how should MN-1 manage runaway contract vats?

[warner]

What is the Problem Being Solved?

As examined in #4219, we don't currently protect against infinite loops in contract vats (which isn't great), but it's not clear what we should do instead. All vats in MN-1 are critical, and a consensus-based metering-triggered vat termination will delete all the vat's data, which is awfully hard to recover from. So having the kernel halt is probably preferable to terminating a critical vat. If we implement #4516 then we might have a way to halt the kernel but allow the cosmos-sdk side to continue limping along, which might be better (but there are a lot of details to figure out).

I don't know what path we should take, but this ticket is about exploring the options and deciding on a plan.

Description of the Design

Security Considerations

Test Plan

[warner]

We don't anticipate non-critical contracts for MN-1.0, but possibly for MN-1.1 (a "liquidity mine" contract, maybe the existing liquidation contract).

It would be nice if we could pause vats when they encounter an error (#)

The most conservative approach is to halt the chain in the face of any error, which is what we already have (as long as bootstrap sets the critical-vat flag on all contract vats). However that doesn't provide as much availabilty as we want. It turns potentially medium-sized emergencies into large emergencies.

The liquidation contract is established by governance, and could be replaced by governance, so terminating that vat is a reasonable approach. We would need to mark it as non-critical, and set up an "unlimited Meter" (which ought to be enough to set the per-delivery computron limit). And we'd need to update our definition of infinity (for which #5295 is an input).

If the AMM contract were non-critical, a failure would be relatively low-impact, however we would lose the one tranche of tokens being sold at that time. We've talked about establishing an heir/successor for each seat to have a path for recovering these assets.

We should conduct a vat failure fire drill before MN-1.