penumbra: switch to persistent testnets

[aubrika]

Bi-weekly testnets have been a great forcing function to make sure that “our code actually runs”. A natural next step for us is to start planning for a context where “our code runs all the time” and the associated set of challenges, both technical and organizational.

Why this feature is required pre-mainnet

Running a persistent testnet requires the team to bias towards:
Minimizing breaking changes which in turns requires:
Being able to reliably identify breaking changes
Planning for integration with existing code and chain state migrations
Coordinate around release changes

In addition to this, it will enable us to exercise our software over longer periods of time and in particular open persistent IBC connections and channels to other testnets for a chain that is updated regularly.

What we need to implement

With the exception of IBC host chain upgrades, all the necessary components are lined up. We should brainstorm what a sustainable update cadence looks like and if any changes to our release process need to happen. The current plan looks like:

• Design a versioning system that reflects different levels of compatibility:
• penumbra: rethink our versioning strategy #3307
• App-breaking changes that require chain upgrades (use ABCI app version)
• Rust-API breaking changes that do not require chain upgrades
• Refactor infrastructure around new versioning system
• Won’t be able to assume “rust major version change => chain upgrade”
• Adjust our infrastructure to support longer lived full nodes
• Plan how we will deploy an upgrade
• Encourage PR reviews in order to correctly flag consensus breaking changes
• Train the team to think through how various classes of changes affect Penumbra’s counterparty chains differently.
• Train each team member to write migrations for the set of changes that they are responsible for.

[hdevalence]

Merging with #1804