Testnet 70, Deimos 6, The Upgrading

[conorsch]

Testnet Release

We are planning two consecutive releases, v0.70.0 and v0.71.0, in order to exercise chain upgrade logic (#1804). Accordingly, we will need to vote on a upgrade proposal to halt the chain, and provide documentation to node operators on the specific upgrade steps.

Testnet name: Deimos 6
Release date: 2024-03-26
Testnet release manager: @conorsch

Testnet name: Deimos 6 [sic]
Release date: 2024-03-28
Testnet release manager: @conorsch

Testnet Release Manager Checklist

Friday before release:

• Construct the genesis data for the release:

  • Create new testnet directory with initial genesis allocations for this testnet by running cd testnets && ./new-testnet.sh
    • This genesis data will be used for testnet-preview with a randomized version of the future testnet's chain ID.

• Check in with team again in a release meeting and update the GitHub milestone to ensure it represents what will make it into the testnet.

• Draft an announcement for peer review to ensure major changes included are comprehensive.

Following Tuesday (release day):

• Verify that testnet-preview.penumbra.zone is operational; it is redeployed on every push to main, and is an exact preview of what is about to be deployed.
• Bump the version number and push its tag, via cargo-release.
  • Run cargo release minor for a new testnet, or cargo release patch for a bugfix. For the latter, make sure you're on a dedicated release branch.
  • Push the commit and newly generated tag, e.g. v0.51.0, to the remote.
• Wait for the "Release" workflow to complete: it'll take ~60m, most of which is the Windows builds.
• Edit the newly created (and published) release object, then click "Generate release notes." Cut and paste the generated text from the bottom to the top of the post, then save it.
• You must manually review the Waiting deployment in the GitHub Actions UI before the deployment will begin. Monitor the GitHub action to ensure it completes after it is approved.
• Delegate to the Penumbra Labs CI validators; use amounts of ~200k penumbra per validator.
• Open a position to provide some initial liquidity: pcli tx lp replicate xyk penumbra:test_usd 20000penumbra --current-price 20
• Update Galileo deployment, following docs.
• Make the announcement to Discord! 🎉🎉🎉
• Submit proposal for upgrade at height xxxxx (approx 48h after first release)
• Encourage others to delegate to validators, so voting has weight

Following Wednesday (vote day):

• Check proposal for passing status. If failed, resubmit and revote.
• Announce in Discord the chain height that network will halt. Share docs on upgrade steps.

Following Thursday (second release day):

• Disable the testnet deploy workflow, because we don't want to reset the chain.
• Do not prepare new testnet dir with unique genesis, since we are preserving the existing network state.
• Make sure to merge a change incrementing TOTAL_HALT_COUNT, so that nodes can resume post-upgrade. app: increment halt counter #4127
• Tag v0.71.0 as usual, directly on main. Push, and wait for release workflows (so new artifacts are published).
• Manually upgrade cluster, following steps in https://github.com/penumbra-zone/penumbra/wiki/Performing-upgrades
• Make sure to emit an --export-archive from at least one node. Publish that archive to https://snapshots.penumbra.zone/.
• Make sure to emit a --migrate-archive from at least one node. Publish that archive to https://snapshots.penumbra.zone/.
• Make sure to record the genesis-start flag from the upgrade and use it consistently. It must land in the docs, as well.
• When all PL-run testnet nodes are running again, announce that.