Minimal state rollovers

[hdevalence]

Currently, each of our testnets has a completely fresh state, independent of any state from a previous testnet. This has worked well enough for development so far, but now that we have more users interested in the project, it would be friendlier not to erase their state. It would also be useful to start working on having a meaningful upgrade path.

In the long term, it would be ideal to be able to faithfully serialize the entire application state into the genesis.json file for a new chain, but in the short term (today and tomorrow, before launching the Orthosie testnet) it would be useful to figure out the bare minimum viable state rollover needed to preserve users' balances.

Because users should be able to sync their state from the chain, this means that we have to make (a minimal subset of) the old chain data available for syncing.

I think this looks like:

• a list of StateFragments recording every existing note;
• a list of nullifiers recording which of them have been spent;
• a list of denominations (asset id preimages).

Clients would need to scan through this data to sync the chain.

As an MVP, we could potentially just insert all of the existing notes and nullifiers into the application state when processing the genesis file. With no other changes, this would mean that the first CompactBlock received during chain scanning would have a summary of the entire previous chain state.

I think this would be fine for now, since we have relatively little data (I checked this morning[^1] and there are only 55k notes; multiplying by ~200 bytes per state fragment gives ~10MB, which is on the upper end of reasonable), but in the longer term we might want to explore other options.

Three that come to mind:

1. Queue up the old state fragments somehow and slot them into blocks of the new chain as it runs forward (e.g., replaying 1000 at a time or something);
2. Cut them into chunks (say 1000 at a time or something), creating "phantom blocks" that only the app knows about, and reporting the next height as the initial height of the new chain.
3. While exporting, attach the heights to the notes and nullifiers, and then use the data in the genesis file to create phantom blocks mirroring the old chain, reporting the initial height of the new chain as the highest height of any of the rolled-over data.

I don't really like either of those, since they both feel kind of hacky. We ideally don't want to be creating a bunch of phantom chain data that doesn't come from a (Tendermint) block (Related: #374).

[hdevalence]

From the Tendermint docs:

initial_height: Height at which Tendermint should begin. If a blockchain is conducting a network upgrade, starting from the stopped height brings uniqueness to previous heights.

We use this with approach (3) to ensure that the historical blocks don't conflict with new blocks.

[hdevalence]

We should do this before going to mainnet, but there's no particular rush to do it, and it may be better to wait, since we won't have to update the state rollover code as we evolve our state, if we don't have any state rollover code.

[hdevalence]

@erwanor Is this issue duplicated by the issues that are part of the upgrades work? Can we close it out?

[erwanor]

Superseded by #1804 and in particular #3081