fix(security): fix concurrency issues in tree key formats, and CPU usage in genesis tree roots

[teor2345]

Motivation

Older Zebra versions did not cache the genesis tree roots.
Zebra currently does a lot of unnecessary writes to the anchors, sprout trees, and history trees.

Since block validation, mempool validation, and RPCs can read these trees, and block validation writes them, this is a potential remotely-triggered CPU or disk denial of service.

Part of #7664, alternative strategy 1 (stop deleting keys).
This is also part of a permanent fix for bug #7618.
Adds docs for part of #7737.

Solution

Improve the database size and performance, particularly during attacks via RPCs, mempool validation, or multiple block writes.

Format Upgrades

• Change the sprout and history key format from tip_height to () (empty key) in the code and docs
• Read the sprout, sapling and orchard genesis trees, then write their cached roots back to the database, and update the docs
• Remove genesis-specific tree code, which uses the wrong format
• Update tests now the genesis block has an entry for an empty tree (rather than using default() which is an empty tree)
• Increment the state minor version

Format Tests

• Check for incorrect sprout or history key formats, and duplicate tip trees
• Check that there aren't any un-cached tree roots for sprout, sapling, and orchard
• Update database snapshot tests
• Disable format validity checks in some tests that create invalid states

Refactors

• Refactor sprout, sapling, and orchard tree and anchor writes so they are all skipped if the trees are the same
  • The history tree changes with every block, so we don't do this check for it
• Refactor sprout, sapling and orchard tree and anchor writes so they automatically cache roots (and use the newer key format)
• This format upgrade is short, so some tests needed changes to wait for a short upgrade, rather than a long upgrade
• Fix metrics for the genesis block

Documentation

• Document the underlying performance bug that this PR fixes, by adding a section to the state upgrades doc.

Review

This is a routine bug fix.

This is a database format change, so it needs 2 reviews.

Reviewer Checklist

• Will the PR name make sense to users?
  • Does it need extra CHANGELOG info? (new features, breaking changes, large changes)
• Are the PR labels correct?
• Does the code do what the ticket and PR says?
  • Does it change concurrent code, unsafe code, or consensus rules?
• How do you know it works? Does it have tests?

[arya2]

lightwalletd_wallet_grpc_tests failed because it marked the database format as upgraded before opening the RPC endpoint.

[teor2345]

lightwalletd_wallet_grpc_tests failed because it marked the database format as upgraded before opening the RPC endpoint.

This happened before the other way around, let me work on a better fix.

[teor2345]

lightwalletd_wallet_grpc_tests failed because it marked the database format as upgraded before opening the RPC endpoint.

@arya2 your (deleted) suggestion here was really good and I ended up implementing it 🙂

It would be nice to have a ~expect_stdout_unordered_lines_match() method in TestChild for these cases.

[teor2345]

We've had two reviews on the production code, and I've just changed docs and test code. So I've taken the extra-reviews label off.