change(db): Make the first stable release forward-compatible with planned state changes

[teor2345]

Motivation

We're going to change how the state database is implemented in ticket #6642, after the first stable release.

It would be nice if all stable Zebra versions could read those states, even if they have some format changes.

Close #6746

Specifications

https://docs.rs/rocksdb/latest/rocksdb/struct.DBCommon.html#method.list_cf
https://docs.rs/rocksdb/latest/rocksdb/struct.DBIteratorWithThreadMode.html

Complex Code or Requirements

This is all fairly straightforward, the tricky part will be tickets #6642 and #4784, where we actually change the state format.

Solution

• Add a minor and patch version to the state disk database format, and log compatibility at startup
• Open all column families in the database, including ones added by future Zebra versions (compatibility with Tracking: Update lightwalletd RPCs for "spend before sync" algorithm #6642)
• If there isn't a new note commitment tree for a height, look for the previous tree (compatibility with Store only the first tree state in each series of identical tree states #4784)

Testing

These changes are covered by existing state and RPC tests.

I don't think we should have specific forward compatibility tests, we can just support it on a best-effort basis.

Review

This is a routine change that we'd like to get in the first stable release.

Reviewer Checklist

• Will the PR name make sense to users?
  • Does it need extra CHANGELOG info? (new features, breaking changes, large changes)
• Are the PR labels correct?
• Does the code do what the ticket and PR says?
  • Does it change concurrent code, unsafe code, or consensus rules?
• How do you know it works? Does it have tests?

Follow Up Work

Actually change the state format.

[codecov]

Codecov Report

Merging #6813 (050a08b) into main (628ddd6) will increase coverage by 0.18%.
The diff coverage is 85.90%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6813      +/-   ##
==========================================
+ Coverage   77.97%   78.15%   +0.18%     
==========================================
  Files         308      308              
  Lines       41023    41113      +90     
==========================================
+ Hits        31988    32133     +145     
+ Misses       9035     8980      -55     

[oxarbitrage]

This is very clear, thank you for making it that way. Looks great.

[teor2345]

This PR is ready for re-review

[teor2345]

Unrelated failure, possibly #6763 but in the RPC code:

{"app":"lightwalletd","error":"Post "[http://127.0.0.1:41291](http://127.0.0.1:41291/)": read tcp 127.0.0.1:45786-\u003e127.0.0.1:41291: read: connection reset by peer","level":"fatal","msg":"error zcashd getbestblockhash rpc","time":"2023-06-06T01:10:56Z"}

https://github.com/ZcashFoundation/zebra/actions/runs/5182998267/jobs/9341002724?pr=6813#step:8:426

[teor2345]

@Mergifyio update

[mergify]

update

✅ Branch has been successfully updated