db: state snapshots

[canepat]

One of the main features of Erigon3 is moving most of the state content (i.e. account/storage/code) and its validity proof (i.e. commitment) to snapshot flat files. This is accomplished by introducing new abstractions for modelling the common requirements these entities have in terms storage, starting from the kind of queries they must support.

Obviously, some refactoring of the existing snapshot support is required in order to accommodate the new abstractions in the cleanest, most coherent and ultimately most useful way.

This is the main tracking issue. Individual issues are listed below:

• b+tree index implementation db: add BTree index for kv snapshots #2367
• bloom filter and existence index db: Bloom filter and existence index for kv snapshots #2386
• layering: separation between dal and datastore in db module db: datastore layer #2416 snapshots: new folder structure #2424 snapshots: KV segments #2427
• decouple SnapshotType from SnapshotPath and generalisation of SnapshotPath snapshots: SnapshotPath.step_range #2419
• definition of timed-series unit (e.g. Timestamp) and move to dal package db: datastore layer #2416 snapshots: SnapshotPath.step_range #2419
• refactoring of Snapshot, SnapshotReader as Segment, SegmentReader snapshots: new folder structure #2424
• introduce KVSegment: 3 types of iterators - pairs, keys, values
• schema definition
  • entity identifier type datastore: DataStore and related refactorings #2467 datastore: EntityName replaces SnapshotType #2491
  • decouple schema (i.e. SnapshotBundleData) from bundle and move to dal package datastore: SnapshotBundle schema #2484
  • use fluent config (a.k.a. builder) pattern for DDL-like definition of entities datastore: SnapshotBundle schema #2484
• index support evolution
  • use specific type directly as index (e.g. RecSplitIndex) and remove Index class
  • refactor IndexBuilder: multiple methods for each type or multiple builders
  • Index reader: new abstraction like Segment which decouples K/I serialization
• snapshot repository
  • refactor to work with any timestamp (BlockNum, TxStep) snapshots: timestamp-based repository #2437
  • refactor to support multiple snapshot repository instances db: provide SnapshotRepository to DataModel #2440
  • 2 instances as starting point: block snapshots and state snapshots datastore: DataStore and related refactorings #2467 datastore: EntityName replaces SnapshotType #2491
• state repository schema datastore: state repository schema #2494
• new queries
  • analysis to figure out which queries are needed for DHII ("query objects" pattern)
  • helper to create queries (i.e. factory)
• Inverted Index (II)
• History (H)
• Domain (D)
• Commitment (C)