State management roadmap

[Mirko-von-Leipzig]

This is an outline of the short, medium and long term roadmap for pathfinder with regards to state management and block sync. This will be short on specifics, especially for the medium and long term. The main impetus for this is the inclusion of p2p as a data source.

Short term

Replace each gateway query with an equivalent p2p one (fallback to gateway). Use gossip'd blocks as a way of tracking the tip of chain in the L2 sync process (as a way for it to target where it wants to get to).

Medium term

Decouple canonical tracking from raw data input. This will let us process block headers for example, without waiting on the state diffs and vice versa.

Allow for starting from block N without requiring the previous blocks.

Both of these will massively speed up the initial sync process.

Long term

This is more a dream list, and less about how to achieve this.

Completely refactor the state management to become far more flexible and decoupled. Enable things like swapping out full state for only the state of certain contracts / classes. In short; become far more configurable. This would allow for creating archive, full-node, light clients, arbitrary bits and bobs in a simple, but understandable fashion.

This would require a new system architecture. One potential solution is discussed in #713, but it is not clear yet if this is solid enough.