Implementation of the state rewind feature for the RocksDB

[xgreenx]

Closes #451

Overview

Added support for the state rewind feature. The feature allows the execution of the blocks in the past and the same execution results to be received. Together with forkless upgrades, execution of any block from the past is possible if historical data exist for the target block height. The default size of historical/rewind window is 7 days.

Also added support for rollback command when state rewind feature is enabled. The command allows the rollback of the state of the blockchain several blocks behind until the end of the historical window.

Implementation details

The change adds a new HistoricalRocksDB type that is the wrapper around regular RocksDB. This type has inside another RocksDB instance that is used to duplicate all tables plus has one more column to store the reverse modifications at each block height. The reverse modification is the opposite to the operation that was done during transition from block height X to X + 1. The screenshot below should describe the idea:

The key of duplicated tables is extended with block height, and the value is the reverse operation to reach the state of entry at the previous height. Having the history of reverse operations, we can iterate back from the latest version of the entry to the previous one.

Using the main property of the RocksDB(sorting keys by default), lookup operations are fast and we don't need to iterate all modifications. It is just enough to find the nearest reverse operation to the target height.

Checklist

• New behavior is reflected in tests

Before requesting review

• I have reviewed the code myself
• I have created follow-up issues caused by this PR and linked them here
  • Optimize iteration over the keys in the database #1997
  • Support historical view for MemoryStore #1995
  • Add caching of the views on the Database level #1993

[Dentosal]

Looks like this might also have breaking changes, at least some pub fields were made pub(crate).

[MitchTurner]

If they were added since the last release, then it should be fine.

[MitchTurner]

If they were added since the last release, then it should be fine.

[MitchTurner]

nit: "While not is_cancelled" is kinda a confusing double negative. Maybe add a helper that is just like

fn is_still_live(&self) -> bool {
    !self.is_cancelled()
}

[MitchTurner]

Maybe we rename ci_checks.sh since this isn't "checking" anymore?

[MitchTurner]

Hmm. When would we want to use this for DaBlockHeight? For example, we default the relayer to not have rollback.

[Voxelot]

I see some risks to this proliferating pattern of creating many distinct rocksdb instances:

• We lose ACID properties, i.e. issues like the offchain and onchain db getting out of sync can now also happen with the historical changes getting out of sync. What protections are in place to safeguard us from a situation where the changes are committed to the historical database but not to the primary database? I'm concerned that we will introduce new complicated edge cases that are hard to resolve if we don't properly encapsulate all the changes into atomic database transactions, leading to brittleness under adverse conditions.
• Resource contention overhead such as too many open files and more complexity in configuring rocksdb cache sizes. For example, should all rocksdb instances have equal cache limits? It seems like to me, we should allocate more cache to the main databases vs the historical databases during regular block production or syncing etc. However, if an RPC node gets lots of heavy traffic for historical views compared to the amount of work it needs to do to sync, then more cache should be allocated for historical changes. It's hard to predict which db instances would need more caching as it will depend on the usecase. If we didn't have separate DB's then rocksdb would automatically make the most of the cache capacity allocated based on whatever usage scenario is thrown at it.

While I understand that the onchain and offchain updates are not atomic as they are driven by separate processes, I'm wondering if we could make the historical changes part of the same database that they are tracking the changes of. Given that the only way to recover from the offchain and onchain db's falling out of sync is to replay historical data, shouldn't we try to make the historical data as rock solid as possible given that it's part of the same CombinedDatabase::commit_changes function which should be atomic?

[xgreenx]

We lose ACID properties, i.e. issues like the offchain and onchain db getting out of sync can now also happen with the historical changes getting out of sync. What protections are in place to safeguard us from a situation where the changes are committed to the historical database but not to the primary database? I'm concerned that we will introduce new complicated edge cases that are hard to resolve if we don't properly encapsulate all the changes into atomic database transactions, leading to brittleness under adverse conditions.

This situation is handled by committing to the historical database first and only afterward to the wrapped database. We can always roll back the history database to the same height as the wrapped database.

Below an example of how we process this case while committing new block height:

Resource contention overhead such as too many open files and more complexity in configuring rocksdb cache sizes. For example, should all rocksdb instances have equal cache limits? It seems like to me, we should allocate more cache to the main databases vs the historical databases during regular block production or syncing etc. However, if an RPC node gets lots of heavy traffic for historical views compared to the amount of work it needs to do to sync, then more cache should be allocated for historical changes. It's hard to predict which db instances would need more caching as it will depend on the usecase. If we didn't have separate DB's then rocksdb would automatically make the most of the cache capacity allocated based on whatever usage scenario is thrown at it.

I agree with these concerns. We need to improve the configurability of these parameters. I think it should be possible to store historical data in the same database. I've split it into two databases to have better control over the database configuration and be able to prune historical data just by deleting the folder without additional tools.

I could try to use only one database.

[Voxelot]

Do you think merging into one db can be a follow-up? Or should we handle this before landing the feature since it'll be disruptive to the database schema with two back to back big db changes?

[Voxelot]

Should we do some kind of consistency check here? There could be a race condition where the latest view is not in sync with the historical database when the snapshots are created.

[xgreenx]

This situation is handled on the GenericDatabase level. When we commit to the database, we lock the current block height.

The same is in the case of view_at. We lock the block height so the view can only be created outside of the commit.

[Voxelot]

Should we return an error here instead? We should be extra cautious about adding any panics