Regtest genesis block timestamp is newer than subsequent blocks upon restart.

[dan-da]

To reproduce:

1. Start a regtest node from genesis and run it for a few minutes to generate some blocks.

$ cargo run -- --data-dir /tmp/regtest_issue --mine --unrestricted-mining --network regtest

2. Display block 0 (genesis) and block 1. (note: some fields removed for clarity)

note: requires 3e21a69 or newer for neptune-cli

$ neptune-cli block-info height/0
height: 0
digest: 04935d4fe177d61cd925daeb05144be4bfeb25115ed5ae1c96d4ebcd4028833d11aae0cc3d23a002
timestamp: 2024-05-06T15:40:00-07:00
is_genesis: true


$ neptune-cli block-info height/1
height: 1
digest: 78c3514623288b6adad670fc8ca5e452486f7257a98d049fdf564b5c578787807791999ef5fbdd52
timestamp: 2024-05-06T15:45:30.412-07:00
is_genesis: false

4. Notice that the timestamp for block 0 pre-dates that of block 1. In this case by 5 minutes. In actuality, they were mined nearly in the same second, but the genesis block gets rounded down to nearest 10 minutes.

5. shut it down. ctrl-c or neptune-cli shutdown and note the time.

6. wait 10 minutes, at least.

7. restart. no need to mine this time.

$ cargo run -- --data-dir /tmp/regtest_issue --network regtest

8. Display block 0 (genesis) and block 1. (again).

$ neptune-cli block-info height/0
height: 0
digest: bf4b52a6d289682fa6a3c1cd9ffbd52949f3dcea3082bb34721ca6c25dcd6db7edd1d63349aadd59
timestamp: 2024-05-06T16:00:00-07:00
is_genesis: true


$ neptune-cli block-info height/1
height: 1
digest: 78c3514623288b6adad670fc8ca5e452486f7257a98d049fdf564b5c578787807791999ef5fbdd52
timestamp: 2024-05-06T15:45:30.412-07:00
is_genesis: false

Note that the genesis timestamp is now larger than block 1. Also, the genesis digest has changed since the first time we ran neptune-core. (bf4b52a6d.. != 04935d4fe)

Analysis

This is an unintended consequence of using a dynamic timestamp for RegTest when creating the genesis block. This timestamp is floored down to the nearest 10 minutes.

The genesis block is never stored in a DB (afaik). As such, upon any restart after 10 minutes, a new genesis block is created.

Surprisingly, this situation is not detected as an error and the rest of the blockchain is loaded. The end result is that we have an invalid blockchain, with block1's parent-hash being different from the (new) genesis block.

I see at least two problems here:

1. We should be detecting this problem upon restart. Somehow validations are failing.
2. imho, we should not be using a dynamic timestamp for regtest. Bitcoin does not, and I suggest we follow their example in this case.
3. if we are going to use dynamic timestamp, then at minimum we must make certain that genesis block gets stored in on-disk block-db and loaded from it correctly.

For reference, here is the start of my log from the 2nd run. Note that there are no warnings or errors, and we load block 19 (tip) just fine.

    Running `target/debug/neptune-core --data-dir /tmp/regtest_issue --network regtest`
2024-05-06T23:01:13.289644955Z  INFO ThreadId(01) neptune_core: Data directory is /tmp/regtest_issue/regtest
2024-05-06T23:01:13.289849421Z  INFO ThreadId(01) neptune_core::models::state::wallet: ***** Reading wallet from /tmp/regtest_issue/regtest/wallet/wallet.dat *****



2024-05-06T23:01:13.290019973Z  INFO ThreadId(01) neptune_core: Now getting wallet state. This may take a while if the database needs pruning.
2024-05-06T23:01:13.302282658Z  INFO ThreadId(01) neptune_core: Got wallet state.
2024-05-06T23:01:13.308683924Z  INFO ThreadId(01) neptune_core: Got block index database
2024-05-06T23:01:13.313152684Z  INFO ThreadId(01) neptune_core: Got peer database
2024-05-06T23:01:13.319768333Z  INFO ThreadId(01) neptune_core: Got archival mutator set
2024-05-06T23:01:13.399069301Z  INFO ThreadId(01) neptune_core: Now listening for incoming transactions
2024-05-06T23:01:13.399398108Z  INFO ThreadId(01) neptune_core: Most known canonical block has height 19
2024-05-06T23:01:13.399426512Z  INFO ThreadId(01) neptune_core: Checking if we need to restore UTXOs
2024-05-06T23:01:13.40313687Z  INFO ThreadId(01) neptune_core::models::state: Checking 19 incoming UTXOs
2024-05-06T23:01:13.416758918Z  INFO ThreadId(01) neptune_core::models::state: No missing monitored UTXOs found in wallet database. Wallet database looks good.
2024-05-06T23:01:13.416805463Z  INFO ThreadId(01) neptune_core: UTXO restoration check complete

[Sword-Smith]

I ran into a related problem when running run-multiple-nodes.sh, waiting 10 minutes, then running run-multiple-nodes-advanced.sh. This triggers the DifferentGenesis behavior of the client which closes connections.

1. I disagree that we should be detecting the problem. The client assumes that the database and disk content is correct. The policy is that if it's not correct, it doesn't get stored.

This is not a huge issue in my opinion. Regtest got a dynamic launch data value since this allows us to test time locks on transactions better, without mocking what the system time is. I'm changing the time resolution for RegTest to 10 hours, making these problems less likely to occur.

[dan-da]

Let's move block validation on startup discussion to #146.

I agree regtest digest changing is not top priority. Though I do hope we make a fixed regtest genesis block before launch, to avoid these kinds of confusions/issues.

For reference, I wanted to be certain that bitcoin-core has a fixed regtest genesis block, so I asked about it on stackexchange, and it does. We don't always have to follow bitcoin's lead, but I think this issue is pointing out there is a good reason for it, and also the fact that different apps will generate different regtest from eachother, such as I initially encountered with the block-explorer.

[Sword-Smith]

RegTest is designed for local testing, like on a local network or in the integration tests that we have, like run-multiple-instances-from-genesis.sh. If you want to run tests in a wider setting where you connect to nodes you do not control yourself, you should be using TestNet instead.

Now that we've changed RegTest's time resolution from 10 minutes to 10 hours I think that solves some of our problems. Maybe you shouldn't be running the block explorer against a RegTest node but rather a TestNet node?