Some nodes do not store rollup headers in host DB

[joeldudley]

I modified func (db *DB) AddRollupHeader(header *common.Header, txHashes []gethcommon.Hash) { to add some logging to check whether a rollup's transactions were being stored:

func (db *DB) AddRollupHeader(shortId uint64, header *common.Header, txHashes []gethcommon.Hash) {
	b := db.kvStore.NewBatch()
	writeRollupHeader(b, header)
	writeRollupHash(b, header)
	for _, txHash := range txHashes {
		log.Info("node %d writing tx %s under number %d", shortId, txHash, header.Number)
		writeRollupNumber(b, txHash, header)
}
...

You can see in the attached logs that for some transactions, one of the nodes does not store them.

For example, if you search the logs for the line "writing tx 0xf5aa419a62c27c54c4a4323df41e7387d60508e3d44e5b50da3f9194b2770edd under number", you'll see that node 0 does not store this transaction.

Weirdly, node 0 does store other transactions for that rollup, just not that specific one. See the following log lines - node 0 only stores a single transaction for that rollup, where the other nodes store three.

Aug  2 19:59:32.024 INF node 0 writing tx 0x38e1408bb7e3a37dd9b7a0b52c8ab8120117b73cf6d12e93cc200b2c33ff214a under number 10
Aug  2 19:59:32.026 INF node 2 writing tx 0xf5aa419a62c27c54c4a4323df41e7387d60508e3d44e5b50da3f9194b2770edd under number 10
Aug  2 19:59:32.026 INF node 2 writing tx 0x7f9af304ba35a4c517eb531213483d6736577197608af937375dea27f6b00d7a under number 10
Aug  2 19:59:32.026 INF node 2 writing tx 0x38e1408bb7e3a37dd9b7a0b52c8ab8120117b73cf6d12e93cc200b2c33ff214a under number 10
Aug  2 19:59:32.027 INF node 4 writing tx 0xf5aa419a62c27c54c4a4323df41e7387d60508e3d44e5b50da3f9194b2770edd under number 10
Aug  2 19:59:32.027 INF node 4 writing tx 0x7f9af304ba35a4c517eb531213483d6736577197608af937375dea27f6b00d7a under number 10
Aug  2 19:59:32.027 INF node 4 writing tx 0x38e1408bb7e3a37dd9b7a0b52c8ab8120117b73cf6d12e93cc200b2c33ff214a under number 10
Aug  2 19:59:32.027 INF node 3 writing tx 0xf5aa419a62c27c54c4a4323df41e7387d60508e3d44e5b50da3f9194b2770edd under number 10
Aug  2 19:59:32.027 INF node 3 writing tx 0x38e1408bb7e3a37dd9b7a0b52c8ab8120117b73cf6d12e93cc200b2c33ff214a under number 10
Aug  2 19:59:32.027 INF node 3 writing tx 0x7f9af304ba35a4c517eb531213483d6736577197608af937375dea27f6b00d7a under number 10
Aug  2 19:59:32.027 INF node 1 writing tx 0xf5aa419a62c27c54c4a4323df41e7387d60508e3d44e5b50da3f9194b2770edd under number 10
Aug  2 19:59:32.028 INF node 1 writing tx 0x7f9af304ba35a4c517eb531213483d6736577197608af937375dea27f6b00d7a under number 10
Aug  2 19:59:32.028 INF node 1 writing tx 0x38e1408bb7e3a37dd9b7a0b52c8ab8120117b73cf6d12e93cc200b2c33ff214a under number 10

We can also see from the logs that the rollup appears to be published and win twice - once by node 0 with only a single transaction, and once by node 4 with many transactions:

Aug  2 19:59:31.354 INF >   Agg0: Publish rollup=r_1985238746473799162(10)[r_3446013531048649987]{proof=b_9857441080405350445(33)}. Num Txs: 1. Txs: [0x38e1408bb7e3a37dd9b7a0b52c8ab8120117b73cf6d12e93cc200b2c33ff214a,].  Root=0x2b1431b71eef3c861f0f5371f83a7ccbf6bbf629f7ed0612a2235d69ba12dc24. 
Aug  2 19:59:31.354 INF >   Agg0: Winner (b_9857441080405350445) r_1985238746473799162(10).
Aug  2 19:59:31.360 INF >   Agg4: Publish rollup=r_8376691643348898319(10)[r_3446013531048649987]{proof=b_9857441080405350445(33)}. Num Txs: 4. Txs: [0xe5c440ba47df76463c9172617d263fee7225dbb7c4b9f2ab33174e458b1a4d04, 0xf5aa419a62c27c54c4a4323df41e7387d60508e3d44e5b50da3f9194b2770edd, 0x7f9af304ba35a4c517eb531213483d6736577197608af937375dea27f6b00d7a, 0x38e1408bb7e3a37dd9b7a0b52c8ab8120117b73cf6d12e93cc200b2c33ff214a,].  Root=0x611e654e5e5f83b30a89a7e2a15ead03068d4aaae302de5e53d9da28b059fd38. 
Aug  2 19:59:31.360 INF >   Agg4: Winner (b_9857441080405350445) r_8376691643348898319(10).

These logs were produced by running the full network sim.

sim-log-2022-08-02_19-58-54-full-network-1930208033.txt