[e3] BlockReader HeaderByHash panic on sync from scratch

[taratorio]

System information

Erigon version: check commit hash

OS & Version:Linux

Commit hash: 2ed4f88

Erigon Command (with flags/config):

nohup ~/erigon/build/bin/erigon --chain=bor-mainnet --bor.heimdall=http://0.0.0.0:1317 --sync.loop.block.limit=10_000 --batchSize=2g --datadir=/erigon-data/bor-heimdall-stage-mainnet --nat=extip:54.38.13.192 --metrics=true --metrics.addr=0.0.0.0 --metrics.port=8999 --pprof=true --pprof.port=6160 --txpool.disable=true --log.console.verbosity=debug &> /erigon-data/bor-heimdall-stage-mainnet/console.log &

Consensus Layer: N/A

Consensus Layer Command (with flags/config): N/A

Chain/Network: Bor mainnet

Expected behaviour

No panics when serving getBlockHeaders66

Actual behaviour

BlockReader HeaderByHash panics

Steps to reproduce the behaviour

Run a fresh sync from scratch

Backtrace

[DBUG] [09-21|14:35:15.748] [1/9 OtterSync] DONE                     in=21.49µs
[INFO] [09-21|14:35:15.748] [2/9 Headers] Waiting for headers...     from=62100028 hash=0x97271dc4392858ca4abbb76ea2e1b51425caa975c36ae71b436ffe1498688763
[DBUG] [09-21|14:35:15.748] [2/9 Headers] Requested skeleton         from=62100027 length=192
[DBUG] [09-21|14:35:16.863] [2/9 Headers] Requested skeleton         from=62100027 length=192
[INFO] [09-21|14:35:17.245] [2/9 Headers] Processed                  highest=62100029 age=0 headers=1 in=1.496 blk/sec=0
[DBUG] [09-21|14:35:17.245] [2/9 Headers] DONE                       in=1.496204567s
[INFO] [09-21|14:35:17.245] [3/9 BorHeimdall] Processing sync events... from=62100029 to=62100029
[INFO] [09-21|14:35:17.299] [3/9 BorHeimdall] Sync events            progress=62100029 lastSpanID=9704 lastSpanID=9704 lastCheckpointId=0 lastMilestoneId=0 lastStateSyncEventID=2963389 total records=0 sync event time=1.239283ms fetch time=0s snap time=106.215062ms waypoint time=0s process time=54.427075ms
[DBUG] [09-21|14:35:17.299] [3/9 BorHeimdall] DONE                   in=54.462745ms
[DBUG] [09-21|14:35:17.299] [4/9 BlockHashes] DONE                   in=52.08µs
[DBUG] [09-21|14:35:17.319] [index] created                          file=/erigon-data/bor-heimdall-stage-mainnet/snapshots/v1-062098-062099-transactions.idx.tmp
[DBUG] [09-21|14:35:17.319] [index] calculating                      file=v1-062098-062099-transactions.idx
[DBUG] [09-21|14:35:17.320] [5/9 Bodies] DONE                        in=21.233833ms
[DBUG] [09-21|14:35:17.333] [6/9 Senders] DONE                       in=12.799711ms
[DBUG] [09-21|14:35:17.568] [index] write                            file=v1-062098-062099-transactions.idx
[DBUG] [09-21|14:35:17.572] [index] created                          file=/erigon-data/bor-heimdall-stage-mainnet/snapshots/v1-062098-062099-transactions-to-block.idx.tmp
[DBUG] [09-21|14:35:17.572] [index] calculating                      file=v1-062098-062099-transactions-to-block.idx
[INFO] [09-21|14:35:17.687] [7/9 Execution] Done Commit every block  blk=62100029 blks=1 blk/s=2.8 txs=73 tx/s=206 gas/s=69.42M buf=7.0MB/2.0GB stepsInDB=0.00 step=2925.8 alloc=22.0GB sys=58.2GB
[DBUG] [09-21|14:35:17.687] [7/9 Execution] DONE                     in=354.288716ms
[DBUG] [09-21|14:35:17.689] [8/9 TxLookup] DONE                      in=1.420933ms
[DBUG] [09-21|14:35:17.689] [9/9 Finish] DONE                        in=10.86µs
[DBUG] [09-21|14:35:17.689] RPC Daemon notified of new headers       from=62100028 to=62100030 amount=1 header sending=7.95µs
[INFO] [09-21|14:35:17.689] Timings (slower than 50ms)               Headers=1.496s BorHeimdall=54ms Execution=354ms alloc=22.1GB sys=58.2GB
[DBUG] [09-21|14:35:17.689] [9/9 Finish] Prune done                  in=6.72µs
[DBUG] [09-21|14:35:17.690] [8/9 TxLookup] Prune done                in=40.16µs
[DBUG] [09-21|14:35:17.690] [7/9 Execution] Prune done               in=692.412µs
[DBUG] [09-21|14:35:17.690] [6/9 Senders] Prune done                 in=3.27µs
[DBUG] [09-21|14:35:17.690] [5/9 Bodies] Prune done                  in=2.21µs
[DBUG] [09-21|14:35:17.690] [4/9 BlockHashes] Prune done             in=4.21µs
[DBUG] [09-21|14:35:17.690] [3/9 BorHeimdall] Prune done             in=1.57µs
[DBUG] [09-21|14:35:17.690] [2/9 Headers] Prune done                 in=900ns
[DBUG] [09-21|14:35:17.690] [snapshots] Prune Blocks                 to=62098000 limit=10
[DBUG] [09-21|14:35:17.690] [snapshots] Prune Bor Blocks             to=62098000 limit=10
[DBUG] [09-21|14:35:17.690] [1/9 OtterSync] Prune done               in=49.95µs
[DBUG] [09-21|14:35:17.759] [1/9 OtterSync] DONE                     in=21.1µs
[INFO] [09-21|14:35:17.759] [2/9 Headers] Waiting for headers...     from=62100029 hash=0x7b5228ba6f28ba567f2777064d3d2be36ac1c85a7dd604af94efd38118a68f30
[DBUG] [09-21|14:35:17.760] [2/9 Headers] Requested skeleton         from=62100028 length=192
[DBUG] [09-21|14:35:17.843] [index] write                            file=v1-062098-062099-transactions-to-block.idx
unexpected fault address 0x720d28534d4c
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x720d28534d4c pc=0xb53acd]

goroutine 2621 gp=0xc0056e8c40 m=63 mp=0xc1ea4fb008 [running]:
runtime.throw({0x2a4d5b3?, 0x27?})
        runtime/panic.go:1023 +0x5c fp=0xc0487436d0 sp=0xc0487436a0 pc=0x46125c
runtime.sigpanic()
        runtime/signal_unix.go:895 +0x285 fp=0xc048743730 sp=0xc0487436d0 pc=0x47a065
github.com/erigontech/erigon-lib/seg.(*Getter).nextPos(0x7211e739d9d8?, 0xe0?)
        github.com/erigontech/[email protected]/seg/decompress.go:578 +0xad fp=0xc048743750 sp=0xc048743730 pc=0xb53acd
github.com/erigontech/erigon-lib/seg.(*Getter).Next(0xc0487438b0, {0xc7b582bb80, 0x0, 0x80})
        github.com/erigontech/[email protected]/seg/decompress.go:689 +0xaf fp=0xc048743820 sp=0xc048743750 pc=0xb5412f
github.com/erigontech/erigon/turbo/snapshotsync/freezeblocks.(*BlockReader).headerFromSnapshotByHash(0x72898e6c1330?, {0xcd, 0x65, 0xc4, 0x46, 0x3e, 0x2e, 0xde, 0xea, 0x14, ...}, ...)
        github.com/erigontech/erigon/turbo/snapshotsync/freezeblocks/block_reader.go:912 +0x259 fp=0xc048743970 sp=0xc048743820 pc=0x16c8899
github.com/erigontech/erigon/turbo/snapshotsync/freezeblocks.(*BlockReader).HeaderByHash(0xc00274ac80, {0xf0?, 0xc048743aa8?}, {0x72898e6c1330?, 0xc1907df580?}, {0xcd, 0x65, 0xc4, 0x46, 0x3e, ...})
        github.com/erigontech/erigon/turbo/snapshotsync/freezeblocks/block_reader.go:519 +0x1b2 fp=0xc048743a20 sp=0xc048743970 pc=0x16c53f2
github.com/erigontech/erigon/eth/protocols/eth.AnswerGetBlockHeadersQuery({0x72898fa6a410, 0xc1907df580}, 0xc7b135f400, {0x72898e6c13d0, 0xc00274ac80})
        github.com/erigontech/erigon/eth/protocols/eth/handlers.go:59 +0xa83 fp=0xc048743c30 sp=0xc048743a20 pc=0x1fb24e3
github.com/erigontech/erigon/p2p/sentry/sentry_multi_client.(*MultiClient).getBlockHeaders66.func1({0x72898fa6a410?, 0xc1907df580?})
        github.com/erigontech/erigon/p2p/sentry/sentry_multi_client/sentry_multi_client.go:493 +0x5a fp=0xc048743c80 sp=0xc048743c30 pc=0x2279e9a
github.com/erigontech/erigon-lib/kv/temporal.(*DB).View(0xc79b9cd620?, {0x32bf660?, 0xc0056ec0f0?}, 0xc205299440)
        github.com/erigontech/[email protected]/kv/temporal/kv_temporal.go:103 +0xbc fp=0xc048743ce8 sp=0xc048743c80 pc=0x13d349c
github.com/erigontech/erigon/p2p/sentry/sentry_multi_client.(*MultiClient).getBlockHeaders66(0xc000a64a50, {0x32bf660, 0xc0056ec0f0}, 0xc195a73360, {0x32df020, 0xc001a16000})
        github.com/erigontech/erigon/p2p/sentry/sentry_multi_client/sentry_multi_client.go:492 +0x1cb fp=0xc048743dd8 sp=0xc048743ce8 pc=0x2279aab
github.com/erigontech/erigon/p2p/sentry/sentry_multi_client.(*MultiClient).handleInboundMessage(0x0?, {0x32bf660?, 0xc0056ec0f0?}, 0x0?, {0x32df020?, 0xc001a16000?})
        github.com/erigontech/erigon/p2p/sentry/sentry_multi_client/sentry_multi_client.go:663 +0x5b fp=0xc048743e28 sp=0xc048743dd8 pc=0x227b39b
github.com/erigontech/erigon/p2p/sentry/sentry_multi_client.(*MultiClient).HandleInboundMessage(0xc000a64a50, {0x32bf660, 0xc0056ec0f0}, 0xc195a73360, {0x32df020, 0xc001a16000})
        github.com/erigontech/erigon/p2p/sentry/sentry_multi_client/sentry_multi_client.go:634 +0xa5 fp=0xc048743ec0 sp=0xc048743e28 pc=0x227af65
github.com/erigontech/erigon/p2p/sentry/sentry_multi_client.(*MultiClient).HandleInboundMessage-fm({0x32bf660?, 0xc0056ec0f0?}, 0xc1d7405450?, {0x32df020?, 0xc001a16000?})
        <autogenerated>:1 +0x46 fp=0xc048743f00 sp=0xc048743ec0 pc=0x227c9a6
github.com/erigontech/erigon-lib/p2p/sentry.pumpStreamLoop[...].func2()
        github.com/erigontech/[email protected]/p2p/sentry/loop.go:126 +0x136 fp=0xc048743fe0 sp=0xc048743f00 pc=0x227c596
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc048743fe8 sp=0xc048743fe0 pc=0x49c601
created by github.com/erigontech/erigon-lib/p2p/sentry.pumpStreamLoop[...] in goroutine 213
        github.com/erigontech/[email protected]/p2p/sentry/loop.go:120 +0x2a7