Error pre-warming cache 0xe8dd57491fb264568bcd0d209b62bda19a4062a4f3f38f102bc62ba7c3491479

[dmitriy-b]

Description
After about 1 day mainnet full syncing with prysm consensus we got following error

Error pre-warming cache 0xe8dd57491fb264568bcd0d209b62bda19a4062a4f3f38f102bc62ba7c3491479

Nethermind.Trie.MissingTrieNodeException: Node 0x99a65b14b9efce52cc580f09ee0f2023f5ae7ada48db7e252b73eb107c49317b is missing from the DB
 ---> Nethermind.Trie.TrieNodeException: Failed to load root hash 0x99a65b14b9efce52cc580f09ee0f2023f5ae7ada48db7e252b73eb107c49317b while loading key 8fa687d53b1e3c9c0802a52d3dd680136b5ae82e944dc1b716fa429f610fb0ac.
Node 0x99a65b14b9efce52cc580f09ee0f2023f5ae7ada48db7e252b73eb107c49317b is missing from the DB
   at Nethermind.Trie.PreCachedTrieStore.LoadRlp(Hash256 address, TreePath& path, Hash256 hash, ReadFlags flags)
   at Nethermind.Trie.PatriciaTree.ResolveNode(TrieNode node, TraverseContext& traverseContext, TreePath& path) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 616
   --- End of inner exception stack trace ---
   at Nethermind.Trie.PatriciaTree.ResolveNode(TrieNode node, TraverseContext& traverseContext, TreePath& path) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 628
   at Nethermind.Trie.PatriciaTree.Get(ReadOnlySpan`1 rawKey, Hash256 rootHash) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 384
   at Nethermind.State.PersistentStorageProvider.LoadFromTreeStorage(StorageCell storageCell) in /src/Nethermind/Nethermind.State/PersistentStorageProvider.cs:line 418
   at Nethermind.State.PersistentStorageProvider.LoadFromTree(StorageCell& storageCell) in /src/Nethermind/Nethermind.State/PersistentStorageProvider.cs:line 365
   at Nethermind.Evm.VirtualMachine`1.InstructionSLoad[TTracingInstructions,TTracingStorage](EvmState vmState, EvmStack`1& stack, Int64& gasAvailable, IReleaseSpec spec) in /src/Nethermind/Nethermind.Evm/VirtualMachine.cs:line 2515
   at Nethermind.Evm.VirtualMachine`1.ExecuteCode[TTracingInstructions,TTracingRefunds,TTracingStorage](EvmState vmState, EvmStack`1& stack, Int64 gasAvailable, IReleaseSpec spec) in /src/Nethermind/Nethermind.Evm/VirtualMachine.cs:line 1559
   at Nethermind.Evm.VirtualMachine`1.Run[TTracingActions](EvmState state, IWorldState worldState, ITxTracer txTracer) in /src/Nethermind/Nethermind.Evm/VirtualMachine.cs:line 221
   at Nethermind.Evm.TransactionProcessing.TransactionProcessorBase.ExecuteEvmCall(Transaction tx, BlockHeader header, IReleaseSpec spec, ITxTracer tracer, ExecutionOptions opts, Int64& gasAvailable, ExecutionEnvironment& env, TransactionSubstate& substate, Int64& spentGas, Byte& statusCode) in /src/Nethermind/Nethermind.Evm/TransactionProcessing/TransactionProcessor.cs:line 515
   at Nethermind.Evm.TransactionProcessing.TransactionProcessorBase.Execute(Transaction tx, BlockExecutionContext& blCtx, ITxTracer tracer, ExecutionOptions opts) in /src/Nethermind/Nethermind.Evm/TransactionProcessing/TransactionProcessor.cs:line 155
   at Nethermind.Consensus.Processing.BlockCachePreWarmer.<>c__DisplayClass11_0.<WarmupTransactions>b__0(Int32 _) in /src/Nethermind/Nethermind.Consensus/Processing/BlockCachePreWarmer.cs:line 131

The syntonization has been broken and restarting of execution / consensus did not help.

Steps to Reproduce (no deterministic)

1. Start nethermind client and sync. I used following configuration:

Metrics.Enabled=true#log=DEBUG#Metrics.NodeName=dmytro-b-test-release-full-mainnet-teku#Seq.MinLevel=Info#JsonRpc.EnabledModules=[Eth,Subscribe,Trace,TxPool,Web3,Personal,Proof,Net,Parity,Health,Rpc,Debug,Admin]#Sync.SnapSync=false#Sync.FastSync=false#Pruning.Mode=Hybrid

2. Wait some time
3. Check error logs

Actual behavior
The syntonization stacks and the error appears a few times per minute

Expected behavior
No errors should appear during the sync

Desktop:
Operating System: Ubuntu 22.04.5
Version: 22.04.5
Installation Method: Docker
Consensus Client: prysm
Nethermind version:

Version: 1.29.0-rc+7320938e
Commit: 7320938e5fe74ad6ac1782ce723ddb7bb7d3d78a
Build Date: 2024-09-30 19:43:50Z
OS: Linux x64
Runtime: .NET 8.0.8

Additional context
The issue happened with prysm only. The rest consensuses work well

Logs
https://seq.nethermind.io/#/events?filter=NodeName%3D'dmytro-b-test-release-full-mainnet-prysm'&to=2024-10-04T02:22:49.100Z