Node crashed with error "invalid square root" and unable to boot up again

[atherkhalil]

Node crashed with error "invalid square root".

Description

A validator node from my network just went down and is unable to boot again. Its giving an "invalid square root" error after trying to sync up with the rest of the nodes and crashing again right away. This has caused the average block time to go up from 2s to 2.7s.

Your environment

• OS: Ubuntu server 22.04 LTS amd64
• Polygon-edge version: v0.8.1
• Branch: Production (dockerhub)
• Cloud hosted (AWS)
• The validators are running on Docker.

Steps to reproduce

• I don't know how to reproduce this error as it happened suddenly.
• Is the chain producing blocks and serving customers atm? Yes, its a production network and is adversely impacting the services due to increased block time and therefore increased tx wait time.

Logs

2023-09-30T08:55:46.541Z [INFO] polygon.server: Data dir: path=./bfic-chain
2023-09-30T08:55:46.783Z [INFO] polygon.blockchain: Current header: hash=0x4fd1ceb7741f4a15ae8398f878b071a186fa522ca165e02f29a7ede205bcbbbd number=4260442
2023-09-30T08:55:46.783Z [INFO] polygon.blockchain: genesis: hash=0x57de3531b20a7c3370910a5acf535fb62eb582e7b06cf77072ac3055d011de8f
2023-09-30T08:55:46.783Z [INFO] polygon.server.ibft.fork_manager.snapshot_validator_set: snapshot was not found, restore snapshot at beginning of current epoch: current epoch=42
2023-09-30T08:55:46.783Z [INFO] polygon.server.ibft.fork_manager.snapshot_validator_set: syncing past snapshots: from=4200000 to=4260442
invalid square root

Proposed solution

I know a way around, but its not actually solving the problem. I can delete the data directory, restart the node with new credentials and wait for it to fully sync with the network but this will take a lot of time as the network has 4 Million + blocks.

[goran-ethernal]

Which consensus are you using? IBFT or Polybft?

[atherkhalil]

Which consensus are you using? IBFT or Polybft?

I'm using IBFT consensus.

[ghost]

Delete the blockchain and trie folders from your data dir. Copy them from one of the other working nodes. Your node will restart then.