[HOLESKY PECTRA] Our head slot is not maintained upon restart

[tbenr]

When a node is restarted (during non finality periods?) while following the chain correctly, it happens that our head can go back to a very old past slot (like 4 days prior).
We should doublecheck how we resume our forkcoice and head.

for instance a restarted holesky node prints:

Syncing     *** Target slot: 3759621, Head slot: 3727463, Remaining slots: 32158, Connected peers: 45

but the dumped fork choice clearly shows that most recent nodes:

[...
    {
      "slot": "3759439",
      "block_root": "0xbde7e9bf5675a61aa857dab6a7cdc42308aa7a11e3e9b41004de4802c4986d8b",
      "parent_root": "0xfa3506110dbf8b04746be6cc9dd981109556f01d987820756e9dd532d671e665",
      "justified_epoch": "115968",
      "finalized_epoch": "115967",
      "weight": "0",
      "validity": "optimistic",
      "execution_block_hash": "0x99e7acfb7bae42ed7f13fde6ba752fa2782655f2a8f3904f7d62cc22490b613c",
      "extra_data": {
        "state_root": "0x0327c0cd3aa87b523e5b703f59f1751a34ca1a928b7e24732f8cf8c580618796",
        "justified_root": "0x1a97f8c5ae8e48ba53af214e621998bb7fc46c091c820666a0d31902b67ad3ca",
        "unrealised_justified_epoch": "115968",
        "unrealized_justified_root": "0x1a97f8c5ae8e48ba53af214e621998bb7fc46c091c820666a0d31902b67ad3ca",
        "unrealised_finalized_epoch": "115967",
        "unrealized_finalized_root": "0xd0285a8ec914f53cf15e9cce336a07c4f11c666e2addec620592d5ff3640ed34"
      }
    },
    {
      "slot": "3759442",
      "block_root": "0xe95256d3a24891e6b3646cf79afab8b67831e3f25149a30c51cf9ef424eb4cb9",
      "parent_root": "0xfa3506110dbf8b04746be6cc9dd981109556f01d987820756e9dd532d671e665",
      "justified_epoch": "115968",
      "finalized_epoch": "115967",
      "weight": "0",
      "validity": "optimistic",
      "execution_block_hash": "0x847080bc89dd3ee47ed20e837c23030af00b87a52a027a4f99e9033d6e25136b",
      "extra_data": {
        "state_root": "0xa37e11529791de296d34da1fe2fe4ae27a5b30b6c4f5debeb5766475eb2e9160",
        "justified_root": "0x1a97f8c5ae8e48ba53af214e621998bb7fc46c091c820666a0d31902b67ad3ca",
        "unrealised_justified_epoch": "115968",
        "unrealized_justified_root": "0x1a97f8c5ae8e48ba53af214e621998bb7fc46c091c820666a0d31902b67ad3ca",
        "unrealised_finalized_epoch": "115967",
        "unrealized_finalized_root": "0xd0285a8ec914f53cf15e9cce336a07c4f11c666e2addec620592d5ff3640ed34"
      }
    }
  ]
  "extra_data": {}
}

[tbenr]

note that all node's weight are back to 0 upon restart

[tbenr]

This could be that we load back all the nodes in proto array, than when we call findHead for the first time, all the heads are in tie so the tie breaker kicks in and give us essentially a random head which then becomes our canonical. We should check if we always get an actual chain head or we get there even with a random block which isn't a head.
Doing the check.

[tbenr]

I remote-debug on a node with the problem and at startup it actually has 815 heads

the choosen head as a canonical it resolve the canonical chain by tie-breaking them because all nodes are initialized with 0 weight.

An option would save in a DB variable our canonical head root once in a while (like on epoch transition) and upon startup add just one vote to that node in the proto array just to make it the head again.