ERROR p2p: Failed to reconnect to peer addrs=[/ip4/x.x.x.x/tcp/4201] error=failed to dial

[LovelyGuy18]

I am trying to make a private network via Internet with geth and prysm.
Now the beacon client is not connected to other peers.

Platform: Ubuntu 22.04
Geth: latest
Prysm: latest

[LovelyGuy18]

time="2024-09-06 10:59:43" level=info msg="Node started p2p server" multiAddr="/ip4/92.113.29.189/tcp/4200/p2p/16Uiu2HAm8uDUnYs2y1d2Ju9WiGhQFtTmo2p7PfztwLotdUaduj17" prefix=p2p
time="2024-09-06 10:59:43" level=info msg="Node started external p2p server" multiAddr="/ip4/92.113.29.189/tcp/4200/p2p/16Uiu2HAm8uDUnYs2y1d2Ju9WiGhQFtTmo2p7PfztwLotdUaduj17" prefix=p2p
2024-09-06T10:59:43.695Z	DEBUG	basichost	basic/basic_host.go:753	host 16Uiu2HAm8uDUnYs2y1d2Ju9WiGhQFtTmo2p7PfztwLotdUaduj17 dialing 16Uiu2HAmTKJKbbnXfMCdysMabDQkCpndRSDWeKJRj3RJfF4fnfKz
2024-09-06T10:59:43.695Z	DEBUG	swarm2	swarm/swarm_dial.go:240	dialing peer	{"from": "16Uiu2HAm8uDUnYs2y1d2Ju9WiGhQFtTmo2p7PfztwLotdUaduj17", "to": "16Uiu2HAmTKJKbbnXfMCdysMabDQkCpndRSDWeKJRj3RJfF4fnfKz"}
2024-09-06T10:59:43.695Z	DEBUG	upgrader	upgrader/listener.go:110	listener <stream.Listener[TCP] /ip4/92.113.29.189/tcp/4200> got connection: /ip4/92.113.29.189/tcp/4200 <---> /ip4/92.113.29.189/tcp/58398
2024-09-06T10:59:43.695Z	DEBUG	swarm2	swarm/limiter.go:187	[limiter] adding a dial job through limiter: /ip4/167.88.165.205/tcp/4201
2024-09-06T10:59:43.695Z	DEBUG	swarm2	swarm/limiter.go:155	[limiter] taking FD token: peer: 16Uiu2HAmTKJKbbnXfMCdysMabDQkCpndRSDWeKJRj3RJfF4fnfKz; addr: /ip4/167.88.165.205/tcp/4201; prev consuming: 0
2024-09-06T10:59:43.695Z	DEBUG	swarm2	swarm/limiter.go:161	[limiter] executing dial; peer: 16Uiu2HAmTKJKbbnXfMCdysMabDQkCpndRSDWeKJRj3RJfF4fnfKz; addr: /ip4/167.88.165.205/tcp/4201; FD consuming: 1; waiting: 0
2024-09-06T10:59:43.695Z	DEBUG	swarm2	swarm/swarm_dial.go:532	16Uiu2HAm8uDUnYs2y1d2Ju9WiGhQFtTmo2p7PfztwLotdUaduj17 swarm dialing 16Uiu2HAmTKJKbbnXfMCdysMabDQkCpndRSDWeKJRj3RJfF4fnfKz /ip4/167.88.165.205/tcp/4201
2024-09-06T10:59:43.695Z	DEBUG	upgrader	upgrader/listener.go:126	accept upgrade error: failed to negotiate security protocol: EOF (/ip4/92.113.29.189/tcp/4200 <--> /ip4/92.113.29.189/tcp/58398)
�[32mINFO �[0m[09-06|10:59:43.696] New local node record                    �[32mseq�[0m=1,725,620,383,693 �[32mid�[0m=f68a922b0b08002f �[32mip�[0m=92.113.29.189 �[32mudp�[0m=4300 �[32mtcp�[0m=4200
time="2024-09-06 10:59:43" level=trace msg="Fork ENR mismatches between peer and local node" error="fork digest of peer with ENR -MK4QPF17f0fYjo7pK9V9OFO7eVgCKoQbDmi7SuvWm2CcKy7aW8iIWnqrN61F3c9Mmq20NvDda0hO1jW0mVVQeRuSTSGAZHGhXJmh2F0dG5ldHOIAAAGAAAAAACEZXRoMpCdcTZPIAAAk___________gmlkgnY0gmlwhKdYpc2Jc2VjcDI1NmsxoQPZ1AHj36_PyTANQAG2sZNX_YwfQYZ4x7QabO2CeOHBbYhzeW5jbmV0cw-DdGNwghBpg3VkcIIQzQ: [157 113 54 79], does not match local value: [95 173 90 152]" prefix=p2p

time="2024-09-06 11:00:03" level=trace msg="Fork ENR mismatches between peer and local node" error="fork digest of peer with ENR -MK4QNOTnGb2PCcA0Yt9d_oBubnel0uPULohV1cnPDT_tDx5EP4ALpLlqvSA2dpTML3HtDEQmESSTTgHMRHaWHqs17aGAZHGhT_ph2F0dG5ldHOIAAAAAAwAAACEZXRoMpCdcTZPIAAAk___________gmlkgnY0gmlwhKdYpc2Jc2VjcDI1NmsxoQKAopwQZ4KL2fQbNZ1kt6RyGtZCjySraqM8ulRvEoQiMIhzeW5jbmV0cw-DdGNwghBog3VkcIIQzA: [157 113 54 79], does not match local value: [95 173 90 152]" prefix=p2p
�[34mTRACE�[0m[09-06|11:00:03.462] >> FINDNODE/v5                           �[34mid�[0m=fbae0cc0a366fdd8 �[34maddr�[0m=167.88.165.205:4300   �[34mreq�[0m=0x62d288dc46664fe6
�[34mTRACE�[0m[09-06|11:00:03.462] >> FINDNODE/v5                           �[34mid�[0m=7d3672ca835c1430 �[34maddr�[0m=167.88.165.205:4301   �[34mreq�[0m=0xff9dfb0d57ae4798
2024-09-06T11:00:03.478Z	DEBUG	upgrader	upgrader/listener.go:126	accept upgrade error: failed to negotiate security protocol: error reading handshake message: read tcp4 92.113.29.189:4200->85.2.5.87:34526: read: connection reset by peer (/ip4/92.113.29.189/tcp/4200 <--> /ip4/85.2.5.87/tcp/34526)

[artjoma]

Prysm 5.1.0, Geth 1.14.8, 6.8.0-36-generic #36-Ubuntu SMP PREEMPT_DYNAMIC Mon Jun 10 10:49:14 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

time="2024-09-19 13:32:04" level=error msg="failed to reconnect to peer" addrs=[/ip4/*.*.*.13/tcp/13000] error="failed to dial: failed to dial 16Uiu2HAmE7FuqkJ3XiaQH2yXqoBmpwkTsUF7qsVE7DUSzDhTJHjT: all dials failed

or

time="2024-09-19 21:10:50" level=error msg="failed to reconnect to peer" addrs=[/ip4/*.*.*.*/tcp/13000] error="failed to dial: failed to dial 16Uiu2HAmJhRa8SHb95L3uS4mPhdQ8ttS23dmHohYnuJnJRZGayV6: all dials failed
  * [/ip4/*.*.*.*/tcp/13000] failed to negotiate security protocol: EOF
  * [/ip4/*.*.*.*/tcp/13000] failed to negotiate security protocol: EOF" peer=16Uiu2HAmJhRa8SHb95L3uS4mPhdQ8ttS23dmHohYnuJnJRZGayV6 prefix=p2p

[garyng2000]

I can attest that this is an on going issue for private network. It seems that the peer selection logic is too conservative or something. In fact, I have setup a 5 node private network to test and eventually, they all got disconnect render the chain useless as the new block can no longer propagate resulting in the whole chain stalled. I believe even production one behave like that but given that there are lots of nodes out there, it is not a problem.