[litep2p-milestone]: Improve litep2p robustness

[lexnv]

The Vision

Improve the robustness and fault-tolerance of the litep2p crate and bring the crate to a higher level of maturity.

The goal is to stabilize the litep2p and use it as the default backend of Substrate-based chains.

The Plan

To achieve this, we'll focus our efforts on:

• Feature Completeness
  • Implement missing features (validate ping payloads, generate unique random ping payloads, ensure kad records TTL)
• Robustness
  • Ensure core code logic does not panic upon unexpected/malicious user behaviors, panics, unwraps, undocumented expects, array indexing without length validation etc
  • Fix incorrect functionality and divergences from the libp2p specification
  • Repair bugs and limit the surface of race conditions
• Maintenance: keep the code simple
  • Avoid code duplicates (transport layer)
  • Coherent APIs and errors
  • Adopt rust clippy and best practices
  • Improve and add documentation
• Performance Improvements
  • Avoid unnecessary clones
  • Improve the query engine
  • Analyze memory and CPU consumption
• Testing
  • increase code coverage by unit-testing sensitive components (multiselect / noise, kad etc)
  • improve our integration testing scenarios
  • add fuzzing
  • create a plan for continuous testing and triaging in our test stack and keep track of triage reports

Feature Completeness

• Implement connection limits #17
• kad: Revisit FIND_NODE and increase test coverage #100
• kad: Implement ADD_PROVIDER command  #166
• kad: Implement GET_PROVIDERS command #167
• transport: Change AddressStore to keep track of more dial addresses #180
• transport/manager: Add ability to accept/reject incoming connections before negotiation  #186
• request-response: Provide more details into rejected requests for a high request failure #188
• identify: Update local node listen addresses #191
• errors: Add DialError error and ListDialFailures event for better error reporting #206
• transport: Implement TransportService::local_peer_id() #224
• kad: Expose all peer records of GET_VALUE query #96

Robustness

• crypto/noise: Make code fault tolerant #124
• ping: Make ping fault tolerant wrt outbound substreams races #133
• kad: Ensure Kademlia Record has TTL #129
• protocol/notif: Fix panic on missing peer state  #143
• transport-manager: Unknown connection opened #172
• kad: Fix substream opening and dialing race #222
• query-executor: Save the task waker on empty futures #219
• crypto/noise: Generate keypair only for Curve25519 #214
• transport: Fix double lock and state overwrite on disconnected peers #179
• kad: Do not update memory store on incoming GetRecordSuccess #190
• transport: Reject secondary connections with different connection IDs #176
• Fix TCP, WebSocket, QUIC leaking connection IDs in reject() #198
• kad: Update connection status of an existing bucket entry #181
• manager: Fix connection limits tracking of rejected connections  #286
• transport: Fix waking up on filtered events from poll_next  #287
• transports: Fix missing Poll::Ready(None) event from listenener #285
• manager: Avoid overflow on stream implementation for TransportContext #283
• manager: Log when polling returns Ready(None) #284
• transport_service: Improve connection stability by downgrading connections on substream inactivity #260
• tcp/websocket/quic: Fix cancel memory leak #272
• kad: Fix not retrieving local records #221
• transport: Abort canceled dial attempts for TCP, WebSocket and Quic #255
• kad/executor: Add timeout for writting frames #277
• kad: Avoid cloning the KademliaMessage and use reference for RoutingTable::closest  #233
• crypto/noise: Verify crypto/noise signature payload #278
• req-resp: Fix panic on connection closed for substream open failure #291
• transport_service/logs: Provide less details for trace logs #292
• req-resp: Fix memory leak of pending substreams #297
• notification: Fix memory leak of pending substreams #296
• mdns/fix: Failed to register opened substream #301
• identify: Replace FuturesUnordered with FuturesStream #302

Maintenance

• transport: Extract common transport code between TCP and WebSocket #70
• Possible use of libp2p_identity::PeerId as PeerId type #73
• kad: Remove unused serde cfg #262
• transport_service: Improve logs and move code from tokio::select macro #254

Testing

• mutlistream_select/tests: Fuzz Message encode / decode #127
• mutlistream_select/tests: Add tests for LengthDelimited #126
• crypto/noise/tests: Increase test coverage #125
• minor: fix tests without websocket feature #215
• kad: Refactor GetRecord query and add tests #97
• ci: Run CI on master and PRs #87
• ci: Enable dependabot #89
• ci: Ensure cargo-machete is working with rust version from CI #303
• chore: Update hickory-resolver to version 0.24.2 #304

Performance

• crypto/noise: Investigate the performance impact of creating a new RNG and DH in snow #123
• substream: Use write_all instead of manually writing bytes  #217
• find_node: Optimize parallelism factor for slow to respond peers #220

Outstanding

• Any plan to support WASM target? #99
• ping: Report back ping failures  #132
• ping: Verify received "pong" payload #134
• identify: Open inbound substream for unknown peer #103
• multistream: Lazy negotiated protocol cannot be closed before receiving messages #60
• Research the possibility of embedding Peerset into NotificationProtocol #34
• Start using V1Lazy when applicable #1
• notification: Invalid handshake state #299
• error: Merge NegociatedError with SubstreamError #128
• clippy: Revisit allowed clippy rules #85

[lexnv]

Thanks everyone, Litep2p is now the default network backend on polkadot https://forum.polkadot.network/t/announcing-the-stabilization-of-the-litep2p-network-backend/13712!