feat: drop msgs to be relayed waiting for too long in the queue

[diegomrsantos]

This PR adds the possibility of configuring the maximum time a message can stay in the non-priority queue and still be considered relevant to be sent to the respective remote peer. If the message has been waiting for this duration or longer at the time is dequeued it will be discarded and not sent.

[codecov]

Codecov Report

Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (b16ec00) 83.27% compared to head (6484d3f) 83.24%.
Report is 3 commits behind head on forward-msgs-non-priority-v2.

Additional details and impacted files

@@                       Coverage Diff                        @@
##           forward-msgs-non-priority-v2    #1017      +/-   ##
================================================================
- Coverage                         83.27%   83.24%   -0.03%     
================================================================
  Files                                91       91              
  Lines                             15428    15438      +10     
================================================================
+ Hits                              12847    12852       +5     
- Misses                             2581     2586       +5     

Files	Coverage Δ
libp2p/protocols/pubsub/gossipsub.nim	87.87% <100.00%> (-0.35%)	⬇️
libp2p/protocols/pubsub/pubsubpeer.nim	94.11% <90.00%> (+1.01%)	⬆️

... and 3 files with indirect coverage changes

[AlejandroCabeza]

At first glance it LGTM.
Maybe it'd be good to take into account Ludovic's opinion, given he's more familiar with the repo?

[kaiserd]

Thanks! Looks good.

Is there a reference as to why 1s was chosen as the default.
How do other implementations / the spec handle this?

If you did not do this yet, I suggest first testing the base PR in a testnet, followed by another run with this PR merged into the base PR forward-msgs-non-priority-v2

[diegomrsantos]

@kaiserd I've been testing all PRs running a local Nimbus client on Holesky. Everything seems fine. The only implementation with this feature I'm aware of is rust-libp2p and last time I checked they were testing it on a branch sigp/rust-libp2p#555.

They use 500ms as the default value. I tried it first but I saw a large number of messages being dropped. I believe this value is network/application-specific and could be hard to guess a default value. 1s was just an initial attempt, I welcome feedback here.

[diegomrsantos]

Those are the other features they implemented sigp/lighthouse#4918 (comment)

[kaiserd]

We need further investigation here.
Let's move this back to draft state for now.