fix: peer renewal connection drop & stream management

[weboko]

Problem

Way too many No stream available issues.

Solution

Seems there are multiple issues causing the problem:

• When doing LightPush sometimes we do a send to peer connection to which was dropped.
  Fix: Drop connection only if peer was renewed, if wasn't - do not change list of peers on a protocol.

• We create and keep way to many streams for protocols per peer.
  Fix: Improve stream management and add locks.

Currently typical connection has way to many redundant streams.

{
    "connectionId": "8qmraw1727395076154",
    "streams": [
      {
        "streamId": "i1",
        "streamProtocol": "/vac/waku/metadata/1.0.0"
      },
      {
        "streamId": "i2",
        "streamProtocol": "/vac/waku/metadata/1.0.0"
      },
      {
        "streamId": "i3",
        "streamProtocol": "/vac/waku/metadata/1.0.0"
      },
      {
        "streamId": "i4",
        "streamProtocol": "/vac/waku/store-query/3.0.0"
      },
      {
        "streamId": "i5",
        "streamProtocol": "/vac/waku/lightpush/2.0.0-beta1"
      },
      {
        "streamId": "i6",
        "streamProtocol": "/vac/waku/filter-subscribe/2.0.0-beta1"
      },
      {
        "streamId": "i7",
        "streamProtocol": "/vac/waku/metadata/1.0.0"
      },
      {
        "streamId": "i8",
        "streamProtocol": "/vac/waku/store-query/3.0.0"
      },
      {
        "streamId": "i9",
        "streamProtocol": "/vac/waku/lightpush/2.0.0-beta1"
      },
      {
        "streamId": "i10",
        "streamProtocol": "/vac/waku/filter-subscribe/2.0.0-beta1"
      },
      {
        "streamId": "i11",
        "streamProtocol": "/vac/waku/metadata/1.0.0"
      },
      {
        "streamId": "i12",
        "streamProtocol": "/vac/waku/store-query/3.0.0"
      },
      {
        "streamId": "i13",
        "streamProtocol": "/vac/waku/lightpush/2.0.0-beta1"
      },
      {
        "streamId": "i14",
        "streamProtocol": "/vac/waku/filter-subscribe/2.0.0-beta1"
      }
    ]
  }

After changes to `StreamManager` in this PR streams are better organized with only one redundant stream.

{
    "connectionId": "6jpx4f1727394454291",
    "streams": [
      {
        "streamId": "i1",
        "streamProtocol": "/vac/waku/store-query/3.0.0"
      },
      {
        "streamId": "i2",
        "streamProtocol": "/vac/waku/lightpush/2.0.0-beta1"
      },
      {
        "streamId": "i3",
        "streamProtocol": "/vac/waku/filter-subscribe/2.0.0-beta1"
      },
      {
        "streamId": "i4",
        "streamProtocol": "/vac/waku/metadata/1.0.0"
      }
    ]
  },

Notes

• Validation: dogfooding app seems to report less stream establishment errors.
• Addresses spike of errors we observe from dogfooding app.

[github-actions]

size-limit report 📦

Path	Size	Loading time (3g)	Running time (snapdragon)	Total time
Waku node	83.61 KB (+0.07% 🔺)	1.7 s (+0.07% 🔺)	9.7 s (-25.95% 🔽)	11.4 s
Waku Simple Light Node	135.26 KB (+0.07% 🔺)	2.8 s (+0.07% 🔺)	11.3 s (-26.02% 🔽)	14 s
ECIES encryption	22.94 KB (0%)	459 ms (0%)	5.5 s (+45.93% 🔺)	6 s
Symmetric encryption	22.39 KB (0%)	448 ms (0%)	2.9 s (-39.19% 🔽)	3.4 s
DNS discovery	72.28 KB (0%)	1.5 s (0%)	8.4 s (-21.78% 🔽)	9.8 s
Peer Exchange discovery	73.85 KB (+0.09% 🔺)	1.5 s (+0.09% 🔺)	8.7 s (-3.61% 🔽)	10.2 s
Local Peer Cache Discovery	67.63 KB (0%)	1.4 s (0%)	8.1 s (+1.19% 🔺)	9.5 s
Privacy preserving protocols	74.79 KB (0%)	1.5 s (0%)	7.6 s (-40.27% 🔽)	9.1 s
Waku Filter	78.68 KB (+0.22% 🔺)	1.6 s (+0.22% 🔺)	6.9 s (-24.7% 🔽)	8.5 s
Waku LightPush	76.89 KB (+0.13% 🔺)	1.6 s (+0.13% 🔺)	10 s (+24.21% 🔺)	11.5 s
History retrieval protocols	76.03 KB (+0.02% 🔺)	1.6 s (+0.02% 🔺)	13.9 s (+73.65% 🔺)	15.4 s
Deterministic Message Hashing	7.38 KB (0%)	148 ms (0%)	1.9 s (-24.55% 🔽)	2 s

[danisharora099]

I tried this with Dogfooding, NO_STREAM_AVAILABLE still creeps in after a few minutes

Let's supersede this with #2137, where I am experimenting with no disconnections, but simply removing (temporarily) the peer from the protocol's management

[weboko]

I tried this with Dogfooding, NO_STREAM_AVAILABLE still creeps in after a few minutes

that's true, dfdea84 addressed issue only partially

this 00c3261 is more comprehensive fix to another side of the problem

please, try and tell me if you experience any, so far for me after and hour of running it works fine

[weboko]

if stream = await connection.newStream(this.multicodec); in createStream succeeds - then it will be attached to connection and hence this.getStreamForCodec will return it.

This is a bit convoluted at first but:

• we need to keep our single source of truth - connection object;
• we need to have a lock so that multiple events schedule only one redundant stream creation;

[danisharora099]

Looks good!

Do you think we should add tests that cover the uncaught functionality solved with this PR?

[weboko]

@waku-org/js-waku I temporarily suppressed this check as it doesn't regress functional testability here
will re-enable it once I root cause it