norm/norp over multipath for low latency bursty traffic

[iofo]

Hi Everyone,
Firstly I would like to thank the authors for creating NORM and NORP.
My use case is for ordered reliable low latency data transmission. We have a low latency line with limited bandwidth. For most of the time we are under the bandwidth limits but sometimes we have microbursts so I would like to spill over to a higher latency but higher bandwidth line. I have setup a multipath tunnel which mostly routes down the low latency line but if we reach the bandwidth limit it will route down the next line and then I run norp socks proxy at either end. Will the NORM protocol handle potential reordering ok? I see very high latencies many many seconds sometimes and I am wondering what config settings I need to be tweaking. I have played with backoff timer to zero as suggested in the documentation because we are not multi casting to a large group. I would also like to send FEC packets in case of drop to avoid a full RTT rerequest so I modified the NORP code to allow setting the FEC parameters. I could potentially mirror all data over the two lines but just wondering if the NORM will sort it all out at the other end or could I be causing NACK storms due to reordered data.
Am I barking up the wrong tree here trying to use NORM and NORP for something they are designed not to do?
Thanks for any suggestions!

[bebopagogo]

If you are using NORP, it uses NORM with its NORM_OBJECT_STREAM mode and provides reliable, ordered byte-stream delivery to proxy TCP connections. NORM's reliability protocol handles out of order delivery well with respect to its ARQ operation. However, NORP uses the "NORM-CC" TCP-friendly congestion control option by default. Similar to TCP, it responds to missing packets (assuming congestion has occurred) by backing off its transmit rate. So, with your hybrid low-latency/high-throughput path adaptation, the NORM receiver may receive out-of-order packets over the low-latency path when some group of packets is shifted over to your higher-latency, higher-throughput path and routing shifts back to the low-latency path. So, in addition to the in-order delivery that NORM uses (which means the stream delivery to the receiver applications has to wait for any high-latency path packets to be delivered), the congestion control may also back-off in transmit rate during these routing events. This can potentially be mitigated in a couple of ways: 1) Configure NORM for fixed-rate operation if you are not worried about the "fairness"
that congestion control provides ... You can configure the rate high, but NORM will transmit at lower rate if the application is only providing data to it at a lower rate, or 2) Use NORM's optional ECN-based congestion control mechanism (NORM-CCE) that depends upon routers in the path doing ECN packet marking to detect congestion. With NORM-CCE, packet loss (or out of order delivery) is not treated as a prime indicator of congestion and thus the rate back-off issue might be overcome.

NORP just does unicast so I assume your application is unicast. In that case, NORM automatically disables the various back-off timeouts used for multicast feedback control and works as unicast ARQ protocol. You can add "proactive FEC" packets (i.e., auto-parity) to the transmission that can also reduce latency of reliable delivery for long-latency links since NACK/retransmission delays can be avoided for packet loss within the bounds of the proactive FEC configuration ... of course, adding "auto parity" to the stream adds overhead, but it low latency is priority over bandwidth efficiency, then it is helpful.

[iofo]

Thank you for this information. I will experiment more with fixed rate and ECN as we do not care about other traffic because we are the only traffic on this line.