Quality Connection with Extremely Dynamic ISP Services (Starlink & 5G T-Mobile)

[Network-Traditions]

***UPDATE 13-DEC-2022 - To date we have been unsuccessful in resolving our issues with #2595, which ultimately requires a full reset of VPS and OMR daily. Subseqeuently, we have developed a work-around which allowed us to reconfigure V2Ray as the default proxy for OMR making the following information relevant again for our deployment. To address problems outlined in #2583, we defined individual RTP UDP port forwards for the needed range utilizing V2Ray forwarding. This resolved the one-way audio problem discussed while allowing the use of V2Ray, which has greater performance and reliability than shadowsocks with our configuration.

UPDATE 9-OCT-2022 - The original discussion below utilizes V2Ray as the default proxy. Due to #2583 we've been testing our configuration with ShadowSocks as the default proxy and are having a very different experience from the following presentation. We will update this discussion when more definitive information is developed as a result of changing the default proxy.

We have been conducting copious testing of various configurations of OMR to maximize the bandwidth or our ISP services, Starlink and 5G T-Mobile while controlling latency to facilitate reliable internet service for a small business server stack with sensitive services such as VOIP and video conferencing. With the configuration we've been using for about a week, it appears we've succeeded in achieving those goals for our production system. It should be stressed that our observations are preliminary and may be inappropriate in contexts not considered or tested to date, however, they should still provide valuable feedback to the development and discussions about OMR.

Internet Service Providers

T-Mobile 5G Business Internet Telit modem in USB 3.0 sled with static publicly reachable IP
StarLink version 2 configured as Ethernet bridge with a CGNAT non-publicly reachable IP

System Component Configuration

VPS - OMR - Hypervisor - pfSense - SMB Servers, Workstations & Devices

Specifications

OpenMPTCProuter: v0.59.1-5.4 r0+16594-ce92de8c8c
OpenMPTCProuter VPS version: Debian 11 5.4.207-mptcp VPS 0.1028
Architecture: x86-64
OpenMPTCProuter VPS provider: IONOS.com 2 vCore 2GB RAM "Type M VPS"
OpenMPTCProuter platform: x86_64 Intel® Celeron® Processor J4125 with 4 Intel i225 "B3" NICs

Geography Context

Starlink earth station origins: Southwest USA
T-Mobile Point of Presence: Typcially Southwest USA
VPS Data Center Origin: Kansas - Missouri Border
Typical OMR to VPS to Kansas City, Missouri Speedtest Latency: 70-120ms

Connection Quality Metrics

• OpenMPTCProuter
• • Network-MPTCP-Bandwidth graphs
• • OpenMPTCProuter Status
• • • localhost "Green" status
• • • Starlink & T-Mobile "Green" status and latency remaining below 200ms perdominantly not exceeding 300ms
• • • OpenMPTCProuter "Green" status
• pfSense Gateway ping monitoring
• • RTT, RTT(standard deviation) and packet loss from pfSense to VPS
• • RTT, RTT(standard deviation) and packet loss from pfSense OpenVPN client tunnels to Southwest USA pfSense OpenVPN servers
• • pfSense speed graphs (form & throughput) during synthetic testing
• VPS
• • Packet Capture Evaluation
• • • Wireshark RST and DupACK Analysis
• Synthetic Testing Tool (Unloaded and loaded Download/Upload latency and jitter)
• • Speedtest.net
• • Fast.com
• • Waveform Bufferbloat
• • CloudFlare Speedtest
• Production System Usage
• • VOIP Communication Quality
• • Video Conferencing Quality
• • Video Streaming up to 4K 60fps ~+30Mbps Stream Quality
• • SMB Service Quality
• • • Web Services
• • • • Nextcloud
• • • • Back Office Applications
• • • • Remote Support Application
• • • • Website Performance
• • • Groupware Communications

Not Firm, But Reasonably Confident Findings to Date

1. Network-MPTCP-Settings-Multipath TCP Checksum = enable
2. Network-Interfaces-Global network options-Packet Sterring = checked
3. Network-Devices = wwan0 & eth 2 no IPV6 and reduce MTU of ISP interfaces to 1496 per ping google.com -f -l 1468 test.

• 1, 2 and to a lesser degree 3 contributed to the following improvements:
• • VPS packet capture RST reduction of up to 10 times.
• • Significant reduction of dropped packet duplicate ACK.
• • Significant reduction of "retransmissions".

4. Network-MPTCP-Settings-Multipath TCP SYN retries = 2
5. System-OpenMPTCProuter-Advanced Settings-IPv4 TCP SYN retries = 2
6. System-OpenMPTCProuter-Advanced Settings-IPv4 TCP SYN retries1 = 1
7. System-OpenMPTCProuter-Advanced Settings-IPv4 TCP SYN retries2 = 4

• 4-7 contributed to rapid TCP mitigation of the wildly fluctuating conditions of packet quality and throughput of ISPs.
• 1-7 contributed to the following:
• • Significant latency reduction (pfSense RTT, RTT(standard deviation), Synthetic Testing Download, Upload and Idle).
• • Virtually continous 0% packet loss on pfSense monitors.
• • No apparent impact on download or upload performance.

8. Network-MPTCP-Settings-Multipath TCP path-manager = fullmesh
9. Network-MPTCP-Settings-Multipath TCP scheduler = ECF
10. Network-MPTCP-Settings-Congestion Control = bbr2
11. Network-MPTCP-Settings-WAN2 (Starlink) = master
12. Network-MPTCP-Settings-WAN1 (T-Mobile) = enabled
13. System-OpenMPTCProuter-Advanced Settings-Enable TCP Low Latency = checked
14. System-OpenMPTCProuter-Advanced Settings-Master interface selection = Balancing
15. Network-SQM Qos-Basic Settings-Enable this SQM instance = checked
16. Network-SQM Qos-Basic Settings-Enable SQM autorate = checked
17. Network-SQM Qos-Basic Settings-Base download speed (kbit/s) (ingress) = 0
18. Network-SQM Qos-Basic Settings-Base upload speed (kbit/s) (ingress) = 0

• 15-18 applied to ISP interfaces eth2 (Starlink) and wwan0 (T-Mobile).
• 15-18 verdict is still out on minimum and maximum download and upload entries of this section.
• 8-18 contribute to maximizing bandwidth download and upload throughput.

Typical Performance

• Speedtest.net

• • Download
• • • Low 60 Mpbs
• • • Typcial 100 Mbps
• • • Peak 150 Mbps
• • Upload
• • • Low 15 Mpbs
• • • Typcial 20 Mbps
• • • Peak 35 Mbps

• Waveform Bufferbloat

• • Download (peak grade "A"; typical grade "B"; low grade "C")
• • • Low 20 Mpbs 15 ms jitter with 0 to 15 ms increased latency
• • • Typcial 50 Mbps 30 ms jitter with up to 40 ms increased latency
• • • Peak 80 Mbps 60 ms jitter with up to 60 ms increased latency
• • Upload (peak grade "A"; typical grade "B"; low grade "C")
• • • Low 10 Mpbs 15 ms jitter with 0 to 15 ms increased latency
• • • Typcial 20 Mbps 30 ms jitter with up to 25 ms increased latency
• • • Peak 35 Mbps 50 ms jitter with up to 50 ms increased latency

• pfSense ping and packet loss

• • Traffic graph per Speedtest.net download and upload testing = square wave in form, not wild saw tooth
• • RTT to VPS (follows traffic load as defined here)
• • • Low 70 ms
• • • Typcial 120 ms
• • • Peak 250 ms
• • RTT (standard deviation) to VPS (follows traffic load as defined here)
• • • Low 20 ms
• • • Typcial 80 ms
• • • Peak 250 ms
• • Packet Loss to VPS (follows traffic load as defined here)
• • • Low 0%
• • • Typcial 0%
• • • Peak 0%

• System-OpenMPTCProuter Status

• • ISP latency (follows traffic load as defined here and may be due to momentary service outage)
• • • Low (Starlink 40 ms; T-Mobile 70 ms)
• • • Typcial (Starlink 100-200 ms; T-Mobile 100-200 ms)
• • • Peak (+300 ms occassional, but recovers quickly: generally within next screen update)
• • ISP Service Lost Server Ping (follows traffic load as defined here and may be due to momentary service outage)
• • • Low (never)
• • • Typcial (rare)
• • • Peak (occassional, but recovers quickly: generally within next screen update)
• • localhost (script or ping connection deficiency) to VPS (follows traffic load as defined here)
• • • Low (never)
• • • Typcial (rare)
• • • Peak (occassional, but recovers quickly: generally within next screen update)
• • OpenMPTCProuter lost VPN connection (follows traffic load as defined here)
• • • Low (never)
• • • Typcial (never)
• • • Peak (rare, but recovers quickly: generally within next screen update)

• SMB Services

• • VOIP Calls (typical delay and clear without jitter)
• • Video Conference (uninterrupted, no buffering or jitter even while conducting Speedtest.net benchmarks)
• • Remote Desktop and ScreenConnect Sessions (no notable delays, jitter or disconnects)
• • 4K 60fps Streaming (reasonably reliable unbuffered playback with steady 30Mbps throughput)
• • OpenVPN TCP client performance (best connection and throughput of configurations tested)
• • 4K Firestick Streaming Performance (a slightly noticable click and screen update delay compared to direct cable service experience)
• • Web Browsing (occasional slightly noticable update display compared to direct cable service experience)

Other Points of Interest

• Asterisk based UDP audio does not work properly with the V2Ray proxy
• • Asterisk behind pfSense/OMR Outbound VOIP One Way Audio #2583 System-OpenMPTCProuter-Settings Wizard-Show advanced settings-Default Proxy = Shadowsocks
• Some historical performance experience is documented in 1WAN is faster than 2WANS aggregation? Why?  #2365