- Sponsor
-
Notifications
You must be signed in to change notification settings - Fork 43
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Question: MPTCP subflows are reconnected every now and then #526
Comments
Hello, Which kernel version are you using on both sides? Can you reproduce the issue with the latest stable release? (or the last LTS one?). (I guess you are using the in-kernel path-manager, right? → Did you change How are your MPTCP endpoints being managed? Note that subflows can also be removed during the connection due to some network issues, but in this case, and for the moment (#440), the in-kernel will not try to re-establish these subflows. |
Hi, Matthieu:
Thanks for your reply.
My test setup is as follows:
The client is running openwrt kernel 6.1.71, and the server is running ubuntu kernel 6.8.0.
I did not change the pm_type.// in ubuntu ***@***.***:~$ sysctl net.mptcp.pm_type
net.mptcp.pm_type = 0
// on the openwrtcat /proc/sys/net/mptcp/pm_type0
You said "The in-kernel PM will close (or re-open) subflows during a connection only when the corresponding endpoint is removed (or re-added)." That is a very useful information to me. I will pursuit my debugging along this line. I am just wondering if there is an easy way to have the kernel explicitly or implicitly show some debugging message about what triggered the subflow recycling?
It seems to me that you are way more familiar with the mptcp detail than me. I will do more study. If I found more clue about my problem, I will share with you.
Thanks.
Kang-sen
|
In your case, this behaviour is likely on the client side, because it is usually the client that initiates the creation of the new subflows. This kernel is quite old (from January 2024), can you eventually upgrade to a more recent version from the v6.1 series, or 6.6.y or newer? Quite a lot of fixes have been addressed between the v6.1.71 and the v6.1.113: 79 commits and 11 files changed, 842 insertions(+), 390 deletions(-)
The trigger should come from the userspace (except a bug), via Netlink. You should be able to add an
|
Upgrade kernel version for openwrt is not a simple thing. We do not do that very frequently.
I ran tcpdump on the server and it seems the subflow establishment is initiated by the client, but subflow release is initiated by the server.
The creation part is easy to explain, because I put "subflow" option in the "ip mptcp endpoint add" command in the client. In the server, I did not run "ip mptcp add" command.
The release part is the mystery to me. I think I will use the trick you suggested to turn on debugging of pm_netlink.c on the server side.
Thanks.
Kang-sen
Here is the tcpdump capture: (ip-10-0-0-60 is the server side, and 198.18.0.6 is the client side)
14:02:13.607142 IP ip-10-0-0-60.9000 > 198.18.0.6.53665: Flags [F.], seq 1231, ack 3949, win 510, options [nop,nop,TS val 1643628437 ecr 1165550665,mptcp 12 dss ack 8800173739716601327], length 0
14:02:13.611161 IP 198.18.0.6.24830 > ip-10-0-0-60.9000: Flags [.], ack 3305, win 21, options [nop,nop,TS val 1165542950 ecr 1643620175,mptcp 12 dss ack 2963542118052365438], length 0
14:02:13.611180 IP 198.18.0.6.53665 > ip-10-0-0-60.9000: Flags [.], ack 1231, win 21, options [nop,nop,TS val 1165550685 ecr 1643627909,mptcp 12 dss ack 2963542118052365438], length 0
14:02:13.611195 IP 198.18.0.6.53665 > ip-10-0-0-60.9000: Flags [F.], seq 3949, ack 1231, win 21, options [nop,nop,TS val 1165551192 ecr 1643627909,mptcp 12 dss ack 2963542118052365438], length 0
14:02:13.611204 IP ip-10-0-0-60.9000 > 198.18.0.6.53665: Flags [.], ack 3950, win 510, options [nop,nop,TS val 1643628441 ecr 1165551192,mptcp 12 dss ack 8800173739716601327], length 0
14:02:13.626537 IP 198.18.0.6.53665 > ip-10-0-0-60.9000: Flags [.], ack 1232, win 21, options [nop,nop,TS val 1165551213 ecr 1643628437,mptcp 12 dss ack 2963542118052365438], length 0
14:02:13.650684 IP 198.18.0.6.58131 > ip-10-0-0-60.9000: Flags [S], seq 2266626873, win 43008, options [mss 1366,sackOK,TS val 1165551237 ecr 0,nop,wscale 11,mptcp 12 join id 1 token 0xa19f688d nonce 0x25db2c24], length 0
14:02:13.650717 IP ip-10-0-0-60.9000 > 198.18.0.6.58131: Flags [S.], seq 1198435055, ack 2266626874, win 65160, options [mss 1460,sackOK,TS val 1643628481 ecr 1165551237,nop,wscale 7,mptcp 16 join id 0 hmac 0xeac45c51436d2143 nonce 0x7335cbfc], length 0
|
I turned on the pm_netlink.c debugging. Here is my dmesg output. I can't spot and specific information about why the subflows are being released.
Thanks.
Kang-sen
---------------------dmesg output:
[ 2736.454244] mptcp:mptcp_pm_create_subflow_or_signal_addr: MPTCP: local 0:0 signal 0:0 subflows 1:8
[ 2747.712436] mptcp:mptcp_pm_nl_work: MPTCP: msk=00000000b1bc91a0 status=64
[ 2747.712444] mptcp:mptcp_pm_nl_rm_addr_or_subflow: MPTCP: address rm_list_nr 1
[ 2747.712447] mptcp:mptcp_pm_nl_rm_addr_or_subflow: MPTCP: -> address rm_list_ids[0]=1 local_id=0 remote_id=1 mpc_id=0
[ 2747.712488] mptcp:mptcp_pm_create_subflow_or_signal_addr: MPTCP: local 0:0 signal 0:0 subflows 1:8
[ 2758.981998] mptcp:mptcp_pm_nl_work: MPTCP: msk=00000000b1bc91a0 status=64
[ 2758.982005] mptcp:mptcp_pm_nl_rm_addr_or_subflow: MPTCP: address rm_list_nr 1
[ 2758.982009] mptcp:mptcp_pm_nl_rm_addr_or_subflow: MPTCP: -> address rm_list_ids[0]=2 local_id=0 remote_id=2 mpc_id=0
[ 2758.982138] mptcp:mptcp_pm_create_subflow_or_signal_addr: MPTCP: local 0:0 signal 0:0 subflows 1:8
[ 2770.162345] mptcp:mptcp_pm_nl_work: MPTCP: msk=00000000b1bc91a0 status=64
[ 2770.162353] mptcp:mptcp_pm_nl_rm_addr_or_subflow: MPTCP: address rm_list_nr 1
[ 2770.162356] mptcp:mptcp_pm_nl_rm_addr_or_subflow: MPTCP: -> address rm_list_ids[0]=1 local_id=0 remote_id=1 mpc_id=0
[ 2770.162393] mptcp:mptcp_pm_create_subflow_or_signal_addr: MPTCP: local 0:0 signal 0:0 subflows 1:8
[ 2781.391267] mptcp:mptcp_pm_nl_work: MPTCP: msk=00000000b1bc91a0 status=64
[ 2781.391275] mptcp:mptcp_pm_nl_rm_addr_or_subflow: MPTCP: address rm_list_nr 1
[ 2781.391279] mptcp:mptcp_pm_nl_rm_addr_or_subflow: MPTCP: -> address rm_list_ids[0]=2 local_id=0 remote_id=2 mpc_id=0
[ 2781.391332] mptcp:mptcp_pm_create_subflow_or_signal_addr: MPTCP: local 0:0 signal 0:0 subflows 1:8
[ 2792.556438] mptcp:mptcp_pm_nl_work: MPTCP: msk=00000000b1bc91a0 status=64
|
Are you saying that because the 'FIN' is initiated by the server? That's not necessarily true: when an MPTCP endpoint is removed on the client, the client will first notify the other peer with a
Do you do that manually?
On the server side I guess, right? Here, we can see that a |
// From tcpdump.log, I saw "rem-addr" mptcp option sent from the client(airbox) to the server(airbase)
14:02:37.182755 IP 198.18.0.6.24830 > ip-10-0-0-60.9000: Flags [.], ack 4427, win 21, options [nop,nop,TS val 1165574493 ecr 1643651717,mptcp 12 dss ack 2963542118052368068,mptcp 4 rem-addr id 2], length 0
14:02:48.355006 IP 198.18.0.6.24830 > ip-10-0-0-60.9000: Flags [.], ack 4469, win 21, options [nop,nop,TS val 1165583882 ecr 1643661105,mptcp 12 dss ack 2963542118052368110,mptcp 4 rem-addr id 1], length 0
// so somehow the client sent the "rem-addr" mptcp option to the server. The question is why, and how to figure out the reason.
Thanks.
Kang-sen
|
If I'm not mistaken, this To come back to a previous question: how are your MPTCP endpoints managed? Manually or via an automated tool (Network-Manager? mptcpd?) or script? |
The way I managed (configured) the mptcp endpoint is with the following command. I supposed that is manual way.
// at the client side:
ip mptcp endpoint add 198.18.0.6 dev tunUDP1 id 1 subflow
ip mptcp endpoint add 198.18.16.6 dev tunUDP2 id 2 subflow backup
No "ip mptcp" commands are entered in the server side.
You mentioned that the rem-addr may be caused by the endpoint being removed. But how come right after the old subflow is removed, almost immediately the same subflow is created again?
Kang-sen
ip mptcpMessage ID: ***@***.***>
|
I am puzzled by the client side "ip mptcp monitor" output. There are only two endpoints, but the monitir output seems to suggest that there are 3 locid, 0,1,2. Where does that locid=0 come from? // ip mptcp monitor at the client: [ SF_CLOSED] token=1f1c87d0 remid=0 locid=0 saddr4=198.18.0.6 daddr4=198.18.140.1 sport=60178 dport=9000 backup=0 —--------------------------------------- root@10053:~/# ip mptcp e —----------------------------------------------------------- |
I don't know. Except a bug, (or something I missed), this should only be triggered by the userspace changing the MPTCP endpoints. Are you sure there is nothing else that would modify the MPTCP endpoints? (mptcpd, network-manager, scripts, etc.) If you cannot find anything doing that (
The addresses of the initial subflow have a special ID: 0. That's in MPTCP specs. |
If you cannot find anything doing that (tcpdump with NLMon might help), it would really help if you can update the kernel, at least to a more recent 6.1.y version.
[ksl] I have not tried the way of running "tcpdump with NLMon". I did upgraded kernel from 6.1.71 to 6.1.109. And the problem with subflows being released and reestablished still happen. I did find something odd, I have two mptcp connections, one runs between openvpn, and the other runs between ssh. Each mptcp connection consists of two subflows. If I kill the openvpn connection, then the subflows between ssh are stable. In any other combinations, the subflows are released and reestablished.Message ID: ***@***.***>
|
I just build the openwrt image with nlmon kernel module. Then I issued the following commands at the openwrt shell prompt:
Then I ran wireshark to examine the nlmon.pcap file. I am not sure how to find readable information. Can you help, Matthieu? |
I have installed nldecap.py on a ubuntu host, but it does not work now. My host runs python3.9. |
Mmh, in your setup, In
Sorry, I don't remember what you are supposed to see exactly. But the idea is to check if MPTCP endpoints are being changed by the userspace. When you do |
Also, why do you need OpenVPN? Typically, MPTCP is used with a proxy, e.g. a transparent proxy intercepting the TCP connections created by a client connected to the router, then communicating using MPTCP with a proxy server, before reaching the end server. What is described on OpenMPTCPRouter website for example. In this case, you don't need to use MPTCP through the OpenVPN tunnel. |
I did a test of running tcpdump to capture the nlmon0 interface. Then I did "ip mptcp endpoint delete" to delete one of the two mptcp endpoints. After I examined the pcap file with wireshark, the wireshark does not show anything noticeable. I tried to run nldecap.py on ubuntu 24.04, and it does not run properly. |
Can you eventually share (a zipped version of) this file here please? Do you have the issue without OpenVPN tunnels (see my previous comment: #526 (comment)).
No, that's what you should see when using an up-to-date kernel: the addresses of the initial subflows should have this special local/remote ID 0. See #526 (comment) |
Matttbe: Thanks for the help. |
@kangsenlu : with Paolo's help, we looked at the nlmon traces:
Just to be sure this has been interpreted correctly, it might help to:
|
Hi, Matthieu: |
That makes sense, the only thing that could cause the behaviour you had.
Not that I think of no, but I rarely use nlmon. But I guess you will only make this mistake once ;) I guess we can close this issue then. |
I have setup mptcp upstream on two hosts. I have enabled mptcp feature, configured both the client host and the server host to use two channels for mptcp subflow purposes. I have seen the client applications openvpn and openssh-client connected to the openvpn and sshd in the server with mptcp connections, and each mptcp connection consists of two subflows of their own.
However, when I run "netstat -net" to monitor all the tcp connections, I noticed all 4 subflows are constantly being established and released somehow. I ran "ip mptcp monitor", the output confirmed what I saw. I have captured the ip mptcp endpoint show output and the in the ip mptcp monitor output here.
On the server host
Ip mptcp monitor
On the client host
The text was updated successfully, but these errors were encountered: