Iperf3 failing to complete handshake.

[bgee-eng]

I have two systems setup to run with udp packets. When I send the client command to the server, the system never completes the initial handshake. With a tcpdump, I can see the client side Sends a total of 3 eth messages, and receives a total of two. On the Server Side tcpdump, I see a total of 3 eth packets received and it sending out only 2. Any idea as to why the client does not continue with the rest of the handshake and stops after Receiving the second packet from the Server?

Server Client

1. Sent 1) Received
2. Receive 2) Sent
3. Sent 3) Receive
4. Sent 4) Receive
5. Received 5) Sent

No other traffic is generated and Iperf3 terminal just hangs.

[davidBar-On]

I am not sure I understand the sequence that you should. E.g. what is meant by "Sent 1) Received"? Sent by Client or Server? Who is the receiver? Does "1)" mean a packet number?

Also, can you add more information about each packet? At least what is the packet size. It would help if you can include the data part of each packet when applicable (e.g. by viewing the tcpdump output using Wireshark).

[bgee-eng]

The left column is intended to be the interaction of the Server and the right column is meant to be the Client side. The number is the order of the packets each respective side either sent or received as discovered on a tcpdump.

MTU is 1500 and the command used was "iperf3 -c -u".

On the client side, the terminal is stuck at "connecting to port".

I am unable to upload any captures here. Can this discussion be continued over email?

[davidBar-On]

Thanks for the clarification. However, I am not sure I understand the sequence as the first message is sent from the client to the server, and if I understand your clarification correctly, packet "1)" is sent from the server to the client.

For now I prefer to continue the discussion here, so others may be able to help. If you cannot upload the captures, can you just specify the length of each of the 5 packets?

[bgee-eng]

I tried a recapture and perhaps this is a better explanation of what I am seeing. Got a total of 17 packets this time around but still the same issue of not competing the handshake.

Packet 1 : Client Sent to Server (tcpdump Length 0, IPV4 Length = 60 decimal)
Packet 2 : Server Sent to Client (tcpdump Length 0, IPV4 Length = 60 decimal)
Packet 3 : Server sent to Client (tcpdump Length 0, IPV4 Length = 60 decimal)
Packet 4 : Client Sent to Server (tcpdump Length 0, IPV4 Length = 60 decimal)
Packet 5 : Server Sent to Client (tcpdump Length 0, IPV4 Length = 60 decimal)
Packet 6 : Client Sent to Server (tcpdump Length 0, IPV4 Length = 52 decimal)
Packet 7 : Client Sent to Server (tcpdump Length 37, IPV4 Length = 89 decimal)
Packet 8 : Server Sent to Client (tcpdump Length 0, IPV4 Length = 52 decimal)
Packet 9 : Server Sent to Client (tcpdump Length 1, IPV4 Length = 53 decimal)
Packet 10 : Client Sent to Server (tcpdump Length 0, IPV4 Length = 52 decimal)
Packet 11 : Client Sent to Server (tcpdump Length 0, IPV4 Length = 52 decimal)
Packet 12 : Client Sent to Server (tcpdump Length 4, IPV4 Length = 56 decimal)
Packet 13 : Server Sent to Client (tcpdump Length 0, IPV4 Length = 52 decimal)
Packet 14 : Client Sent to Server (tcpdump Length 118, IPV4 Length = 170 decimal)
Packet 15 : Client Sent to Server (tcpdump Length 118, IPV4 Length = 170 decimal)
Packet 16 : Client Sent to Server (tcpdump Length 118, IPV4 Length = 170 decimal)
Packet 17 : Client Sent to Server (tcpdump Length 118, IPV4 Length = 170 decimal)

[davidBar-On]

It seems that the problem happens in the parameters exchange phase (i.e. that the 1 data byte in packet 12 is 0x09). Packets 14-17 are probably the packet that contains the parameters the client sent and its retries. More info may help the evaluation:

1. What is the iperf3 version of the client and the server (from iperf3 -v)?
2. What is the operating system(s) used for the client and server?
3. Is the packets log from the server or client side? Did packet 14 (tcpdump length 118) received by the server machine? The the server send a reply that was not received by the client machine?
4. After this problem happens, is the server stack? I.e. is it required to kill the server and start new one to repeat the packet sequence?
5. Running the client and server with the -V -d options may add more information. If used, what is the last message printed by the client and server?

[bgee-eng]

1. Iperf3 Version is 3.9 (cJSON 1.7.13) on both systems.
2. OS is Ubuntu 20.04.5 LTS on both systems.
3. The Packet Log Provided was captured on the Server Side. Since it was captured there, I assume the server received Packet 14. I don't see any server replies after packet 14.
4. If I do not kill the server, and Ctrl C out of the Client and retry, the handshake fails earlier on. For example I might see only 9 packets come through in the same order as above. Killing the server and restarting results in 17 packets.
5. Server only ever Prints "Server Listening on Port". Last thing client prints is 'Target Bitrate:1048576'

[davidBar-On]

Last thing client prints is 'Target Bitrate:1048576'

Just to make sure, did you set the -d debug option for the client? From the analysis below, I expect that the last message the client prints is the parameters sent to the server: "send_parameters: ...".

I believe that the 4 data bytes (tcpdump length) of packet 12 is the length of the packet 14 which includes the parameters JSON. If this is correct, then packet 12 4 data bytes value should be 118 (0x0076). Is this correct or the value is different? (Using the -w option of tcpdump you can view the value by Wireshark.)

If this is correct, then after receiving the JSON length in packet 12 (probably acked by packet 13), the server is trying to read this amount of data (received in packet 14). It seems that somehow the server is stack in reading, e.e because there is a mismatch between the length in packet 12 and the length of packet 14.

Can you send the full client command that you are using (obscuring the IP address if needed)? Did you try TCP test works (by removing the -u option)?

[bgee-eng]

Command for server : "iperf3 -s -V -d"
Command for client: "iperf3 -c ip_addr -u -V -d"

Yes, the last four bytes of packet 12 are (0x0000_0076).

I did try the tcp mode and I still have trouble. I did not capture the tcp mode packets.

Just to clarify,
Packet 12 tells the server to listen for a message of size 118?
Packet 13 is the acknowledge from the server to client.
Packet 14 is the message from client to server also of size 118. I'm not seeing the mismatch in length.

[davidBar-On]

Packet 12 tells the server to listen for a message of size 118?

Correct. As the length agrees with packet 14, I am not sure what may be the issue, especially because the client doesn't show the "send_parameters: ..." which should be printed before packet 14 is sent.

Are you able to install/build a newer iperf3 version? It includes --rcv-timeout and --snd-timeout options that may help (--snd-timeout is not available in all OSs - use iperf3 -h to determine that). If you set bot options to 5000ms for example, for both client and server, then then if the client or server will timeout it may help determining where the problem is. You may also run tests once with greater timeout for the server and then greater timeout for the client, to determine if either or both timeout. Also, different timeout for each option can determine whether the timeout is for sending or receiving.

[bgee-eng]

I upgraded iperf3 on both systems to 3.13. After adding the --rcv-timeout 5000 and --snd-timeout 5000, I get the following on the side that is the client.

Control Connection MSS 1448
Setting UDP Block size to 1448
send_parameters:
{
"udp": true,
"omit": 0,
"time": 10,
"num": 0,
"blockcount": 0,
"parallel": 1,
"bandwidth": 1048576,
"pacing_timer": 1000,
"client_version": "3.13"
}
Time:...
Connecting to host...
Cookie: ...
Target Bitrate: 1048576
iperf3: error - unable to send control message - port may not be available, the other side may have stopped running, ect.: Broken Pipe

The only thing new is that last line compared to the previous version. The "send_paramenters" always came before the "Target Bitrate".
The Server side is still running, no messages printed on that side.

[davidBar-On]

Thanks for the information. It seem that for some reason the server got stack during the parameters exchange with the client. From the information available I can't tell what may be the problem. I therefore prepared a debug version, with several additional debug messages: https://github.com/davidBar-On/iperf/tree/issue-1470-client-server-handshake-fail.

Can you build and try this version? The client may show additional "select failed ..." message and the server should show several additional ""**TEST: ..." messages and/or "select failed ..." message.

[bgee-eng]

After installing your debug version and running the same commands as above, the client side results in the exact same messages.

The Server now displays:

"**TEST: calling get_parameters();

[davidBar-On]

Thanks for taking the time for testing. As the issue seem to be in get_parameters(), I have added several test messages. Can you retry the test with the updated code?

[bgee-eng]

The Server Side now prints the following:

**TEST: calling get_parameters();
**TEST in get_parameters: enter;
**TEST in JSON_read: enter;
**TEST in JSON_read: first Nread() returned JSON size= 8e000000;
**TEST in JSON_read: ntol() returned hsize=8e;
**TEST in JSON_read: calloc() returned 0x55bbea24fde0;

[bgee-eng]

Another interesting datapoint. If I pass the -R parameter on the client side, iperf3 seems to work without issues. Not sure if that could help explain anything.

[bgee-eng]

The -R sometimes fails after sending the 10 data packets. In this scenario, the Server prints the following:

**TEST in JSON_read: enter;
**TEST in JSON_read: first Nread() returned JSON size= 42010000;
**TEST in JSON_read: ntol() returned hsize=142;
**TEST in JSON_read: calloc() returned 0x55ba504a8a70;

[davidBar-On]

Another interesting datapoint. If I pass the -R parameter on the client side, iperf3 seems to work ... The -R sometimes fails after sending the 10 data packets. ...

This is strange, as in the stage of failure the -R does not have any effect, except for changing the size of the packet that includes the parameters list (packet 14).

As calloc() was successful, it seems that the failure is in Nread() which reads the packets. I have added test messages to Nread(). Can you try this version?

In the last set of Nread() messages, one of the test messages should be "**TEST in JSON_read: ntol() returned hzise=.... (0x...)". This is the value received in packet 12 and should be the same as the length of packet 14. Can you verify that?

[bgee-eng]

Server Side:
**TEST in Nread: enter nleft=37, buf=0x56051b8e594;
**TEST in Nread: before read() nleft=37 buf=0x56051b8e594;
**TEST in Nread: after read() r=37, nleft=37;
**TEST in Nread: before rtun from read() nleft=0, count=37;
**TEST: calling get_parameters();
**TEST in get_parameters: enter;
**TEST in JSON_read: enter;
**TEST in Nread: enter nleft=4, buf=0x7ffdd03DEFB4;
**TEST in Nread: before read() nleft=4 buf=0x7ffdd03DEFB4;
**TEST in Nread: after read() r=4, nleft=4;
**TEST in Nread: before rtun from read() nleft=0, count=4;
**TEST in JSON_read: first Nread() returned JSON size= 8e000000;
**TEST in JSON_read: ntol() returned hsize=8e;
**TEST in JSON_read: calloc() returned 0x55bbea24fde0;
**TEST in Nread: enter nleft=142, buf=0x56051fb8eee0;
**TEST in Nread: before read() nleft=142 buf=0x56051fb8eee0;

Packet 8 ends in 00 8E
Packet 10 size is 142 = 0x8E.
sane as Packet 11, 12, 13, 14, and 15.

[davidBar-On]

It seems that read() is not returning. This is probably because of some problem in the TCP stack, as it is not acking receiving the last packet from the client (and therefore the client retries sending it).

This may be a system issue, not iperf3, but there is still one test approach I can think of. Before the read() I added select() with timeout of 3 secs. I hope that maybe there is some timing issue, so waiting to something to be received before the read() may help.

Can you try this version? Note the "**TEST in Nread: select() return value=...." message when select() returns. Return value 0 means it timed out and 1 means success (regardless of errno value).

[bgee-eng]

Server Side:
**TEST in Nread: enter nleft=37, buf=0x56051b8e594;
**TEST in Nread: before select() fd=4;
**TEST in Nread: select() return value=1, errno=0 - Success;
**TEST in Nread: before read() nleft=37 buf=0x56051b8e594;
**TEST in Nread: after read() r=37, nleft=37;
**TEST in Nread: before rtun from read() nleft=0, count=37;
**TEST: calling get_parameters();
**TEST in get_parameters: enter;
**TEST in JSON_read: enter;
**TEST in Nread: enter nleft=4, buf=0x7ffdd03DEFB4;
**TEST in Nread: before select() fd=4;
**TEST in Nread: select() return value=0, errno=0 - Success;
**TEST in Nread: before read() nleft=4 buf=0x7ffdd03DEFB4;
**TEST in Nread: after read() r=4, nleft=4;
**TEST in Nread: before rtun from read() nleft=0, count=4;
**TEST in JSON_read: first Nread() returned JSON size= 8e000000;
**TEST in JSON_read: ntol() returned hsize=8e;
**TEST in JSON_read: calloc() returned 0x55bbea24fde0;
**TEST in Nread: enter nleft=142, buf=0x56051fb8eee0;
**TEST in Nread: before select() fd=4;
**TEST in Nread: select() return value=0, errno=0 - Success;
**TEST in Nread: before read() nleft=142 buf=0x56051fb8eee0;

Looks like there were two timeouts. It doesn't look like we are missing a packet on the server side. What might be causing this? Corrupted packet?

[davidBar-On]

I am not sure what is the reason for the first timeout (before reading the 4 bytes). The only thing I can think of that there may be a delay in receiving the data. In the tcpdump output, do you see a large time diff (seconds) between the 4 bytes and the 0x8e bytes packets?

As it seems that the select() doesn't help I removed it and made the last change I can think of. In Nread() added to each message a time stamp in micro-sec, relative to the start of this Nread. Also I changed the read() so it will read only one byte - if the large packet is partly received we will see that. Can you try this version?

[bgee-eng]

**TEST in Nread: enter nleft=37, buf=0x560d3f98a594;
**TEST in Nread: now=252332992310, time-diff=2;
**TEST in Nread (time=12): before read() nleft=37 buf=0x560d3f98a594;
**TEST in Nread (time = 51): after read() r=1, nleft=37;
**TEST in Nread (time=67): before read() nleft=36 buf=0x560d3f98a595;
**TEST in Nread (time = 87): after read() r=1, nleft=36;
**TEST in Nread (time=101): before read() nleft=35 buf=0x560d3f98a596;
**TEST in Nread (time = 131): after read() r=1, nleft=35;
**TEST in Nread (time=146): before read() nleft=34 buf=0x560d3f98a597;
**TEST in Nread (time = 164): after read() r=1, nleft=34;
......
**TEST in Nread (time=1346): before read() nleft=1 buf=0x560d3f98a5b8;
**TEST in Nread (time = 1372): after read() r=1, nleft=1;
**TEST in Nread (time=1389): before rtun from read() nleft=0, count=37;
**TEST: calling get_parameters();
**TEST in get_parameters: enter;
**TEST in JSON_read: enter;
**TEST in Nread: enter nleft=4, buf=0x7ffc54b607f4;
**TEST in Nread: now=252332993905, time-diff=0;
**TEST in Nread: before select() fd=4;
**TEST in Nread (time =19): before read() nleft=4 buf=0x7ffc54b607f4;
**TEST in Nread (time = 5231): after read() r=1, nleft=4;
**TEST in Nread (time =5253): before read() nleft=3 buf=0x7ffc54b607f5;
**TEST in Nread (time = 5274): after read() r=1, nleft=3;
**TEST in Nread (time = 5293): before read() nleft=2 buf=0x7ffc54b607f6;
**TEST in Nread (time = 5315): after read() r=1, nleft=2;
**TEST in Nread (time =5354): before read() nleft=1 buf=0x7ffc54b607f7;
**TEST in Nread (time = 5389): after read() r=1, nleft=1;
**TEST in Nread (time=5407): before rtun from read() nleft=0, count=4;
**TEST in JSON_read: first Nread() returned JSON size= 8e000000;
**TEST in JSON_read: ntol() returned hsize=142 (0x8e);
**TEST in JSON_read: calloc() returned 0x560d3f98ade0;
**TEST in Nread: enter nleft=142, buf=0x560d3f98ade0;
**TEST in Nread: now=252332999460, time-diff=0;
**TEST in Nread (time =25): before read() nleft=142 buf=0x560d3f98ade0;

Looks like there is a 5ms delay between the first byte and the subsequent bytes for the 4 byte message. Since the packet arrives at the same time, any idea what might cause this?

For the 142 byte message, why doesn't it try to read that? Below is the timestamps from TCP Dump on the server.

TCP Dump shows:
Packet 1 (12:29:14.903479) Client-> Server: Length 0
Packet 2 (12:29:14.903591) Server-> Client: Length 0
Packet 3 (12:29:14.909034) Client-> Server: Length 0
Packet 4 (12:29:14.909054) Client-> Server: Length 37
Packet 5 (12:29:14.909174) Server-> Client: Length 0
Packet 6 (12:29:14.910869) Server-> Client: Length 1
Packet 7 (12:29:14.916072) Client-> Server: Length 0
Packet 8 (12:29:14.916085) Client-> Server: Length 4
Packet 9 (12:29:14.916134) Server-> Client: Length 0
Packet 10 (12:29:14.916519) Client-> Server: Length 142
Packet 11 (12:29:15.131351) Client-> Server: Length 142
Packet 12 (12:29:15.343397) Client-> Server: Length 142
Packet 13 (12:29:15.767560) Client-> Server: Length 142
Packet 14 (12:29:16.632076) Client-> Server: Length 142
Packet 15 (12:29:18.329045) Client-> Server: Length 142
Packet 16 (12:29:21.722755) Client-> Server: Length 142

[davidBar-On]

Looks like there is a 5ms delay between the first byte and the subsequent bytes for the 4 byte message. Since the packet arrives at the same time, any idea what might cause this?

The 5ms delay is the time between Packets 6 and 8. After sending packet 6 the server immediately waits for packet 8.

For the 142 byte message, why doesn't it try to read that? Below is the timestamps from TCP Dump on the server.

This is the problem ... For some reason the TCP stack stops working, at least for the server's connection, so it is even not sending the ack for receiving the packet (this is done by the TCP stack, not iperf3).

Currently I don't have any idea for iperf3 changes that can help analysis. Other approaches should be used to analyze the problem. Some actions I can think of:

1. Ensure that the 142 bytes packet is not too large (although I am not sure that this is a real possibility). Run sysctl -a | grep tcp_rmem (as administrator). The first number is the minimum TCP receive buffer size.
2. Run the iperf3 client on the server's machine and the server on the client's machine. Is there any change?
3. When the server is not responding, run a second server and try to connect to it from a second client (using a different port, e.g. -p 5211). Is the second server stack at the same point or before it?
4. Measure the server's machine and iper3 app CPU usage. Is the machine CPU overloaded? Does the server consume CPU when it is stuck (e.g. because the TCP stack is in infinite loop and this si why it is not working)?
5. Use other apps that creates a TCP connection, to see if this is a general problem or just iperf3 problem. E.g. see here for using nc (netcat) or here for using telnet.
6. Use gdb to try to find where the server is stuck: find the server's pid by ps -a; run gdb --pid <pid>; use gdb's bt command to see the backtrace.

[bgee-eng]

1. net.ipv4.tcp_rmem = 4096 131072 6291456
2. Nothing changes on switching client and host
3. Changing ports when server doesn't respond results in "Unable to connect to server : Connection Refused"
4. CPU usage doesn't change and is low.
5. nc -z -v 22 results in [tcp/ssh] suceeded!.
   nc -z -v -u 123 results in "udp port ntp port unreachable" from tcpdump of other pc.

in iperf_accept at iperf_server_api.c:153
in iperf_run_server at iperf_server_api.c:579
in run at main.c:153
in main at main.c:108

[davidBar-On]

@bgee-eng, that for your effort to try resolving this issue. From the info you sent I don't see something that can help, but the following may help:

1. Can you verify in the tcpdump output that packet 10 (142 bytes - the parameters JSON) was sent to port 5021 as e.g. packet 4 (37 bytes - the cookie)?

2. Using nc, try sending TCP data to simulate the failed iperf3 interface, using iperf3 port and the attached different_line_lengths.txt file. Check whether all the file lines were successfully transmitted (note that the lines start with 1,2,...,9,A,B,....,Z)

• On the server side run nc -v clientIP -l 5021
• On the client side run: nc -v serverIP 5021 <different_line_lengths.txt

different_line_lengths.txt

3. I changed Nwrite() to wite() one char at a time. The idea is to see whether there is a problem related to the server's buffers handling. Can you test using this version? Nwrite() new test messages includes the target fd (file description). Can you list these test messages so I can verify that the fd is o.k.?

[bgee-eng]

1. All of the packets on tcp dump to the server are on port 5201. The port associated with the client changes.

2. Server: "Cannot assigned requested address"
   Client: "Connection Refused"

3. Print from Server Side with new version:

Starting Test: protocol : UDP, 1 streams, 1448 Byte Blocks, omitting 0 seconds, 10 second test, tos 0
**TEST in Nwrite: enter count=1, fd=4;
**TEST in Nwrite: now=354865265859, time-diff=0;
**TEST in Nwrite(time =20): after write() r=1, nleft =1;
**TEST in Nwrite(time =35): before retun from Nwrite() nleft = 0, count =1;
**TEST in Nread: enter nleft=1448, buf=0x560d3f98ade0;
**TEST in Nread: now=354865265935, time-diff=0;
**TEST in Nread (time =9): before read() nleft=1448 buf=0x560d3f98ade0;
**TEST in Nread (time =30): after read() r=-1, nleft=1448;
**TEST in Nread (time =57): read() failed. errno=11 - Resource Temporarily unavailable;
**TEST in Nread (time =73): before retun from Nread() nleft=1448, count=1448;
interval_len 1.001224 bytes_transferred 0
interval forces keep
...
iperf3: Server Restart (#1) during active test due to idle timeout for receiving data
iperf3: error - idle timeout for receiving data

[davidBar-On]

Server: "Cannot assigned requested address"
Client: "Connection Refused"

Oops! My mistake. No "clientIP" should be specified for the server ... Sorry for the mistake. The nc server command should be nc -v -l 5021. Can you retry running? (Also, please make sure that iperf3 is not running in parallel.)

**TEST in Nread: enter nleft=1448, buf=0x560d3f98ade0;

It seems that the parameters exchange was successful and that the server is trying to read the first "1448 Byte Block" (first data packet). Does tcpdump show that this is the case? I.e. that there are other packets after the 142 bytes packet. If this is the case, can you send the new tcpdump output?

**TEST in Nread (time =9): before read() nleft=1448 buf=0x560d3f98ade0;
**TEST in Nread (time =30): after read() r=-1, nleft=1448;
**TEST in Nread (time =57): read() failed. errno=11 - Resource Temporarily unavailable;

Was there any related error on the client side that may have caused this "Resource Temporarily unavailable;" error on the server side?

[bgee-eng]

The "nc -v -l" results in "getnameinfo: Temporary failure in name resolution"

There are significantly more packets that show up in tcpdump. closer to 60. biggest length size is 27. There are a bunch of smaller 1 or 2 byte lengths. Do you think the issue is related to our TCP rx memory buffer? The ping delay between the systems is close to 10 ms.

The thing from the client side is "ignoring short interval with no data." It prints test complete with summary results, but all values are 0.

[davidBar-On]

The "nc -v -l" results in "getnameinfo: Temporary failure in name resolution"

Per this answer try to add -n to the nc command: nc -v -l -n 5021.

There are significantly more packets that show up in tcpdump. closer to 60. biggest length size is 27. There are a bunch of smaller 1 or 2 byte lengths.

This is because Nwrite() now send byte by byte (the TCP stack on the client side may comment some bytes into one packet). I will have to change the code so that the data packet (the 1448 bytes) will be sent as one packet. This will allow to see if such packet was sent. Will do that tomorrow.

Do you think the issue is related to our TCP rx memory buffer? The ping delay between the systems is close to 10 ms.

That may be the case, but I am not sure what may cause such problem.
Ping is not using the TCP stack. 10ms may not be an issue, but it can be interesting to test what happens to the delay with large packets. E.g. ping -s 1448.

The thing from the client side is "ignoring short interval with no data." It prints test complete with summary results, but all values are 0.

Did the client print all the 10 intervals before quitting? In any case, can you try a test using TCP with relatively short packet size? E.g. iperf3 -c <IP> -V -d -l 512?

[bgee-eng]

Ok after passing the "-n" parameter, I can see that only down to line "W" shows up on the server side. Client side reports succeeded.

Sending Ping to the larger packet increases the latency by 1 ms.

The Client prints 1 interval while the host prints 4 intervals. As I repeat the process, iperf3 seems to be failing at different points in the handshake. This includes seeing on the client "errno -32 Broken pipe: unable to send control message - port may not be available"

[davidBar-On]

Ok after passing the "-n" parameter, I can see that only down to line "W" shows up on the server side. Client side reports succeeded.

1. Did the nc server report any error? What happens if you run the client again, keeping the same server running? Does the server show the messages again up to "W"?

Sending Ping to the larger packet increases the latency by 1 ms

That probably means that the delay is not an issue.

The Client prints 1 interval while the host prints 4 intervals. As I repeat the process, iperf3 seems to be failing at different points in the handshake. This includes seeing on the client "errno -32 Broken pipe: unable to send control message - port may not be available"

2. I assume by "host" you mean "server". I prepared yet another iperf3 version where only the JSON parameters (the 142 bytes packet) is sent byte by byte, since it seems to help. Can you retry the test using UDP and see what happens? Also, please check using tcpdump if and how many 1448 packets (the data packets) were received by the server machine.
3. Can you also try the new version using TCP with relatively short packets, e.g. -l 1448 (default is 128K)? Since with TCP the client waits for acks from the server, it may give more information. Also here, please check with tcpdump how many data packets were received by the server machine.

[bgee-eng]

the nc server did not report any errors. On repeated runs, the client continues to report suceeded runs. The Server continues printing the text up to "W".

UDP Iperf3
Packet 1 Client-> Server: Length 0
Packet 2 Server-> Client: Length 0
Packet 3 Client-> Server: Length 0
Packet 4 Client-> Server: Length 37
Packet 5 Server-> Client: Length 0
Packet 6 Server-> Client: Length 1
Packet 7 Client-> Server: Length 0
Packet 8 Client-> Server: Length 4
Packet 9 Client-> Server: Length 1
Packet 10 Client-> Server: Length 1
Packet 11 Client-> Server: Length 1
Packet 12 Client-> Server: Length 1
Packet 13 Client-> Server: Length 1
Packet 14 Server-> Client: Length 0
Packet 15 Server-> Client: Length 0
Packet 16 Server-> Client: Length 0
Packet 17 Server-> Client: Length 0
Packet 18 Server-> Client: Length 0
Packet 19 Server-> Client: Length 0
Packet 20 Client-> Server: Length 1
Packet 21 Client-> Server: Length 1
Packet 22 Client-> Server: Length 1
Packet 23 Client-> Server: Length 1
Packet 24 Server-> Client: Length 0
Packet 25 Server-> Client: Length 0
Packet 26 Server-> Client: Length 0
Packet 27 Client-> Server: Length 133
Packet 28 Server-> Client: Length 0
Packet 29 Server-> Client: Length 1
Packet 30 Client-> Server: UDP Length 4
Packet 31 Server-> Client: UDP Length 4
Packet 32 Client-> Server: Length 0
Packet 33 Server-> Client: Length 2
Packet 34 Server-> Client: Length 2
Packet 35 Server-> Client: Length 2
Packet 36 Server-> Client: Length 2
Packet 37 Server-> Client: Length 2
Packet 38 Server-> Client: Length 2
Packet 39 Server-> Client: Length 2
Packet 40 Client-> Server: Length 0
Packet 41 Server-> Client: Length 2
Packet 42 Server-> Client: Length 2

TCP Iperf3 with 1448 length
Packet 1 Client-> Server: Length 0
Packet 2 Server-> Client: Length 0
Packet 3 Client-> Server: Length 0
Packet 4 Client-> Server: Length 37
Packet 5 Server-> Client: Length 0
Packet 6 Server-> Client: Length 1
Packet 7 Client-> Server: Length 0
Packet 8 Client-> Server: Length 4
Packet 9 Client-> Server: Length 1
Packet 10 Client-> Server: Length 1
Packet 11 Server-> Client: Length 0
Packet 12 Server-> Client: Length 0
Packet 13 Server-> Client: Length 0
Packet 14 Client-> Server: Length 1
Packet 15 Client-> Server: Length 1
Packet 16 Client-> Server: Length 1
Packet 17 Client-> Server: Length 1
Packet 18 Client-> Server: Length 1
Packet 19 Server-> Client: Length 0
Packet 20 Server-> Client: Length 0
Packet 21 Server-> Client: Length 0
Packet 22 Server-> Client: Length 0
Packet 23 Server-> Client: Length 0
Packet 24 Client-> Server: Length 1
Packet 25 Client-> Server: Length 1
Packet 26 Server-> Client: Length 0
Packet 27 Server-> Client: Length 0
Packet 28 Client-> Server: Length 113
Packet 29 Server-> Client: Length 0
Packet 30 Server-> Client: Length 1
Packet 31 Client-> Server: Length 0
Packet 32 Server-> Client: Length 0
Packet 33 Client-> Server: Length 0
Packet 34 Client-> Server: Length 37
Packet 35 Server-> Client: Length 0
Packet 36 Client-> Server: Length 0
Packet 37 Server-> Client: Length 2
Packet 38 Server-> Client: Length 2
Packet 39 Server-> Client: Length 2
Packet 40 Server-> Client: Length 2
Packet 41 Server-> Client: Length 2
Packet 42 Server-> Client: Length 2
Packet 43 Server-> Client: Length 0
Packet 44 Server-> Client: Length 0
Packet 45 Client-> Server: Length 0
Packet 46 Client-> Server: Length 0
Packet 47 Server-> Client: Length 2
Packet 48 Server-> Client: Length 2

[davidBar-On]

@bgee-eng, I am not sure what are the two bytes packets the server sends at end of the list you sent for both UDP and TCP. My guess is that these are repeated retries of sending the "START" (0x01) and "RUNNING" (0x02) statuses. If my guess is correct, it means that somehow the client-server connection was broken at this stage somehow.

I don't have further ideas for how to continue evaluating this issue remotely. Unless others that follow this discussion can help, it seems that local evaluation is needed.

I suggest to first make sure that nc is working well in both directions. It will also help to have two nc instances running in parallel, each sending data to the other other direction.

Also, evaluating the traffic using Wireshark can help. This is by recording the tcpdump for Wireshark (using -w option) or using Wireshark directly for capturing the traffic.

Another evaluation approach is to try understanding the network architecture and find whether there may be a component that is causing issues. For example, are there routers/switches between the server and client machines that may cause a problem?

[bgee-eng]

Ok thanks for all the debug support.