Http client has an issue with hyper when body is not read.

[petm-dec]

Hello,

I have a hyper server that sends data as response to GET requests. The amount of data is 500K (or more).
We observe "connection reset by peer" on the client side, but only if the body is not read.
If I add this to the handler function, the issue goes away:
let _ = hyper::body::aggregate(request.into_body()).await;

This does not happen with a go http server, which always reads the body to avoid issues with badly behaving clients. Also does not happen with a rust server implemented in rocket.

The client in this case is python asyncio based, either aiohttp or httpx. But only under special circumstances.
I suspect the client does not correctly implement the RFC as discussed for example here: golang/go#15527 (comment)

Note that in this case, the size of the body is 0! But still I need to read it away, or I see errors on the client side.

I did a tcpdump trace. This showed the following difference between hyper reading the body or not when the error (on the client side) occurs:

In the case where the body is read explicitly, I see the connection SYN & ACK, the GET request, a number of packets of various sizes containing the data sent back to the client (some have a PUSH flag). Sometimes the connection is closed by the client (FIN), sometimes it is reused for the next request. The last packet seen is an ACK packet , from client to server, for the last received data.

Here is the tcpdump output where the error occurs. The server uses port 10081.
Here we see: SEQ / RST / ACK, before the ACK is sent from client to server for the last data, the server seems to write a RST, i.e. forcibly closes the connection? Does anyone know what could be the reason for that?

14:15:54.805969 IP 192.168.50.245.10081 > 192.168.42.107.35934: Flags [.], seq 1163001:1168793, ack 1120, win 32, options [nop,nop,TS val 1204328876 ecr 407035265], length 5792
14:15:54.805973 IP 192.168.42.107.35934 > 192.168.50.245.10081: Flags [.], ack 1168793, win 543, options [nop,nop,TS val 407035267 ecr 1204328876], length 0
14:15:54.806024 IP 192.168.50.245.10081 > 192.168.42.107.35934: Flags [.], seq 1168793:1176033, ack 1120, win 32, options [nop,nop,TS val 1204328876 ecr 407035265], length 7240
14:15:54.806029 IP 192.168.42.107.35934 > 192.168.50.245.10081: Flags [.], ack 1176033, win 547, options [nop,nop,TS val 407035267 ecr 1204328876], length 0
14:15:54.806079 IP 192.168.50.245.10081 > 192.168.42.107.35934: Flags [P.], seq 1176033:1178929, ack 1120, win 32, options [nop,nop,TS val 1204328876 ecr 407035265], length 2896
14:15:54.806080 IP 192.168.50.245.10081 > 192.168.42.107.35934: Flags [.], seq 1178929:1180377, ack 1120, win 32, options [nop,nop,TS val 1204328876 ecr 407035265], length 1448
14:15:54.806081 IP 192.168.50.245.10081 > 192.168.42.107.35934: Flags [R.], seq 1180377, ack 1120, win 32, options [nop,nop,TS val 1204328876 ecr 407035265], length 0
14:15:54.806081 IP 192.168.50.245.10081 > 192.168.42.107.35934: Flags [R], seq 3494006057, win 0, length 0
14:15:54.806082 IP 192.168.50.245.10081 > 192.168.42.107.35934: Flags [R], seq 3494011849, win 0, length 0
14:15:54.806083 IP 192.168.50.245.10081 > 192.168.42.107.35934: Flags [R], seq 3494017641, win 0, length 0
14:15:54.806083 IP 192.168.50.245.10081 > 192.168.42.107.35934: Flags [R], seq 3494023433, win 0, length 0
14:15:54.806091 IP 192.168.42.107.35934 > 192.168.50.245.10081: Flags [.], ack 1178929, win 548, options [nop,nop,TS val 407035267 ecr 1204328876], length 0

[sfackler]

The RST is terminating the connection since the request body was not fully read, and the server can't continue to reuse that connection.

Hyper is designed to be a low level interface. If you want to always read request bodies, you would need to write that logic in your handler (or use a higher level server API that wraps hyper and does it for you). Rocket is presumably an example of one of those higher level APIs that does so.

[petm-dec]

I tested it with another client, same hyper server, not reading the body. Here I do 2 GET requests, and I observe that the connection is reused and the server does not send RST.

Also, if I actually read the body, then len() is always 0 (the client does not really send data in this case).

Should the server, at TCP level, not send the ACK before sending RST in any case?

[sfackler]

Is one client sending Conent-Length: 0 and the other sending a chunked 0-length body?

The kernel handles ACKs and RSTs - I don't think there's anything the userspace library can do about that.

[petm-dec]

The client with issue is not sending content-length: 0, but I see Content-Type: chunked, so maybe yes a 0-length body.
I see that now hyper wants to close the connection because data is present (RST means close(2) with data present in the receive buffer).

But is the timing correct? RST is sent before the ACK.
Does this not mean that the connection is somehow closed, before hyper could verify that the data that were sent have arrived, since the RST is sent before the ACK 1178929 were received.

The issue happens in about 25% of cases. A run I just did resulted in a few reuses of the connection (no FIN or RST seen between GET requests). Then a number of larger segments with delayed ACK's, followed by RST while many ACKs are still under way; these get each answered with another RST because the connection has been closed before the ACKs for the data have been processed.

It looks like hyper is closing a connection, before it could ensure that all data have been received:

# request (2 SEQ packets + ACK)  reusing connection
16:52:51.051437 IP 192.168.42.107.40956 > 192.168.50.245.10081: Flags [P.], seq 374:742, ack 530097, win 235, options [nop,nop,TS val 416451510 ecr 1213745114], length 368                                
16:52:51.051540 IP 192.168.42.107.40956 > 192.168.50.245.10081: Flags [P.], seq 742:747, ack 530097, win 235, options [nop,nop,TS val 416451511 ecr 1213745114], length 5                                  
16:52:51.051752 IP 192.168.50.245.10081 > 192.168.42.107.40956: Flags [P.], seq 530097:595257, ack 747, win 32, options [nop,nop,TS val 1213745121 ecr 416451510], length 65160                            
16:52:51.051752 IP 192.168.50.245.10081 > 192.168.42.107.40956: Flags [P.], seq 530097:595257, ack 747, win 32, options [nop,nop,TS val 1213745121 ecr 416451510], length 65160                            
16:52:51.051811 IP 192.168.50.245.10081 > 192.168.42.107.40956: Flags [P.], seq 595257:660417, ack 747, win 32, options [nop,nop,TS val 1213745121 ecr 416451510], length 65160                            
16:52:51.051862 IP 192.168.50.245.10081 > 192.168.42.107.40956: Flags [P.], seq 660417:664761, ack 747, win 32, options [nop,nop,TS val 1213745121 ecr 416451510], length 4344                             
16:52:51.052048 IP 192.168.50.245.10081 > 192.168.42.107.40956: Flags [R.], seq 664761, ack 747, win 32, options [nop,nop,TS val 1213745121 ecr 416451510], length 0                                       
16:52:51.052122 IP 192.168.42.107.40956 > 192.168.50.245.10081: Flags [.], ack 537337, win 239, options [nop,nop,TS val 416451511 ecr 1213745121], length 0                                                
16:52:51.052134 IP 192.168.50.245.10081 > 192.168.42.107.40956: Flags [R], seq 348986763, win 0, length 0                                                                                                  
# ACK arriving after the first RST, connection closed prematurely?
16:52:51.052214 IP 192.168.42.107.40956 > 192.168.50.245.10081: Flags [.], ack 544577, win 242, options [nop,nop,TS val 416451511 ecr 1213745121], length 0                                                
16:52:51.052214 IP 192.168.42.107.40956 > 192.168.50.245.10081: Flags [.], ack 550369, win 245, options [nop,nop,TS val 416451511 ecr 1213745121], length 0                                                
16:52:51.052218 IP 192.168.50.245.10081 > 192.168.42.107.40956: Flags [R], seq 348994003, win 0, length 0                                                                                                  
16:52:51.052227 IP 192.168.50.245.10081 > 192.168.42.107.40956: Flags [R], seq 348999795, win 0, length 0                                                                                                  
...

[sfackler]

I haven't checked the code, but my hypothesis is that Hyper is going to read from the stream until it sees the end of the headers. If the final read happens to also include the empty chunked body, it detects that it's read the entire request and is willing to keep the connection open. If it doesn't see the empty chunked body in the final read (e.g. because it was sent in a separate TCP packet that hasn't yet been read by the kernel) it will terminate the connection after responding if the handler code does not force it to process the body.

[petm-dec]

Thanks for your explanation.

In that case, one should always do this and read the body,
because otherwise sending the reply might be discarded,
but for Http it is valid, to ignore the body and send a result.

In hyper this then could lead to an abort of the handling and not properly finish sending the reply.

It is no problem to add that to our code. Maybe one of the examples should reflect this?