server hanging with 8 threads

[vcombey]

Bug Report

1. the bug occurs in rocket = "0.4.5" and master at rev = "549c924"

2. where the issue occurs: NAME="Ubuntu" VERSION="20.04 LTS (Focal Fossa)"

I have workers = 4 in Rocket.toml

3. the server is hanging, it stop reponding to any request
   It occurs randomly, but in the log there are multiple route /video/upload wich did not respond: this is the last lines of the logs.

 POST /video/upload:=> Matched: POST /video/upload (upload)
 Error: accept error: tls handshake eof
 POST /video/upload:
     => Matched: POST /video/upload (upload)
 POST /video/upload:
   => Matched: POST /video/upload (upload)
this route has the particularity of having a DataGuard.

and ls /proc/1/task gives

1  417	418  419  420  421  422  423  471  472

So it seems that all 4 workers are blocked in a connection, but I don't know exactly how to reproduce/why the bug appears

[SergioBenitez]

This is by design, or rather, an unfortunate repercussion of building on hyper, which has no mechanism by which to time out stalling connections. As a result, if a connection is held open, a thread is consumed. If all worker threads are consumed, the server cannot make progress.

In 0.5, we migrate to async, which will allow the server to make progress at the expense of memory usage due to maintaining open connections; hyper continues to have no proper time out functionality. #1405 tracks one possible fix, though it's unclear if the approach is implementable externally, given hyper's deficiencies. Thus, the only sound advice remains: always deploy a Rocket application behind a reverse proxy like NGINX that properly terminates connections. We should look to make this advice explicit in Rocket's guide. Let's track that in #171.

[Lazare-42]

Thanks for your answer @SergioBenitez. We were probably under a "SlowLoris" attack, or having someone with a really slow internet connection uploading some videos.

So we will configure our NGINX to properly terminate those connections.

Again, thank you for your swift answer.

[SergioBenitez]

@Lazare-42 It may not have been an attack. Folks run scans of the web all the time, with clients that act strangely, purposefully or accidentally, perhaps expecting the server to "defend" itself. If your server is expecting long-lived connections, I would also advocate for increasing Rocket's worker count; 0.5 will ameliorate this latter issue entirely.