Requests hanging for a long period before getting processed

[ShaleenAg]

We are running a Django Ninja app with Gunicorn + Uvicorn workers, and using Grafana/Tempo for tracing, One thing we have noticed is that a lot of our requests are in a pending state for a long period of time before they get executed, and on looking at traces on Grafana, we are getting a lot of Http receive events (500-700+) on each request before it gets processed, this has really eaten in our response times, Is there a way we can reduce this latency and get the request processed faster?

For more context, we have a mix of sync and async apis and none of our apis are I/O heavy and rely on third party api calls for the majority of processing, We tried testing it out in with only async apis running, and even then response processing latency is very high.

We are running 5 workers on a 2vcpu pods , and even if we do 3 parallel requests, latency gets very large, our actual api response times should be 10-15 seconds and we are getting 30 second response times on 3 parallel requests, and if we do 10+ parallel requests, the response times jump to minutes, despite the fact we are running 10 pods with each pod running 5 workers, so theoretically, we should easily be able to serve 20-30 parallel requests.

Is there anything we can do to debug this further? this might be a possible duplicate of #2467. On going through traces on Grafana, most of our latency is tied up in http receive spans after which the request gets processed.

Example span on grafana: image