Throttling requests to Azure OpenAI based on the current usage

[Ceglowa]

Hi,
I am trying out LiteLLM for load balancing the requests to Azure OpenAI.
I understand that you can have multiple deployments of models and use LiteLLM for routing your requests to certain deployments.
However, is it possible to have one deployment of OpenAI model, then have multiple requests being sent and then LiteLLM with usage of Redis can throttle the requests to avoid the RateLimitErrors? So that, we don't need to retry the requests and we could have some waiting time to make sure that the request will pass.

Thanks for the reply in advance

[ishaan-jaff]

Yes - use routing_strategy="usage-based-routing-v2"

[ishaan-jaff]

doc: https://docs.litellm.ai/docs/routing#routing-strategies---weighted-pick-rate-limit-aware-least-busy-latency-based-cost-based

[Ceglowa]

Thanks for the hint. I was trying it out yesterday. I have a question if I understand its behaviour correctly.
Suppose the following use case:

• I only have one deployment of Azure OpenAI
• This deployment has a 90 RPM
• In the same second I want to send 100 requests that are being asynchronously sent through LiteLLM to my Azure OpenAI deployment

Do i need to care about doing some waiting time between those calls? I was trying it out and I saw that it was very prone to errors. Or in those case when the number of calls is exceeding the RPM and I don't have any available deployments, I need to add some waiting time between the requests?

[ishaan-jaff]

@Ceglowa Can i see your config.yaml and what errors you see?

If it's faster we can debug this over a call (promise to have this solved by end of call). Sharing a link to my cal for your convenience:
https://calendly.com/d/4mp-gd3-k5k/berriai-1-1-onboarding-litellm-hosted-version?month=2023-10

[denisergashbaev]

@ishaan-jaff This feature seems to be missing in the documentation. Could you also please point at the implementation in the code of this new strategy?

[ishaan-jaff]

docs: https://docs.litellm.ai/docs/routing#advanced---routing-strategies-%EF%B8%8F
code for this: https://github.com/BerriAI/litellm/blob/main/litellm/router_strategy/lowest_tpm_rpm_v2.py

@denisergashbaev - I also DM'd you on Linkedin to setup a support channel so we can help with your issues faster