Rate limit per key and model

[olad32]

Hi, is it planned to be able to apply rate limit tpm/rpm on key but with a different rate limit per model ? the same way we can already apply a budget for a key and a specific model as describe here https://litellm.vercel.app/docs/proxy/users#add-model-specific-budgets-to-keys
As of today a key can grant access to multiple model, but the rate limit applied globally to multiple model does not makes a lot of sense in practical use.
Thanks

[krrishdholakia]

hi @olad32 that's interesting

when would you do it this way?

[olad32]

This is defined by contract with each customer, this is the max tpm/rpm needed for the customer, the tpm/rpm is here to set a limit on llm instance that are shared between multiple customers, and guarantee a fair usage

[krrishdholakia]

So is the customer buying a pack for specific models with specific tpm/rpm limits

[krrishdholakia]

if not - why not just set it globally and let them use across models?

Context - trying to understand your scenario better

[salman-lilly]

@krrishdholakia when i am applying rpm limit per key to 10 then i am getting response of more than 10 request per minute.Some request are failing with error that is expected but why i am still getting more than 10 request responses any idea? ({"error":{"message":"Max parallel request limit reached Hit limit for api_key). Note-(Model deployment rpm is more than 100. i am setting rpm on key basis to control number of request per key)....(https://litellm.vercel.app/docs/proxy/users#add-model-specific-budgets-to-keys)why this behaviour?

[krrishdholakia]

@salman-lilly can you please file a ticket with steps to repro?

if you have multiple instances running without redis cache, then it could be due to a difference in in-memory cache values across instances.