Rate limit per key and model

[olad32]

Hi, is it planned to be able to apply rate limit tpm/rpm on key but with a different rate limit per model ? the same way we can already apply a budget for a key and a specific model as describe here https://litellm.vercel.app/docs/proxy/users#add-model-specific-budgets-to-keys
As of today a key can grant access to multiple model, but the rate limit applied globally to multiple model does not makes a lot of sense in practical use.
Thanks

[krrishdholakia]

hi @olad32 that's interesting

when would you do it this way?

[olad32]

hi, i want to be able to distribute one key to each of my customer projects, each key allows them to access to a predefined list of models (associated with the key : this is already possible), but with a different tpm/rpm for each model as i don't want to globally rate limit them accross multiple model, i need a more fine grained rate limit, per key and model.

[krrishdholakia]

curious - how do you know how much tpm/rpm to assign per model per project?

[olad32]

This is defined by contract with each customer, this is the max tpm/rpm needed for the customer, the tpm/rpm is here to set a limit on llm instance that are shared between multiple customers, and guarantee a fair usage

[krrishdholakia]

So is the customer buying a pack for specific models with specific tpm/rpm limits

[krrishdholakia]

if not - why not just set it globally and let them use across models?

Context - trying to understand your scenario better