More fine grain governance for budget allocation

[scampion]

Manage budget per user and model group make sense if you want to manage proprietary model and self hosted model.

Alice:

• Proprietary Models: 10 $ / day
• Open Source Models self hosted: free

Bob:

• OpenAI Azure GPT3.5 : 10 000 tokens / day
• Mistral 7B Orca: unlimited

Etc ...

Another way to manage that it's per virtual key but we must be able to restrict them to certain model

[ishaan-jaff]

@scampion thanks for the discussion

• we allow you to set budgets and models allow per key https://docs.litellm.ai/docs/proxy/virtual_keys#request
  Is this what you were looking for ?

curl 'http://0.0.0.0:8000/key/generate' \
--header 'Authorization: Bearer <your-master-key>' \
--header 'Content-Type: application/json' \
--data-raw '{
  "models": ["gpt-3.5-turbo", "gpt-4", "claude-2"],
  "duration": "20m",
  "metadata": {"user": "[email protected]"},
  "team_id": "core-infra",
  "max_budget": 10,
}'

[scampion]

Exactly, I am developing a simple app right now for that.

I will integrate that with Traefik, the oauth2-proxy image to let end user self-manage it.

I need to think about how to restrict via ACL the access to certain model for certain team also but I think that it could be managed by the app

[ishaan-jaff]

@scampion we have a UI for access control - users need to SSO sign in and then they can generate keys https://docs.litellm.ai/docs/proxy/ui