Multiple LiteLLM instances

[olad32]

Hi, is this possible to run multiple instances in parallel to be able to scale horizontaly ? Even with the database features in use ?
Is there anything to know to be able to rollout a new LiteLLM config on multiple instances without downtime ? Eg update model config
Thanks

[krrishdholakia]

Hey @olad32 you should be able to do this. I believe you'd need to change the db connection pool limit for a single instance, to do this well (we currently set it to 100, which could be the max for some systems).

We have a db table for the LLM config that we were planning on using for this.

A problem I was trying to figure out was:

• models need keys. Should the table store the keys?

[krrishdholakia]

If you have time today, would love to do a quick call and talk through this:

https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

[krrishdholakia]

@olad32 just pushed a fix to let you control db connection pool + timeouts for better scalability - https://docs.litellm.ai/docs/proxy/configs#configure-db-pool-limits--connection-timeouts

Should be out in the next release.

Would love to do a quick call and talk through the reload config file issue - https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Let me know if any time this / next week works!

[olad32]

Thanks for the configurable pool options.
Regarding the config reload topic, the static config.yaml is the simplest and most performant option for sure, but it would add flexibility to have an option in litellm proxy to be able to update config at runtime. Persisting the config in a database is one solution, litellm proxy could periodically check for updated config in the database, this would not alter performance too much.
As a side note I just saw that the proxy API has a new model endpoint, if this only update the static config in memory, it will not work in multi litellm proxy instance environnement behind a Load Balancer.

For now another option exists via the rolling update kuberntes strategy which can recreate each pod (ie litellm proxy instance) one by one with the new static config.yaml, but it generates a bit of noise on the cluster (recreate each pod), not ideal but manageable. One thing is mandatory for this option to work, litellm proxy must handle gracefull shutdown, ie handle sigterm signal sent by kubernetes and wait for current requests to end before effectively shutting down, especially usefull for long streaming response. In fact gracefulll shutdown is always a good thing to handle.

[olad32]

(i can't find a schedule to make a call as we are obviously at the earth opposite ;) )

[ishaan-jaff]

@olad32 we're on this call if you're free right now: https://meet.google.com/zav-wexx-tbe

[krrishdholakia]

Hi @olad32 i guess the question was:

• if you do /config/update and add a new model to the config, would that be stored in the db?

Curious - what're you trying to update about litellm proxy? (is it just adding new models?)

[olad32]

Hi @krrishdholakia, exactly, i want to be able to update existing models parameters (like rpm per model) and add new models, and the new configuration must be applied by all litellm proxy instances that runs in parrallel. This is needed to be able to auto scale LiteLLM itself and LLMs instances upstream (add a new llm instance).

[ishaan-jaff]

Hi @olad32 just wanted to follow up

1. If we add the ability to add new models through the Admin UI, would you be able to try it and give us feedback ?
2. I'd love to set up a direct 1:1 support channel for you, feel free to pick the channel that works best for you

• Linkedin https://www.linkedin.com/in/reffajnaahsi/
• Twitter: https://twitter.com/ishaan_jaff
• Discord: https://discord.com/invite/wuPM9dRgDw
• Other - please let me know if you'd prefer another channel