v1.52.16.dev4

What's Changed

• LiteLLM Minor Fixes & Improvements (11/24/2024) by @krrishdholakia in #6890
• (feat) pass through llm endpoints - add PATCH support (vertex context caching requires for update ops) by @ishaan-jaff in #6924
• sonnet supports pdf, haiku does not by @paul-gauthier in #6928
• (feat) DataDog Logger - Add Failure logging + use Standard Logging payload by @ishaan-jaff in #6929
• (feat) log proxy auth errors on datadog by @ishaan-jaff in #6931
• (feat) Allow using include to include external YAML files in a config.yaml by @ishaan-jaff in #6922

Full Changelog: v1.52.16...v1.52.16.dev4

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.52.16.dev4

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	200.0	223.2904107688168	6.215640236973433	0.0	1860	0	174.75808199998255	3944.991313999992
Aggregated	Passed ✅	200.0	223.2904107688168	6.215640236973433	0.0	1860	0	174.75808199998255	3944.991313999992

v1.52.16.dev1

What's Changed

• LiteLLM Minor Fixes & Improvements (11/24/2024) by @krrishdholakia in #6890
• (feat) pass through llm endpoints - add PATCH support (vertex context caching requires for update ops) by @ishaan-jaff in #6924
• sonnet supports pdf, haiku does not by @paul-gauthier in #6928
• (feat) DataDog Logger - Add Failure logging + use Standard Logging payload by @ishaan-jaff in #6929
• (feat) log proxy auth errors on datadog by @ishaan-jaff in #6931
• (feat) Allow using include to include external YAML files in a config.yaml by @ishaan-jaff in #6922
• (feat) dd logger - set tags according to the values set by those env vars by @ishaan-jaff in #6933

Full Changelog: v1.52.16...v1.52.16.dev1

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.52.16.dev1

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.52.16.dev1

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	250.0	285.0974372649336	6.039486955708498	0.0	1808	0	224.19419400000606	3263.23956899995
Aggregated	Passed ✅	250.0	285.0974372649336	6.039486955708498	0.0	1808	0	224.19419400000606	3263.23956899995

v1.52.16

What's Changed

• feat - allow sending tags on vertex pass through requests by @ishaan-jaff in #6876
• (feat) Add support for using @google/generative-ai JS with LiteLLM Proxy by @ishaan-jaff in #6899
• (UI fix) UI does not reload when you login / open a new tab by @ishaan-jaff in #6909
• (fix) pass through endpoints - run logging async + use thread pool executor for sync logging callbacks by @ishaan-jaff in #6907
• (redis fix) - fix AbstractConnection.__init__() got an unexpected keyword argument 'ssl' by @ishaan-jaff in #6908
• (docs) Simplify /vertex_ai/ pass through docs by @ishaan-jaff in #6910

Full Changelog: v1.52.15...v1.52.16

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.52.16

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	230.0	261.12838541230485	6.194778256251999	0.0	1853	0	206.21302299997524	2167.8605710000056
Aggregated	Passed ✅	230.0	261.12838541230485	6.194778256251999	0.0	1853	0	206.21302299997524	2167.8605710000056

v1.52.15

What's Changed

• (feat) use @google-cloud/vertexai js sdk with litellm by @ishaan-jaff in #6873
• (chore) fix new .js tests running for vertex.js by @ishaan-jaff in #6875
• Bump cross-spawn from 7.0.3 to 7.0.6 in /ui/litellm-dashboard by @dependabot in #6865
• (Perf / latency improvement) improve pass through endpoint latency to ~50ms (before PR was 400ms) by @ishaan-jaff in #6874
• LiteLLM Minor Fixes & Improvements (11/23/2024) by @krrishdholakia in #6870
• Litellm dev 11 23 2024 by @krrishdholakia in #6881
• docs - have 1 section for routing +load balancing by @ishaan-jaff in #6884
• (QOL improvement) Provider budget routing - allow using 1s, 1d, 1mo, 2mo etc by @ishaan-jaff in #6885
• (feat) - provider budget improvements - ensure provider budgets work with multiple proxy instances + improve latency to ~90ms by @ishaan-jaff in #6886

Full Changelog: v1.52.14...v1.52.15

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.52.15

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Failed ❌	280.0	454.59782761891177	5.830264051408934	0.010023376592680114	1745	3	139.27931299997454	5766.263976999994
Aggregated	Failed ❌	280.0	454.59782761891177	5.830264051408934	0.010023376592680114	1745	3	139.27931299997454	5766.263976999994

v1.52.14

What's Changed

• (fix) passthrough - allow internal users to access /anthropic by @ishaan-jaff in #6843
• LiteLLM Minor Fixes & Improvements (11/21/2024) by @krrishdholakia in #6837
• fix latency issues on google ai studio by @ishaan-jaff in #6852
• (fix) add linting check to ban creating AsyncHTTPHandler during LLM calling by @ishaan-jaff in #6855
• (feat) Add usage tracking for streaming /anthropic passthrough routes by @ishaan-jaff in #6842
• (Feat) Allow passing litellm_metadata to pass through endpoints + Add e2e tests for /anthropic/ usage tracking by @ishaan-jaff in #6864

Full Changelog: v1.52.12...v1.52.14

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.52.14

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	260.0	292.32742033908687	6.002121672811824	0.0	1796	0	222.04342999998516	2700.951708000048
Aggregated	Passed ✅	260.0	292.32742033908687	6.002121672811824	0.0	1796	0	222.04342999998516	2700.951708000048

v1.52.10.staging.2

Full Changelog: v1.52.10.staging.1...v1.52.10.staging.2

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.52.10.staging.2

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	230.0	260.85031392210647	6.091362356611515	0.0	1823	0	196.95026900001267	3095.300408000014
Aggregated	Passed ✅	230.0	260.85031392210647	6.091362356611515	0.0	1823	0	196.95026900001267	3095.300408000014

v1.52.10.staging.1

Full Changelog: v1.52.10...v1.52.10.staging.1

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.52.10.staging.1

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	250.0	276.3844152464711	6.156861360191656	0.0	1842	0	213.47366499998088	2957.452922000016
Aggregated	Passed ✅	250.0	276.3844152464711	6.156861360191656	0.0	1842	0	213.47366499998088	2957.452922000016

v1.52.10-stable

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:litellm_stable_nov_21-stable

Full Changelog: v1.52.10.staging.2...v1.52.10-stable

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.52.10-stable

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	190.0	210.34423537712797	6.281907830899626	0.0	1880	0	174.39816099999916	1348.7341720000074
Aggregated	Passed ✅	190.0	210.34423537712797	6.281907830899626	0.0	1880	0	174.39816099999916	1348.7341720000074

v1.52.12

What's Changed

• LiteLLM Minor Fixes & Improvements (11/19/2024) by @krrishdholakia in #6820
• Add gpt-4o-2024-11-20 by @Manouchehri in #6832
• LiteLLM Minor Fixes & Improvements (11/20/2024) by @krrishdholakia in #6831
• Litellm dev 11 20 2024 by @krrishdholakia in #6838
• (refactor) anthropic - move _process_response in transformation.py by @ishaan-jaff in #6834
• (feat) add usage / cost tracking for Anthropic passthrough routes by @ishaan-jaff in #6835
• (testing) - add e2e tests for anthropic pass through endpoints by @ishaan-jaff in #6840
• (fix) don't block proxy startup if license check fails & using prometheus by @ishaan-jaff in #6839

Full Changelog: v1.52.11...v1.52.12

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.52.12

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	260.0	288.3101155320099	6.056613494123171	0.0	1812	0	231.241644000022	2338.7360799999897
Aggregated	Passed ✅	260.0	288.3101155320099	6.056613494123171	0.0	1812	0	231.241644000022	2338.7360799999897

v1.52.11

What's Changed

• (docs improvement) remove emojis, use guides section, categorize uncategorized docs by @ishaan-jaff in #6796
• (docs) simplify left nav names + use a section for making llm requests by @ishaan-jaff in #6799
• Bump cross-spawn from 7.0.3 to 7.0.5 in /ui by @dependabot in #6779
• Docs - use 1 page for all logging integrations on proxy + add logging features at top level by @ishaan-jaff in #6805
• (docs) add docstrings for all /key, /user, /team, /customer endpoints by @ishaan-jaff in #6804
• LiteLLM Minor Fixes & Improvements (11/15/2024) by @krrishdholakia in #6746
• (Proxy) add support for DOCS_URL and REDOC_URL by @ishaan-jaff in #6806
• feat - add fireworks_ai/qwen2p5-coder-32b-instruct by @ishaan-jaff in #6818
• Litellm stable pr 10 30 2024 by @krrishdholakia in #6821
• (Feat) Add provider specific budget routing by @ishaan-jaff in #6817
• (feat) provider budget routing improvements by @ishaan-jaff in #6827

Full Changelog: v1.52.10...v1.52.11

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.52.11

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Failed ❌	270.0	300.82403009385007	6.198177352347725	0.0	1854	0	229.45128300000306	3106.586268000001
Aggregated	Failed ❌	270.0	300.82403009385007	6.198177352347725	0.0	1854	0	229.45128300000306	3106.586268000001