v1.27.8

What's Changed

• fix(utils.py): support returning caching streaming response for function calling streaming calls by @krrishdholakia in #2203
• build(proxy_server.py): fix /spend/logs query bug by @krrishdholakia in #2212

Full Changelog: v1.27.7...v1.27.8

v1.27.7

Use ClickhouseDB for low latency LLM Analytics / Spend Reports

(sub 1s analytics, with 100M logs)

Getting started with ClickHouse DB + LiteLLM Proxy

Docs + Docker compose for getting started with clickhouse: https://docs.litellm.ai/docs/proxy/logging#logging-proxy-inputoutput---clickhouse

Step 1: Create a config.yaml file and set litellm_settings: success_callback

model_list:
 - model_name: gpt-3.5-turbo
    litellm_params:
      model: gpt-3.5-turbo
litellm_settings:
  success_callback: ["clickhouse"]

Step 2: Set Required env variables for clickhouse

Env Variables for self hosted click house

CLICKHOUSE_HOST = "localhost"
CLICKHOUSE_PORT = "8123"
CLICKHOUSE_USERNAME = "admin"
CLICKHOUSE_PASSWORD = "admin"

Step 3: Start the proxy, make a test request

New Models

Mistral on Azure AI Studio

Sample Usage

Ensure you have the /v1 in your api_base

from litellm import completion
import os

response = completion(
    model="mistral/Mistral-large-dfgfj", 
    api_base="https://Mistral-large-dfgfj-serverless.eastus2.inference.ai.azure.com/v1",
    api_key = "JGbKodRcTp****"
    messages=[
       {"role": "user", "content": "hello from litellm"}
   ],
)
print(response)

[LiteLLM Proxy] Using Mistral Models

Set this on your litellm proxy config.yaml

Ensure you have the /v1 in your api_base

model_list:
  - model_name: mistral
    litellm_params:
      model: mistral/Mistral-large-dfgfj
      api_base: https://Mistral-large-dfgfj-serverless.eastus2.inference.ai.azure.com/v1
      api_key: JGbKodRcTp****

What's Changed

• [Docs] use azure ai studio + mistral large by @ishaan-jaff in #2205
• [Feat] Start Self hosted clickhouse server by @ishaan-jaff in #2206
• [FEAT] Admin UI - View /spend/logs from clickhouse data by @ishaan-jaff in #2210
• [Docs] Use Clickhouse DB + Docker compose by @ishaan-jaff in #2211

Full Changelog: v1.27.6...v1.27.7

v1.27.6

New Models

• azure/text-embedding-3-large
• azure/text-embedding-3-small
• mistral/mistral-large-latest

Log LLM Output in ClickHouse DB

litellm.success_callback = ["clickhouse"]
await litellm.acompletion(
    model="gpt-3.5-turbo",
    messages=[{"role": "user", "content": f"This is a test"}],
    max_tokens=10,
    temperature=0.7,
    user="ishaan-2",
)

What's Changed

• [FEAT] add cost for azure/text-embedding-3-large, azure/text-embedding-3-small by @ishaan-jaff in #2198
• [FEAT] Use Logging on clickhouse by @ishaan-jaff in #2187
• Litellm custom callback fix by @krrishdholakia in #2202

Full Changelog: v1.27.4...v1.27.6

v1.27.4

What's Changed

• Allow end-users to opt out of llm api calls by @krrishdholakia in #2174
• [Docs] open router - clarify we support all models by @ishaan-jaff in #2186
• (docs) using openai compatible endpoints by @ishaan-jaff in #2189
• [Fix] Fix health check when API base set for OpenAI compatible models by @ishaan-jaff in #2188
• fix(proxy_server.py): allow user to set team tpm/rpm limits/budget/models by @krrishdholakia in #2183

Full Changelog: v1.27.1...v1.27.4

v1.27.1

What's Changed

• Default user values for new user on SSO by @krrishdholakia in #2172
• fix(langfuse.py): support time to first token logging on langfuse by @krrishdholakia in #2165
• fix(utils.py): stricter azure function calling tests by @krrishdholakia in #2175

Full Changelog: v1.26.13...v1.27.1

v1.26.13

What's Changed

• [FIX] BUG where extra tokens created in litellm verification token table by @ishaan-jaff in #2150
• Support for Athina logging by @vivek-athina in #2163
• [FEAT] Support extra headers - OpenAI / Azure by @ishaan-jaff in #2164
• [FEAT] Support Groq AI by @ishaan-jaff in #2168

Sample Usage

from litellm import completion
import os

os.environ['GROQ_API_KEY'] = ""
response = completion(
    model="groq/llama2-70b-4096", 
    messages=[
       {"role": "user", "content": "hello from litellm"}
   ],
)
print(response)

New Contributors

• @vivek-athina made their first contribution in #2163

Full Changelog: v1.26.10...v1.26.13

v1.26.11

What's Changed

• Allow admin to ban keywords on proxy by @krrishdholakia in #2147
• fix(utils.py): ensure argument is always a string by @krrishdholakia in #2141

Full Changelog: v1.26.9...v1.26.11

v1.26.10

What's Changed

• Allow admin to ban keywords on proxy by @krrishdholakia in #2147
• fix(utils.py): ensure argument is always a string by @krrishdholakia in #2141

Full Changelog: v1.26.9...v1.26.10

v1.26.9

What's Changed

• Adding PromptLayer support of tags by @samyxdev in #1491
• update generic SSO login by @adrien-f in #2129
• [FEAT] LiteLLM Admin UI work with keycloak by @ishaan-jaff in #2140
• [FEAT] Admin UI - Use Hosted Logo for Custom Branding by @ishaan-jaff in #2143
• [FEAT] Set TPM/RPM Limits per User by @ishaan-jaff in #2148
• feat(proxy_server.py): add support for blocked user lists (enterprise-only) by @krrishdholakia in #2144
  

New Contributors

• @samyxdev made their first contribution in #1491

Full Changelog: v1.26.8...v1.26.9

v1.26.8

Enterprise - Admin UI - Use Custom Branding

Use your companies custom branding on the LiteLLM Admin UI
Docs here: https://docs.litellm.ai/docs/proxy/ui#custom-branding-admin-ui
We allow you to

• Customize the UI Logo
• Customize the UI color theme

What's Changed

• Enable /team/update, /team/delete endpoints + create teams with user defined roles by @krrishdholakia in #2119
• Admin UI - Use Custom Branding by @ishaan-jaff in #2123
• fix(utils.py): support streaming cached response logging by @krrishdholakia in #2124
• fix(router.py): mask the api key in debug statements on router by @krrishdholakia in #2125

Full Changelog: v1.26.7...v1.26.8