The Feature

Support retry policies when calling completion() / text_completion() without requiring Router. Example:

import litellm
from litellm import RetryPolicy

retry_policy = RetryPolicy(
    TimeoutErrorRetries=num_retries,
    RateLimitErrorRetries=num_retries,
    InternalServerErrorRetries=num_retries,
    # We don't retry on errors that are unlikely to be transient
    # (e.g. bad request, invalid auth credentials)
    BadRequestErrorRetries=0,
    AuthenticationErrorRetries=0,
    ContentPolicyViolationErrorRetries=0,
)

litellm.completion(
    model="openai/gpt-4o-mini",
    messages=[{"role": "user", "content": "Example content"}],
    retry_strategy="exponential_backoff_retry",
    retry_policy=retry_policy,
)

Motivation, pitch

The DSPy library (https://github.com/stanfordnlp/dspy) depends on LiteLLM for issuing LLM calls. When these calls fail due to transient network errors or rate limiting, we want to retry with exponential backoff. However, when these calls fail due to user error (e.g. bad API keys, malformed requests), we want to fail fast.

DSPy users configure LLM keys and parameters using constructor arguments to the dspy.LM class (and optionally be setting environment variables like `OPENAI_API_KEY'), for example:

llm = dspy.LM(model="openai/gpt-4o-mini", api_key="<my key>", model_type="chat")
llm("Who invented deep learing?")

# Env var alternative
os.environ["OPENAI_API_KEY"] = "<my_key>"
llm = dspy.LM(model="openai/gpt-4o-mini", model_type="chat")
llm("Who invented deep learnng?")

DSPy currently wraps litellm.completion() and litellm.text_completion() to implement this interface. See https://github.com/stanfordnlp/dspy/blob/8bc3439052eb80ba4e5ba340c348a6e3b2c94d7c/dspy/clients/lm.py#L78-L87 / https://github.com/stanfordnlp/dspy/blob/8bc3439052eb80ba4e5ba340c348a6e3b2c94d7c/dspy/clients/lm.py#L166-L216. Currently, these interfaces don't support specifying a retry policy.

We've attempted to work around this by constructing a Router internally, but Router construction requires us to fetch the api key and base and pass them to a model_list (due to OpenAI / Azure OpenAI initialization -

Twitter / LinkedIn details

No response