[Bug]: Bedrock Cross-Region Inference not working - apac models

[MarcusRosen-Rio]

What happened?

Hi,

I'm trying to configure a LiteLLM proxy container for Bedrock Cross-Region Inference. The solution is running in ap-southeast-2 (Sydney) AWS region. I used the instructions from the docs. AWS Access Credentials are loaded via environmental variables and the other Bedrock Models are working fine (omitted from the config below)

LiteLLM container error is:
LiteLLM Proxy:ERROR: proxy_server.py:3498 - litellm.proxy.proxy_server.chat_completion(): Exception occured - litellm.NotFoundError: BedrockException - Bedrock HTTPX: Unknown provider=apac, model=apac.anthropic.claude-3-5-sonnet-20240620-v1:0

Setup

LiteLLM container version: v1.52.15

Docker File:

# Use the provided base image
FROM ghcr.io/berriai/litellm:main-v1.52.15

# Set the working directory to /app
WORKDIR /app

# Copy the configuration file into the container at /app
COPY config.yaml .

# Make sure your entrypoint.sh is executable
RUN chmod +x docker/entrypoint.sh

# Expose the necessary port
EXPOSE 4000/tcp

# Override the CMD instruction with your desired command and arguments
# WARNING: FOR PROD DO NOT USE `--detailed_debug` it slows down response times, instead use the following CMD
# CMD ["--port", "4000", "--config", "config.yaml"]

CMD ["--port", "4000", "--config", "config.yaml"]

Proxy Config:

model_list:
  
  - model_name: claude-3.5-sonnet-sydney
    litellm_params:
      model: bedrock/apac.anthropic.claude-3-5-sonnet-20240620-v1:0 # inference-profile
      aws_region_name: ap-southeast-2
      guardrailConfig: {
        "guardrailIdentifier": os.environ/BEDROCK_GUARDRAIL_ID_AP_SOUTHEAST_2, # The identifier (ID) for the guardrail.
        "guardrailVersion": os.environ/BEDROCK_GUARDRAIL_VERSION_AP_SOUTHEAST_2, # The version of the guardrail.
        "trace": "enabled", # The trace behavior for the guardrail. Can either be "disabled" or "enabled"
      }

Is the apac region models not supported for cross-region inference?

Relevant log output

litellm-1    | INFO:     172.18.0.5:49112 - "GET /v1/models HTTP/1.1" 200 OK
litellm-1    | 23:07:27 - LiteLLM Proxy:ERROR: proxy_server.py:3498 - litellm.proxy.proxy_server.chat_completion(): Exception occured - litellm.NotFoundError: BedrockException - Bedrock HTTPX: Unknown provider=apac, model=apac.anthropic.claude-3-5-sonnet-20240620-v1:0
litellm-1    | Received Model Group=claude-3.5-sonnet-sydney
litellm-1    | Available Model Group Fallbacks=None
litellm-1    | Traceback (most recent call last):
litellm-1    |   File "/usr/local/lib/python3.11/site-packages/litellm/main.py", line 2622, in completion
litellm-1    |     response = bedrock_chat_completion.completion(
litellm-1    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm-1    |   File "/usr/local/lib/python3.11/site-packages/litellm/llms/bedrock/chat/invoke_handler.py", line 811, in completion
litellm-1    |     raise BedrockError(
litellm-1    | litellm.llms.bedrock.common_utils.BedrockError: Bedrock HTTPX: Unknown provider=apac, model=apac.anthropic.claude-3-5-sonnet-20240620-v1:0
litellm-1    |
litellm-1    | During handling of the above exception, another exception occurred:
litellm-1    |
litellm-1    | Traceback (most recent call last):
litellm-1    |   File "/usr/local/lib/python3.11/site-packages/litellm/proxy/proxy_server.py", line 3387, in chat_completion
litellm-1    |     responses = await llm_responses
litellm-1    |                 ^^^^^^^^^^^^^^^^^^^
litellm-1    |   File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 827, in acompletion
litellm-1    |     raise e
litellm-1    |   File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 803, in acompletion
litellm-1    |     response = await self.async_function_with_fallbacks(**kwargs)
litellm-1    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm-1    |   File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 2805, in async_function_with_fallbacks
litellm-1    |     raise original_exception
litellm-1    |   File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 2641, in async_function_with_fallbacks
litellm-1    |     response = await self.async_function_with_retries(*args, **kwargs)
litellm-1    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm-1    |   File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 2907, in async_function_with_retries
litellm-1    |     self.should_retry_this_error(
litellm-1    |   File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 3055, in should_retry_this_error
litellm-1    |     raise error
litellm-1    |   File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 2889, in async_function_with_retries
litellm-1    |     response = await self.make_call(original_function, *args, **kwargs)
litellm-1    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm-1    |   File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 2986, in make_call
litellm-1    |     response = await response
litellm-1    |                ^^^^^^^^^^^^^^
litellm-1    |   File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 957, in _acompletion
litellm-1    |     raise e
litellm-1    |   File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 925, in _acompletion
litellm-1    |     response = await _response
litellm-1    |                ^^^^^^^^^^^^^^^
litellm-1    |   File "/usr/local/lib/python3.11/site-packages/litellm/utils.py", line 1175, in wrapper_async
litellm-1    |     raise e
litellm-1    |   File "/usr/local/lib/python3.11/site-packages/litellm/utils.py", line 1031, in wrapper_async
litellm-1    |     result = await original_function(*args, **kwargs)
litellm-1    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm-1    |   File "/usr/local/lib/python3.11/site-packages/litellm/main.py", line 503, in acompletion
litellm-1    |     raise exception_type(
litellm-1    |   File "/usr/local/lib/python3.11/site-packages/litellm/main.py", line 473, in acompletion
litellm-1    |     init_response = await loop.run_in_executor(None, func_with_context)
litellm-1    |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm-1    |   File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
litellm-1    |     result = self.fn(*self.args, **self.kwargs)
litellm-1    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm-1    |   File "/usr/local/lib/python3.11/site-packages/litellm/utils.py", line 699, in wrapper
litellm-1    |     result = original_function(*args, **kwargs)
litellm-1    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm-1    |   File "/usr/local/lib/python3.11/site-packages/litellm/main.py", line 3059, in completion
litellm-1    |     raise exception_type(
litellm-1    |           ^^^^^^^^^^^^^^^
litellm-1    |   File "/usr/local/lib/python3.11/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py", line 2136, in exception_type
litellm-1    |     raise e
litellm-1    |   File "/usr/local/lib/python3.11/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py", line 851, in exception_type
litellm-1    |     raise NotFoundError(
litellm-1    | litellm.exceptions.NotFoundError: litellm.NotFoundError: BedrockException - Bedrock HTTPX: Unknown provider=apac, model=apac.anthropic.claude-3-5-sonnet-20240620-v1:0
litellm-1    | Received Model Group=claude-3.5-sonnet-sydney
litellm-1    | Available Model Group Fallbacks=None

Twitter / LinkedIn details

No response