[Question]: How to use Bedrock Claude 3.5 Sonnet with LlamaIndex

[Breamer]

Question Validation

• I have searched both the documentation and discord for an answer.

Question

Hi, I'm using Bedrock Claude 3 Sonnet to create document embeddings, running a python 3.11.7 code on an AWS lambda function using this code:

Bedrock(
model="anthropic.claude-3-sonnet-20240229-v1:0",
aws_access_key_id=bedrock_access_key_id,
aws_secret_access_key=bedrock_secret_access_key,
temperature=0,
region_name=bedrock_region
)

This is working fine with Claude 3 Sonnet, however, if I change the model to try using the new Claude 3.5 Sonnet (model="anthropic.claude-3-5-sonnet-20240620-v1:0"), it throws the following error.

llama_index\llms\bedrock\base.py
ValueError: context_size argument not provided andmodel provided refers to a non-foundation model. Please specify the context_size

What does "non-foundation model" means?
What value should be set for this parameter?
I tried to set 512, and it generates error afterwards, when trying to generate prompts for those embedded documents.

"ValueError: Calculated available context size -586 was not non-negative."

I'm using the following package versions:
llama_index==0.10.38
llama-index-embeddings-bedrock==0.1.5
llama-index-llms-bedrock==0.1.8

[dosubot]

@Breamer Hello there! I'm here to help you with any bugs, questions, or contributions. Let's tackle this issue together.

To resolve the context_size error when switching from Claude 3 Sonnet to Claude 3.5 Sonnet in your AWS Lambda function using the Bedrock model, you need to ensure that the context size for the model is correctly set. Both anthropic.claude-3-sonnet-20240229-v1:0 and anthropic.claude-3-5-sonnet-20240620-v1:0 have a context size of 200,000.

Here is the relevant code snippet to get the context size for the model:

def bedrock_modelname_to_context_size(modelname: str) -> int:
    if modelname not in BEDROCK_MODELS:
        raise ValueError(
            f"Unknown model: {modelname}. Please provide a valid Bedrock model name. "
            "Known models are: " + ", ".join(BEDROCK_MODELS.keys())
        )

    return BEDROCK_MODELS[modelname]

You can use this function to set the context size for anthropic.claude-3-5-sonnet-20240620-v1:0:

model_name = "anthropic.claude-3-5-sonnet-20240620-v1:0"
context_size = bedrock_modelname_to_context_size(model_name)

Ensure that your AWS Lambda function is using this context size when configuring the model. Additionally, when initializing the Bedrock class, make sure to pass the context_size parameter correctly:

from llama_index.llms.bedrock import Bedrock

bedrock_model = Bedrock(
    model=model_name,
    aws_access_key_id=bedrock_access_key_id,
    aws_secret_access_key=bedrock_secret_access_key,
    temperature=0,
    region_name=bedrock_region,
    context_size=context_size,
)

This will ensure that the context_size is properly set and used in your AWS Lambda function. The Bedrock class in the LlamaIndex framework has a context_size parameter that you need to set appropriately to avoid the error ^[1][2][3].

To continue talking to Dosu, mention @dosu.

[logan-markewich]

This error is not coming from our code. Its coming from bedrock

As for what It means exactly, no idea. My best guess, you should provide the full context window size (200000 I believe?)

[logan-markewich]

Bedrock(...., context_window=200000)

[Breamer]

After adding the context size parameter with a 200000 value, everything started to work, but still I don't see the same code as the bot suggested: there is no bedrock_modelname_to_context_size function nor a BEDROCK_MODELS property. Are those suggestions pointing at old code or newer versions?