How to use IAM role to access Amazon Bedrock models correctly?

[hongbo-miao]

I am able to follow this tutorial to run Amazon Bedrock -> LiteLLM -> Open WebUI successfully on local using docker compose: https://willdady.com/streamlining-ai-experimentation-with-open-webui-litellm-and-amazon-bedrock

In this case, LiteLLM is using AWS credentials from ~/.aws/credentials.

LiteLLM's GET /models returns

{
  "data": [
    {
      "id": "bedrock-claude-3-5-sonnet",
      "object": "model",
      "created": 1677610602,
      "owned_by": "openai"
    },
    {
      "id": "bedrock-claude-3-haiku",
      "object": "model",
      "created": 1677610602,
      "owned_by": "openai"
    }
  ],
  "object": "list"
}

Now I am trying to deploy LiteLLM to the Amazon EKS and I try to use IAM role to access Amazon Bedrock models.

Here is my IAM Role LiteLLMRole-hm-litellm-service-account

• Permissions

  {
      "Version": "2012-10-17",
      "Statement": [
          {
              "Action": [
                  "bedrock:InvokeModel",
                  "bedrock:InvokeModelWithResponseStream"
              ],
              "Effect": "Allow",
              "Resource": [
                  "arn:aws:bedrock:*::foundation-model/anthropic.claude-3-5-sonnet-20240620-v1:0",
                  "arn:aws:bedrock:*::foundation-model/anthropic.claude-3-haiku-20240307-v1:0"
              ]
          }
      ]
  }

• Trust relationships

  {
      "Version": "2012-10-17",
      "Statement": [
          {
              "Effect": "Allow",
              "Principal": {
                  "Federated": "arn:aws:iam::xxxxxxxxxxxx:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/70D9E464CED49A4937EAAAF959727767"
              },
              "Action": "sts:AssumeRoleWithWebIdentity",
              "Condition": {
                  "StringEquals": {
                      "oidc.eks.us-west-2.amazonaws.com/id/70D9E464CED49A4937EAAAF959727767:sub": "system:serviceaccount:production-hm-litellm:hm-litellm-service-account",
                      "oidc.eks.us-west-2.amazonaws.com/id/70D9E464CED49A4937EAAAF959727767:aud": "sts.amazonaws.com"
                  }
              }
          }
      ]
  }

Here is my Kubernetes manifest files:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: hm-litellm-service-account
  namespace: production-hm-litellm
  annotations:
    # https://docs.aws.amazon.com/eks/latest/userguide/associate-service-account-role.html
    eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxxxx:role/LiteLLMRole-hm-litellm-service-account
  labels:
    app.kubernetes.io/name: hm-litellm-service-account
    app.kubernetes.io/part-of: production-hm-litellm
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: hm-litellm-config-map
  namespace: production-hm-litellm
  labels:
    app.kubernetes.io/name: hm-litellm-config-map
    app.kubernetes.io/part-of: production-hm-litellm
data:
  proxy_server_config.yaml: |
    # https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids.html
    model_list:
      - model_name: bedrock-claude-3-5-sonnet
        litellm_params:
          model: bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0
        aws_region_name: us-west-2
        aws_session_name: litellm
        aws_role_name: arn:aws:iam::xxxxxxxxxxxx:role/LiteLLMRole-hm-litellm-service-account
      - model_name: bedrock-claude-3-haiku
        litellm_params:
          model: bedrock/anthropic.claude-3-haiku-20240307-v1:0
        aws_region_name: us-west-2
        aws_session_name: litellm
        aws_role_name: arn:aws:iam::xxxxxxxxxxxx:role/LiteLLMRole-hm-litellm-service-account
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hm-litellm-deployment
  namespace: production-hm-litellm
  labels:
    app.kubernetes.io/name: hm-litellm-deployment
    app.kubernetes.io/part-of: production-hm-litellm
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hm-litellm
  template:
    metadata:
      labels:
        app: hm-litellm
    spec:
      serviceAccountName: hm-litellm-service-account # I am not sure if here really needs, as `model_list` in config map also set each IAM role for each model. So maybe no need.
      containers:
        - name: litellm
          image: ghcr.io/berriai/litellm:main-v1.46.8
          ports:
            - name: litellm
              protocol: TCP
              containerPort: 4000
          volumeMounts:
            - name: litellm-volume
              mountPath: /app/proxy_server_config.yaml
              subPath: proxy_server_config.yaml
      volumes:
        - name: litellm-volume
          configMap:
            name: hm-litellm-config-map
---
apiVersion: v1
kind: Service
metadata:
  name: hm-litellm-service
  namespace: production-hm-litellm
  labels:
    app.kubernetes.io/name: hm-litellm-service
    app.kubernetes.io/part-of: production-hm-litellm
spec:
  type: ClusterIP
  selector:
    app: hm-litellm
  ports:
    - name: litellm
      protocol: TCP
      targetPort: litellm
      port: 80

Inside, xxxxxxxxxxxx is my AWS account ID.

However, right now LiteLLM GET /models returns empty model list

{
    "data": [],
    "object": "list"
}

Any guide would be appreciate. Thanks!

[krrishdholakia]

what do your server logs show?

[hongbo-miao]

Thanks @krrishdholakia for following up! Unfortunately, there was no error in the LiteLLM log at that time.

But I found the solution and posted at #5873 (comment)

[hongbo-miao]

I think somehow LiteLLM does not read /app/proxy_server_config.yaml by default.

I put a new location at /app/config.yaml and added

command: ["litellm", "--port", "4000", "--config", "/app/config.yaml", "--detailed_debug"]

in deployment, now it works!

Here are updated version:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: hm-litellm-config-map
  namespace: production-hm-litellm
  labels:
    app.kubernetes.io/name: hm-litellm-config-map
    app.kubernetes.io/part-of: production-hm-litellm
data:
  config.yaml: |  # Updated
    # https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids.html
    model_list:
      - model_name: bedrock-claude-3-5-sonnet
        litellm_params:
          model: bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0
        aws_region_name: us-west-2
        aws_session_name: litellm
        aws_role_name: arn:aws:iam::xxxxxxxxxxxx:role/LiteLLMRole-hm-litellm-service-account
      - model_name: bedrock-claude-3-haiku
        litellm_params:
          model: bedrock/anthropic.claude-3-haiku-20240307-v1:0
        aws_region_name: us-west-2
        aws_session_name: litellm
        aws_role_name: arn:aws:iam::xxxxxxxxxxxx:role/LiteLLMRole-hm-litellm-service-account
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hm-litellm-deployment
  namespace: production-hm-litellm
  labels:
    app.kubernetes.io/name: hm-litellm-deployment
    app.kubernetes.io/part-of: production-hm-litellm
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hm-litellm
  template:
    metadata:
      labels:
        app: hm-litellm
    spec:
      serviceAccountName: hm-litellm-service-account # I am not sure if here really needs, as `model_list` in config map also set each IAM role for each model. So maybe no need.
      containers:
        - name: litellm
          image: ghcr.io/berriai/litellm:main-v1.46.8
          ports:
            - name: litellm
              protocol: TCP
              containerPort: 4000
          command: ["litellm", "--port", "4000", "--config", "/app/config.yaml", "--detailed_debug"] # New added
          volumeMounts:
            - name: litellm-volume
              mountPath: /app/config.yaml # Updated
              subPath: config.yaml # Updated
      volumes:
        - name: litellm-volume
          configMap:
            name: hm-litellm-config-map