Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug: ai-proxy does not correctly escape AWS Bedrock inference profile ARN in signature and request #14309

Open
1 task done
muscionig opened this issue Feb 25, 2025 · 3 comments · May be fixed by #14310
Open
1 task done

Comments

@muscionig
Copy link

muscionig commented Feb 25, 2025

Is there an existing issue for this?

  • I have searched the existing issues

Kong version ($ kong version)

Kong 3.9.0

Current Behavior

When using AWS Bedrock inference profiles, which are formatted as: arn:aws:bedrock:us-east-1:<account_id>:application-inference-profile/<profile_id>, kong is unable to route the request.
I have experience two behaviors:

  1. if the ai-proxy is configured with the unescaped ARN, Kong fails with:
{"error":{"message":"transformation failed from type bedrock://llm/v1/chat: no generation candidates received from Bedrock, or max_tokens too short"}}
  1. when configured with the escaped one (arn%3Aaws%3Abedrock%3Aus-east-1%3A<account_id>%3Aapplication-inference-profile%2F<profile_id>), Kong fails with:
{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method."}

Expected Behavior

Kong should correctly format and escape the ARN when generating the SigV4 signature and constructing the request URL to match AWS’s expected format.

  • The Converse API requires the URL to be formatted as:
    /model/<model-id/inference-profile>/converse
    
  • The AWS CLI (in debug mode) shows that this URL is then encoded when making the request to Bedrock.

Steps To Reproduce

  1. Create the plugin with the unescaped model_name ARN

    • Example configuration (following the kong documentation):
      curl -X POST http://localhost:8001/services/bedrock-service/plugins \
       --header 'Content-Type: application/json' \
       --header 'accept: application/json' \
       --data '{
       "name": "ai-proxy",
       "instance_name": "ai-proxy-bedrock",
       "config": {
         "route_type": "llm/v1/chat",
         "auth": {
           "allow_override": false,
           "aws_access_key_id": "<YOUR_AWS_ACCESS_KEY_ID>",
           "aws_secret_access_key": "<YOUR_AWS_SECRET_ACCESS_KEY>"
         },
         "model": {
           "provider": "bedrock",
           "name": "arn:aws:bedrock:us-east-1:<account_id>:application-inference-profile/<profile_id>",
           "options": {
             "bedrock": {
               "aws_region": "us-east-1"
             }
           }
         }
       }
       }'
      
      This will fail with the transformation error reported above.
  2. Create the plugin with the escaped model_name ARN

    • Example configuration (following the kong documentation):
      curl -X POST http://localhost:8001/services/bedrock-service/plugins \
       --header 'Content-Type: application/json' \
       ...
           "provider": "bedrock",
           "name": "arn%3Aaws%3Abedrock%3Aus-east-1%3A<account_id>%3Aapplication-inference-profile%2F<profile_id>",
       ...
       }'
      
      This will fail with the sigv4 error.

Anything else?

  • I did check the aws cli with the inference profile to rule out authentication errors with the inference profile, and the profile works using the converse endpoint.
  • I verified the URL behavior by enabling the File Log plugin on /dev/stdout.
@muscionig muscionig linked a pull request Feb 25, 2025 that will close this issue
3 tasks
@muscionig
Copy link
Author

I have attempted a fix in PR #14310.

While going through the source code I noticed that the plugin allows for setting and overriding the upstream_path. I did try that, but the overridden path is then used for the sigv4 authentication.

Another possible way would be to allow the inference profile by recommending the model.options.upstream_path edits, and by adding ngx.unescape_uri(parsed_url.path) in the authentication process.

@muscionig muscionig changed the title Bug: AI-Proxy Does Not Correctly Escape AWS Bedrock Inference Profile ARN in Signature Bug: ai-proxy does not correctly escape AWS Bedrock inference profile ARN in signature and request Feb 25, 2025
@tysoekong
Copy link
Contributor

tysoekong commented Feb 25, 2025

Hey @muscionig thanks so much for this, I actually hadn't seen that you can just specify the model ARN directly for Bedrock-converse, I thought it was just for InvokeModel, so I didn't check it. But yeah I had this same problem with "upstream_path" which is:

  1. fixed but
  2. getting removed soon, the field is not needed with the next release

I will either ask if your fix can be merged directly, or I will bundle it into the next 3.10 main and tag you as contributor (because we have a giant PR of fixes coming already, it's quicker).

@muscionig
Copy link
Author

Hi @tysoekong, thanks for the update and for considering my changes! I’m totally fine with the fix being integrated into a different PR and with the collaborator approach.

Is the PR public? I’d love to test it in the meantime to ensure everything works as expected.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants