Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: set up gen ai inference attributes for foundational text models in java v1 sdk #19

Closed
wants to merge 2 commits into from

Conversation

yiyuan-he
Copy link

@yiyuan-he yiyuan-he commented Nov 5, 2024

Description of changes:
Adding auto-instrumentation support for GenAI inference parameters for Java V1 AWS SDK.

The following foundational text models are supported:

  • AI21 Jamba
  • Amazon Titan
  • Anthropic Claude
  • Cohere Command R
  • Meta Llama
  • Mistral AI

Full list can be found here.

Note: Removed support for old Cohere Command models since they already throw a 404 response for Java V1.

Note: There is a backwards compatibility issue with the existing tests as they rely on older versions of the AWS SDK. Since Bedrock is newer resource we are forced to introduce a newer SDK version which implicitly adds in behavior that is not backwards compatible with the existing tests. It seems these dependency issues in the tests are a known issue in upstream which is why they rely on such old AWS resource versions in the first place. Synced with @mxiamxia and we decided to temporarily disable these tests for now as we figure out a long-term solution. We rely on the contract tests to verify the SDK quality for the time being.

New inference parameter attributes added according to OpenTelemetry Semantic Conventions for GenAI attributes:

  • gen_ai.request.max_tokens
  • gen_ai.request.temperature
  • gen_ai.request.top_p
  • gen_ai.response.finish_reasons
  • gen_ai.usage.input_tokens
  • gen_ai.usage.output_tokens

Test Plan:
Set up sample app to make Bedrock Runtime InvokeModel API calls to the supported foundational models and verified the auto-instrumentation attributes.

AI21 Jamba
ai21-jamba

Amazon Titan
amazon-titan

Anthropic Claude
anthropic-claude

Cohere Command
cohere-command-r

Meta Llama
meta-llama

Mistral AI
mistral-ai

@yiyuan-he yiyuan-he force-pushed the gen-ai-support-v1 branch 2 times, most recently from a49de66 to 660191e Compare November 6, 2024 22:41
@yiyuan-he yiyuan-he force-pushed the gen-ai-support-v1 branch 4 times, most recently from e8eaec9 to b428c01 Compare November 8, 2024 03:27
@yiyuan-he yiyuan-he changed the title feat: set up gen ai inference attributes for foundational text models [not ready] feat: set up gen ai inference attributes for foundational text models [not ready for merge] Nov 8, 2024
@yiyuan-he yiyuan-he changed the title feat: set up gen ai inference attributes for foundational text models [not ready for merge] feat: set up gen ai inference attributes for foundational text models in java v1 sdk Nov 9, 2024
@yiyuan-he
Copy link
Author

Deprecating in favor of new PR

@yiyuan-he yiyuan-he closed this Nov 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant