Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OpenAI: Instrument chat completions #2404

Closed
4 tasks done
kaylareopelle opened this issue Jan 22, 2024 · 2 comments · Fixed by #2398
Closed
4 tasks done

OpenAI: Instrument chat completions #2404

kaylareopelle opened this issue Jan 22, 2024 · 2 comments · Fixed by #2398
Assignees
Labels
jan-mar qtr Possible FY Q4 candidate

Comments

@kaylareopelle
Copy link
Contributor

kaylareopelle commented Jan 22, 2024

Gem method: OpenAI::Client#chat
OpenAI Chat Completions Documentation
OpenAI Chat Completions API Reference

  • Create span when method is called if there’s a current transaction; Span name: Llm/completion/OpenAI/Create
  • Generate metric when method is called: Metric name: Ruby/ML/OpenAI/<version>
  • Create custom event LlmChatCompletionSummary (example attributes)(Internal)
  • Create custom event LlmChatCompletionMessage (example attributes)(Internal)

Notes:

  • There’s a deprecated API called completions. That’s not what we’re talking about. We’re talking about “chat completions”, which is accessed in the Ruby gem through OpenAI::Client#chat
  • Using the chat completions example linked above, the output should be one summary event and five message events. Four of the message events come from the request and one of the message events comes from the response.
  • Summary events and messages events should be created as part of the same unit of work because they are both derived from the OpenAI::Client#chat method.
@workato-integration
Copy link

@kaylareopelle kaylareopelle added this to the OpenAI Instrumentation milestone Jan 22, 2024
@kaylareopelle kaylareopelle changed the title Instrument Chat Completions Instrument chat completions Jan 22, 2024
@kaylareopelle kaylareopelle changed the title Instrument chat completions OpenAI: Instrument chat completions Jan 22, 2024
@kaylareopelle kaylareopelle self-assigned this Jan 22, 2024
@kaylareopelle kaylareopelle modified the milestones: OpenAI Instrumentation, Ruby Agent - AIM Support (OpenAI) Jan 22, 2024
@kaylareopelle kaylareopelle moved this from Triage to To do in Ruby Engineering Board Jan 22, 2024
@kaylareopelle kaylareopelle moved this from To do to In progress in Ruby Engineering Board Jan 22, 2024
@hannahramadan hannahramadan added the jan-mar qtr Possible FY Q4 candidate label Feb 13, 2024
@kaylareopelle kaylareopelle linked a pull request Feb 14, 2024 that will close this issue
@kaylareopelle
Copy link
Contributor Author

Closed by #2398

@github-project-automation github-project-automation bot moved this from In progress to Code Complete/Done in Ruby Engineering Board Feb 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
jan-mar qtr Possible FY Q4 candidate
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

2 participants