From f1f990eb81403c1163d2b79e47406026b742bfa8 Mon Sep 17 00:00:00 2001 From: Liudmila Molkova Date: Mon, 2 Sep 2024 20:03:41 -0700 Subject: [PATCH 01/17] Expand GenAI project to cover instrumentation libraries --- projects/gen-ai.md | 141 ++++++++++++++++++++++++++++++++++++++++ projects/llm-semconv.md | 93 -------------------------- 2 files changed, 141 insertions(+), 93 deletions(-) create mode 100644 projects/gen-ai.md delete mode 100644 projects/llm-semconv.md diff --git a/projects/gen-ai.md b/projects/gen-ai.md new file mode 100644 index 00000000..bbb68758 --- /dev/null +++ b/projects/gen-ai.md @@ -0,0 +1,141 @@ +# Generative AI Observability + +>[!IMPORTANT] This project extends LLM Semantic Conventions project + +## Description + +Generative AI observability is evolving fast. The industry is still learning how to use GenAI products which creates new opportunities and challenges in the observability space. GenAI observability is focused on development time and verbose telemetry. Non-deterministic nature of GenAI creates the need to measure the quality and safety of responses which become part of observability. + +There are multiple GenAI-focused observability projects out there such as [Arize Phoenix](https://docs.arize.com/phoenix), [Langfuse](https://langfuse.com/), [LangSmith](https://www.langchain.com/langsmith), [Langtrace](https://langtrace.ai/), [OpenLit](https://github.com/openlit/openlit), [Traceloop](https://www.traceloop.com/docs/introduction) and many others. + +Some of these projects are based on OpenTelemetry, others are "inspired by" it, but don't have out-of-the-box integration with OTel. + +The scope of GenAI observability includes instrumentations of various GenAI clients, frameworks, response evaluations, as well as model (server-side) instrumentations. + +Example Use Cases: + +- **Debugging**: Standardizing log data and error reporting for efficient error resolution in LLM applications. +- **Evaluation**: Collecting signal for evaluation of LLM performance, including response accuracy, context retention, and user feedback. +- **Cost Tracking**: Monitoring operational costs like computation time and API calls for resource optimization. +- **AI Safety**: Monitoring for ethical LLM use, including privacy compliance, bias detection, and secure data handling. + +For Spans: Instrumentation and debugging guidance for multi-step LLM applications, with recommendations on user privacy and intellectual property security, marking sensitive attributes and events as opt-in. +For Metrics: Tracking model performance by collecting and correlating data like model names, versions, parameters, latency, and evaluation scores. + +### Current challenges + +Existing GenAI observability projects are driven independently by individual vendors which don't have a common space to collaborate on common instrumentation libraries. + +[OpenLLMetry](https://github.com/traceloop/openllmetry) project was created with the intention [to create space under CNCF](https://github.com/cncf/sandbox/issues/67) and gave a start to OTel [LLM Semantic Convention](https://github.com/open-telemetry/community/blob/main/projects/llm-semconv.md) project. + +OpenLLMetry provides general-purpose OTel instrumentations for GenAI clients, frameworks and related components; they are governed by the [Traceloop](https://www.traceloop.com/docs/introduction). + +By providing GenAI community with the space under OpenTelemetry we will be able to start hosting vendor-agnostic instrumentation libraries (such as WIP one [OpenAI Python](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/2759)) and eventually we will be able to move popular OpenLLMetry-owned `opentelemetry-instrumentation-*` libraries to OTel (or to upstream libraries). + +It helps GenAI observability community to + +- find solutions for new observability problems such as verbose events or correlation with async response evaluations +- discuss implementation challenges +- coordinate and prioritize common efforts + +### Goals, objectives, and requirements + +GenAI observability project is a new OTel cross-cutting SIG that intends to provide the GenAI community with a space to evolve and maintain GenAI related instrumentations in a **vendor-agnostic way**. + +It includes: + +- LLM Semantic Conventions +- Instrumentations for GenAI client libraries, frameworks, or other components. + +The eventual goal is to contribute instrumentation libraries to upstream components or to otel-contrib repos when upstream contributions are not possible. + +During initial development and experimentation, we may need to release libraries more frequently than it's possible for contrib and then may temporarily host instrumentation libraries in a `opentelemetry-{language}-genai` repository(ies). + +This would help GenAI observability vendors that will depend on these libraries to patch issues and adapt to external changes. + +## Deliverables + +- develop common instrumentation libraries for popular GenAI clients and frameworks +- suggest and, when possible, contribute them to upstream libraries +- define and evolve GenAI Semantic Conventions +- propose and implement changes in the OTel API/SDK/contrib/etc that are necessary for GenAI efforts + +## Staffing / Help Wanted + +The initial set of approvers for each language must include: + +- representatives of at least 2 different observability vendors that are active members of OTel LLM SemConv WG +- at least one existing approver or maintainer of the contrib, instrumentation or core OTel repo in the corresponding language + +**Project Leads:** + +- @drewby +- @nirga + +**Sponsoring Members:** + +- @lmolkova +- TBD + +**Engineers:** +- Python + - maintainers/approvers: + - @drewby + - @karthikscale3 + - @lzchen + - @nirga + - TBD + - contributors: + - @lmolkova + - TBD +- JS + - maintainers/approvers: + - @drewby + - @karthikscale3 + - @nirga + - TBD + - contributors: + - TBD + +## Meeting Times + +Alternating weekly meetings to accommodate different time zones: + +- Week 1: Wednesdays at 10 AM PST +- Week 2: Wednesdays at 11 PM PST + +(will likely add more slots depending on the demand) + +### Meeting Links + +- Meeting Calendar: [calendar-semconv@opentelemetry.io](https://groups.google.com/a/opentelemetry.io/g/calendar-semconv) +- Meeting Notes: [Google Doc](https://docs.google.com/document/d/1EKIeDgBGXQPGehUigIRLwAUpRGa7-1kXB736EaYuJ2M) + +## Discussion + +* Slack: [#otel-llm-semconv-wg](https://cloud-native.slack.com/archives/C06KR7ARS3X) +(TODO: rename to `#otel-gen-ai-wg`) + +## Timeline + +- Semantic Conventions for GenAI clients + - Experimental, covering most important features: Apr 2025 + - Stable conventions: Date TBD +- Official OTel Instrumentation libraries for popular clients in JS and Python: + - OpenAI instrumentation library shipped: Nov 2024 + - Other popular client instrumentations supported by OpenLLMetry: Apr 2025 + +## Labels + +- `gen-ai` + +## Linked Issues and PRs + +- Issue: [Introduce semantic conventions for modern AI (LLMs, vector databases, etc.) #327](https://github.com/open-telemetry/semantic-conventions/issues/327) +- PR: [LLM Semantic Conventions: Initial PR #825](https://github.com/open-telemetry/semantic-conventions/pull/825) +- PR: [Add LLM semantic conventions #639](https://github.com/open-telemetry/semantic-conventions/pull/639) - Closed/Replaced with 825 +- PR: [[WIP] Add Initial Support for Instrumenting OpenAI Python Library - Chat Completion Create](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/2759) +## Project Board + +* Project Board: [LLM Client Semantic Conventions](https://github.com/orgs/open-telemetry/projects/82) +(TODO: rename to `GenAI Client Instrumentation`) \ No newline at end of file diff --git a/projects/llm-semconv.md b/projects/llm-semconv.md deleted file mode 100644 index 7346fcef..00000000 --- a/projects/llm-semconv.md +++ /dev/null @@ -1,93 +0,0 @@ -# LLM Observability Semantic Conventions - -## Description - -This workgroup aims to identify and define a standardized vocabulary and specification for observability of Large Language Model (LLM) based applications. The effort would explore various LLM-based application APIs, LLM tasks and LLM workflows for multistage reasoning. Key areas include: - -- Evaluating existing vendor-specific conventions (which may not be currently standardized but implementation specific) -- Developing a standardized vocabulary for observability specific semantic conventions for single stage and multistage LLM workflows -- Initial scope focused on metrics and traces (which could later be expanded to other telemetry data signals) - -Example Use Cases: - -- **Debugging**: Standardizing log data and error reporting for efficient error resolution in LLM applications. -- **Evaluation**: Collecting signal for evaluation of LLM performance, including response accuracy, context retention, and user feedback. -- **Cost Tracking**: Monitoring operational costs like computation time and API calls for resource optimization. -- **AI Safety**: Monitoring for ethical LLM use, including privacy compliance, bias detection, and secure data handling. - -For Spans: Instrumentation and debugging guidance for multi-step LLM applications, with recommendations on user privacy and intellectual property security, marking sensitive attributes and events as opt-in. - -For Metrics: Tracking model performance by collecting and correlating data like model names, versions, parameters, latency, and evaluation scores. - -## Deliverables - -- Semantic conventions for LLMs, allowing flexibility for evolution in this fast-paced field. -- Reference prototypes demonstrating the application of these conventions in data collection, processing, and analysis. -- Recommended practices for monitoring LLM applications, including implementation strategies. - -## Staffing / Help Wanted - -We will gather further staff as the project is established. We need to -make sure there is enough representation across both vendors and customers -who will benefit from these semantic conventions. - -**Project Leads:** - -- @drewby -- @nirga - -**Sponsoring Members:** - -- @lmolkova -- @alolita - -**Engineers:** - -- @nirga -- @galkleinman -- @sudivate -- @cartermp -- @gyliu513 -- *Positions open for more engineers* - -**Maintainers:** - -- To be determined - -## Meeting Times - -Alternating weekly meetings to accommodate different time zones: - -- Week 1: Wednesdays at 10 AM PST -- Week 2: Wednesdays at 11 PM PST - -### Meeting Links - -- Meeting Calendar: [calendar-semconv@opentelemetry.io](https://groups.google.com/a/opentelemetry.io/g/calendar-semconv) -- Meeting Notes: [Google Doc](https://docs.google.com/document/d/1EKIeDgBGXQPGehUigIRLwAUpRGa7-1kXB736EaYuJ2M) - -## Discussion - -* Slack: [#otel-llm-semconv-wg](https://cloud-native.slack.com/archives/C06KR7ARS3X) - -## Timeline - -1. Initial Draft: February 2024 -1. Review and Refinement: April 2024 -1. Prototypes for .NET, Python, JS: June 2024 -1. Initial Merge Target: July 2024 -1. Ongoing Review for Stability: Date TBD - -## Labels - -- `LLM` - -## Linked Issues and PRs - -- Issue: [Introduce semantic conventions for modern AI (LLMs, vector databases, etc.) #327](https://github.com/open-telemetry/semantic-conventions/issues/327) -- PR: [LLM Semantic Conventions: Initial PR #825](https://github.com/open-telemetry/semantic-conventions/pull/825) -- PR: [Add LLM semantic conventions #639](https://github.com/open-telemetry/semantic-conventions/pull/639) - Closed/Replaced with 825 - -## Project Board - -* Project Board: [LLM Client Semantic Conventions](https://github.com/orgs/open-telemetry/projects/82) From eb794868431c8f2d049fa7ee1d1a5bc956764fd2 Mon Sep 17 00:00:00 2001 From: Liudmila Molkova Date: Tue, 3 Sep 2024 12:44:37 -0700 Subject: [PATCH 02/17] up --- projects/gen-ai.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/projects/gen-ai.md b/projects/gen-ai.md index bbb68758..b0a1150a 100644 --- a/projects/gen-ai.md +++ b/projects/gen-ai.md @@ -84,18 +84,20 @@ The initial set of approvers for each language must include: - @karthikscale3 - @lzchen - @nirga - - TBD - contributors: + - @alizenhom - @lmolkova - - TBD + - *looking for more contributors* - JS - maintainers/approvers: - @drewby - @karthikscale3 - @nirga - - TBD + - *looking for existing OTel JS approvers/maintainers for general JS & OTel feedback* - contributors: - - TBD + - @alizenhom + - *looking for more contributors* +- Other languages: not in the scope (yet). ## Meeting Times From 377146f95ef02b9a3104dd6d24b884d12827ce97 Mon Sep 17 00:00:00 2001 From: Liudmila Molkova Date: Wed, 4 Sep 2024 20:32:00 -0700 Subject: [PATCH 03/17] Update projects/gen-ai.md Co-authored-by: Riccardo Magliocchetti --- projects/gen-ai.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/projects/gen-ai.md b/projects/gen-ai.md index b0a1150a..35923d5a 100644 --- a/projects/gen-ai.md +++ b/projects/gen-ai.md @@ -28,7 +28,7 @@ Existing GenAI observability projects are driven independently by individual ven [OpenLLMetry](https://github.com/traceloop/openllmetry) project was created with the intention [to create space under CNCF](https://github.com/cncf/sandbox/issues/67) and gave a start to OTel [LLM Semantic Convention](https://github.com/open-telemetry/community/blob/main/projects/llm-semconv.md) project. -OpenLLMetry provides general-purpose OTel instrumentations for GenAI clients, frameworks and related components; they are governed by the [Traceloop](https://www.traceloop.com/docs/introduction). +OpenLLMetry provides general-purpose OTel instrumentations for GenAI clients, frameworks and related components; they are governed by [Traceloop](https://www.traceloop.com/docs/introduction). By providing GenAI community with the space under OpenTelemetry we will be able to start hosting vendor-agnostic instrumentation libraries (such as WIP one [OpenAI Python](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/2759)) and eventually we will be able to move popular OpenLLMetry-owned `opentelemetry-instrumentation-*` libraries to OTel (or to upstream libraries). From c49fac454415ff9f6b1e541300a61e1360962e6d Mon Sep 17 00:00:00 2001 From: Liudmila Molkova Date: Wed, 4 Sep 2024 20:32:06 -0700 Subject: [PATCH 04/17] Update projects/gen-ai.md Co-authored-by: Riccardo Magliocchetti --- projects/gen-ai.md | 1 + 1 file changed, 1 insertion(+) diff --git a/projects/gen-ai.md b/projects/gen-ai.md index 35923d5a..36582a5f 100644 --- a/projects/gen-ai.md +++ b/projects/gen-ai.md @@ -84,6 +84,7 @@ The initial set of approvers for each language must include: - @karthikscale3 - @lzchen - @nirga + - @xrmx - contributors: - @alizenhom - @lmolkova From 638b9ae8c1f06da2b788d1f3e06198827414edfb Mon Sep 17 00:00:00 2001 From: Liudmila Molkova Date: Wed, 4 Sep 2024 21:43:24 -0700 Subject: [PATCH 05/17] goalsgit status --- projects/gen-ai.md | 30 +++++++++++++++++++++--------- 1 file changed, 21 insertions(+), 9 deletions(-) diff --git a/projects/gen-ai.md b/projects/gen-ai.md index 36582a5f..08ddbd41 100644 --- a/projects/gen-ai.md +++ b/projects/gen-ai.md @@ -55,10 +55,19 @@ This would help GenAI observability vendors that will depend on these libraries ## Deliverables -- develop common instrumentation libraries for popular GenAI clients and frameworks -- suggest and, when possible, contribute them to upstream libraries -- define and evolve GenAI Semantic Conventions -- propose and implement changes in the OTel API/SDK/contrib/etc that are necessary for GenAI efforts +Immediate term: +- Ship OTel instrumentation libraries for OpenAI (or any other GenAI client) in Python and JS following exiting conventions + +Middle term: +- Ship OpenTelemetry (or native) instrumentations for popular GenAI client libraries in Python and JS covering chat calls +- Evolve GenAI semantic conventions to cover other popular GenAI operations such as embeddings, image or audio generation + +As a result we should have feature parity with OpenLLMetry for a set of client instrumentation libraries that all vendors can depends upon. + +Long term: +- Implement instrumentations for GenAI orchestrators and LLM frameworks for popular libraries in different language +- Evolve GenAI and other relevant conventions (DB) to cover complex multi-step scenarios such as RAG +- Propose mature instrumentations to upstream libraries/frameworks ## Staffing / Help Wanted @@ -84,31 +93,34 @@ The initial set of approvers for each language must include: - @karthikscale3 - @lzchen - @nirga - - @xrmx + - @xrmx - contributors: - @alizenhom + - @galkleinman - @lmolkova - *looking for more contributors* -- JS +- JavaScript - maintainers/approvers: - @drewby - @karthikscale3 - @nirga + - @trentm - *looking for existing OTel JS approvers/maintainers for general JS & OTel feedback* - contributors: - @alizenhom + - @galkleinman - *looking for more contributors* - Other languages: not in the scope (yet). ## Meeting Times -Alternating weekly meetings to accommodate different time zones: +To be determined (will run a vote). + +We'll use LLM Semantic Convention meetings until more suitable time for new participants is determined: - Week 1: Wednesdays at 10 AM PST - Week 2: Wednesdays at 11 PM PST -(will likely add more slots depending on the demand) - ### Meeting Links - Meeting Calendar: [calendar-semconv@opentelemetry.io](https://groups.google.com/a/opentelemetry.io/g/calendar-semconv) From a31268eea469f03fa8a835c7d9e6b39251543347 Mon Sep 17 00:00:00 2001 From: Liudmila Molkova Date: Wed, 4 Sep 2024 22:03:30 -0700 Subject: [PATCH 06/17] update use cases --- projects/gen-ai.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/projects/gen-ai.md b/projects/gen-ai.md index 08ddbd41..68772974 100644 --- a/projects/gen-ai.md +++ b/projects/gen-ai.md @@ -14,13 +14,13 @@ The scope of GenAI observability includes instrumentations of various GenAI clie Example Use Cases: -- **Debugging**: Standardizing log data and error reporting for efficient error resolution in LLM applications. -- **Evaluation**: Collecting signal for evaluation of LLM performance, including response accuracy, context retention, and user feedback. -- **Cost Tracking**: Monitoring operational costs like computation time and API calls for resource optimization. -- **AI Safety**: Monitoring for ethical LLM use, including privacy compliance, bias detection, and secure data handling. - -For Spans: Instrumentation and debugging guidance for multi-step LLM applications, with recommendations on user privacy and intellectual property security, marking sensitive attributes and events as opt-in. -For Metrics: Tracking model performance by collecting and correlating data like model names, versions, parameters, latency, and evaluation scores. +- **Debugging**: Tracing provides causality for multi-step LLM applications allowing to understand the operation flow and root-cause errors. +- **Evaluation**: Input/output events record non-deterministic model responses that can be used for debugging or automated evaluations + measuring LLM response accuracy and other quality characteristics. +- **AI Safety**: GenAI telemetry, along with specialized evaluators, can help ensure ethical use of LLMs, maintain privacy compliance, detect bias, and safeguard data security. +- **Cost Tracking**: Usage metrics based on model self-reporting capabilities provide insights into operational costs. +- **Performance optimization**: Latency, throughput, and error rate of LLM operations provide traditional performance monitoring layer for + applications running in production. ### Current challenges @@ -150,6 +150,7 @@ We'll use LLM Semantic Convention meetings until more suitable time for new part - PR: [LLM Semantic Conventions: Initial PR #825](https://github.com/open-telemetry/semantic-conventions/pull/825) - PR: [Add LLM semantic conventions #639](https://github.com/open-telemetry/semantic-conventions/pull/639) - Closed/Replaced with 825 - PR: [[WIP] Add Initial Support for Instrumenting OpenAI Python Library - Chat Completion Create](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/2759) + ## Project Board * Project Board: [LLM Client Semantic Conventions](https://github.com/orgs/open-telemetry/projects/82) From 0c18e42dc270a6d6b9b660d6a5e0a232d8cf7bc0 Mon Sep 17 00:00:00 2001 From: Liudmila Molkova Date: Thu, 5 Sep 2024 19:32:37 -0700 Subject: [PATCH 07/17] Apply suggestions from code review Co-authored-by: patcher9 Co-authored-by: Emily S --- projects/gen-ai.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/projects/gen-ai.md b/projects/gen-ai.md index 68772974..be84c1dd 100644 --- a/projects/gen-ai.md +++ b/projects/gen-ai.md @@ -28,9 +28,9 @@ Existing GenAI observability projects are driven independently by individual ven [OpenLLMetry](https://github.com/traceloop/openllmetry) project was created with the intention [to create space under CNCF](https://github.com/cncf/sandbox/issues/67) and gave a start to OTel [LLM Semantic Convention](https://github.com/open-telemetry/community/blob/main/projects/llm-semconv.md) project. -OpenLLMetry provides general-purpose OTel instrumentations for GenAI clients, frameworks and related components; they are governed by [Traceloop](https://www.traceloop.com/docs/introduction). +GenAI Instrumentation libraries like OpenLLMetry, OpenLIT, LangTrace and other provides general-purpose OTel instrumentations for GenAI clients, frameworks and related components. -By providing GenAI community with the space under OpenTelemetry we will be able to start hosting vendor-agnostic instrumentation libraries (such as WIP one [OpenAI Python](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/2759)) and eventually we will be able to move popular OpenLLMetry-owned `opentelemetry-instrumentation-*` libraries to OTel (or to upstream libraries). +By providing GenAI community with the space under OpenTelemetry we will be able to start hosting vendor-agnostic instrumentation libraries (such as WIP one [OpenAI Python](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/2759)) and [OpenAI JS](https://github.com/open-telemetry/opentelemetry-js-contrib/pull/2402) and eventually we will be able to move instrumentations from current GenAI Observabilty tools to OTel (or to upstream libraries). It helps GenAI observability community to @@ -56,16 +56,16 @@ This would help GenAI observability vendors that will depend on these libraries ## Deliverables Immediate term: -- Ship OTel instrumentation libraries for OpenAI (or any other GenAI client) in Python and JS following exiting conventions +- Ship OTel instrumentation libraries for OpenAI (or any other GenAI client) in Python and JS following existing conventions Middle term: - Ship OpenTelemetry (or native) instrumentations for popular GenAI client libraries in Python and JS covering chat calls - Evolve GenAI semantic conventions to cover other popular GenAI operations such as embeddings, image or audio generation -As a result we should have feature parity with OpenLLMetry for a set of client instrumentation libraries that all vendors can depends upon. +As a result we should have feature parity with the instrumentations of existing GenAI Observability vendors for a set of client instrumentation libraries that all vendors can depends upon. Long term: -- Implement instrumentations for GenAI orchestrators and LLM frameworks for popular libraries in different language +- Implement instrumentations for GenAI orchestrators and LLM frameworks for popular libraries in different languages - Evolve GenAI and other relevant conventions (DB) to cover complex multi-step scenarios such as RAG - Propose mature instrumentations to upstream libraries/frameworks @@ -150,7 +150,7 @@ We'll use LLM Semantic Convention meetings until more suitable time for new part - PR: [LLM Semantic Conventions: Initial PR #825](https://github.com/open-telemetry/semantic-conventions/pull/825) - PR: [Add LLM semantic conventions #639](https://github.com/open-telemetry/semantic-conventions/pull/639) - Closed/Replaced with 825 - PR: [[WIP] Add Initial Support for Instrumenting OpenAI Python Library - Chat Completion Create](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/2759) - +- PR: [[WIP] Add Initial Support for Instrumenting OpenAI JS Library - Chat Completion Create](https://github.com/open-telemetry/opentelemetry-js-contrib/pull/2402) ## Project Board * Project Board: [LLM Client Semantic Conventions](https://github.com/orgs/open-telemetry/projects/82) From 5a1f034d7cfadd9ba4802d3ffdaf9acb504877f6 Mon Sep 17 00:00:00 2001 From: Liudmila Molkova Date: Thu, 5 Sep 2024 19:38:16 -0700 Subject: [PATCH 08/17] up --- .cspell.yaml | 6 ++++++ projects/gen-ai.md | 5 +++++ 2 files changed, 11 insertions(+) diff --git a/.cspell.yaml b/.cspell.yaml index 195fc8d5..b5d0853d 100644 --- a/.cspell.yaml +++ b/.cspell.yaml @@ -16,6 +16,7 @@ ignoreRegExpList: - GitHub Handle - GitHub Handle in YML words: + - Arize - Ashpole - Collibra - Coralogix @@ -38,6 +39,8 @@ words: - keptn - kubecon - k8sclusterreceiver + - Langfuse + - Langtrace - liatrio - lightstep - logz @@ -110,6 +113,7 @@ words: - fong - frzifus - gbbr + - genai - gitter - gitdm - henrikrexed @@ -146,6 +150,7 @@ words: - mateuszrzeszutek - mayur - mayurkale + - Metry - mdelfabro - mhausenblas - mirabella @@ -210,6 +215,7 @@ words: - sudivate - suereth - supermajority + - Traceloop - tarnovski - tekton - thisthat diff --git a/projects/gen-ai.md b/projects/gen-ai.md index be84c1dd..5fbb37a8 100644 --- a/projects/gen-ai.md +++ b/projects/gen-ai.md @@ -95,9 +95,12 @@ The initial set of approvers for each language must include: - @nirga - @xrmx - contributors: + - @AmanAgarwal041 - @alizenhom - @galkleinman + - @gyliu513 - @lmolkova + - @patcher9 - *looking for more contributors* - JavaScript - maintainers/approvers: @@ -107,8 +110,10 @@ The initial set of approvers for each language must include: - @trentm - *looking for existing OTel JS approvers/maintainers for general JS & OTel feedback* - contributors: + - @AmanAgarwal041 - @alizenhom - @galkleinman + - @patcher9 - *looking for more contributors* - Other languages: not in the scope (yet). From d2929a89d9cd63023184107180d6f1ce3c940ee0 Mon Sep 17 00:00:00 2001 From: Liudmila Molkova Date: Thu, 5 Sep 2024 19:49:17 -0700 Subject: [PATCH 09/17] more use-cases --- projects/gen-ai.md | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/projects/gen-ai.md b/projects/gen-ai.md index 5fbb37a8..ecf38835 100644 --- a/projects/gen-ai.md +++ b/projects/gen-ai.md @@ -15,12 +15,17 @@ The scope of GenAI observability includes instrumentations of various GenAI clie Example Use Cases: - **Debugging**: Tracing provides causality for multi-step LLM applications allowing to understand the operation flow and root-cause errors. -- **Evaluation**: Input/output events record non-deterministic model responses that can be used for debugging or automated evaluations - measuring LLM response accuracy and other quality characteristics. +- **Evaluation**: Prompt/completion events record non-deterministic model responses that can be used for debugging or automated evaluations + measuring LLM response accuracy and other quality characteristics. In addition to automated analysis, recording user feedback tied to LLM outputs + can provide insights into the quality of model outputs and how they affect user engagement. - **AI Safety**: GenAI telemetry, along with specialized evaluators, can help ensure ethical use of LLMs, maintain privacy compliance, detect bias, and safeguard data security. - **Cost Tracking**: Usage metrics based on model self-reporting capabilities provide insights into operational costs. - **Performance optimization**: Latency, throughput, and error rate of LLM operations provide traditional performance monitoring layer for applications running in production. +- **Incident Response Automation**: Integrating observability data with AI operations platforms can enable automated incident responses, + such as restarting failing model components or scaling up resources in response to increased demand, without manual intervention. +- **Version Control and Rollback**: Tracking model versions in observability data helps monitor the performance impact of model updates, + allowing quick rollback to previous versions in case a new release introduces errors or degradations in quality. ### Current challenges @@ -30,7 +35,7 @@ Existing GenAI observability projects are driven independently by individual ven GenAI Instrumentation libraries like OpenLLMetry, OpenLIT, LangTrace and other provides general-purpose OTel instrumentations for GenAI clients, frameworks and related components. -By providing GenAI community with the space under OpenTelemetry we will be able to start hosting vendor-agnostic instrumentation libraries (such as WIP one [OpenAI Python](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/2759)) and [OpenAI JS](https://github.com/open-telemetry/opentelemetry-js-contrib/pull/2402) and eventually we will be able to move instrumentations from current GenAI Observabilty tools to OTel (or to upstream libraries). +By providing GenAI community with the space under OpenTelemetry we will be able to start hosting vendor-agnostic instrumentation libraries (such as WIP one [OpenAI Python](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/2759)) and [OpenAI JS](https://github.com/open-telemetry/opentelemetry-js-contrib/pull/2402) and eventually we will be able to move instrumentations from current GenAI Observability tools to OTel (or to upstream libraries). It helps GenAI observability community to @@ -62,7 +67,8 @@ Middle term: - Ship OpenTelemetry (or native) instrumentations for popular GenAI client libraries in Python and JS covering chat calls - Evolve GenAI semantic conventions to cover other popular GenAI operations such as embeddings, image or audio generation -As a result we should have feature parity with the instrumentations of existing GenAI Observability vendors for a set of client instrumentation libraries that all vendors can depends upon. +As a result we should have feature parity with the instrumentations of existing GenAI Observability vendors for a set of client +instrumentation libraries that all vendors can depend upon. Long term: - Implement instrumentations for GenAI orchestrators and LLM frameworks for popular libraries in different languages From c6c39984143d789dfa3eafc483edb1e1311b6594 Mon Sep 17 00:00:00 2001 From: Liudmila Molkova Date: Thu, 5 Sep 2024 20:31:03 -0700 Subject: [PATCH 10/17] clarify roles --- projects/gen-ai.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/projects/gen-ai.md b/projects/gen-ai.md index ecf38835..e3652b32 100644 --- a/projects/gen-ai.md +++ b/projects/gen-ai.md @@ -94,12 +94,14 @@ The initial set of approvers for each language must include: **Engineers:** - Python - - maintainers/approvers: + - maintainers: + - @lzchen + - TBD + - approvers/codeowners: - @drewby - @karthikscale3 - - @lzchen - @nirga - - @xrmx + - @xrmx- - contributors: - @AmanAgarwal041 - @alizenhom @@ -109,11 +111,13 @@ The initial set of approvers for each language must include: - @patcher9 - *looking for more contributors* - JavaScript - - maintainers/approvers: + - maintainers: + - @trentm + - TBD + - approvers/codeowners: - @drewby - @karthikscale3 - @nirga - - @trentm - *looking for existing OTel JS approvers/maintainers for general JS & OTel feedback* - contributors: - @AmanAgarwal041 From 20b1b20b12b356fcaab093c97ec08f1f4265f769 Mon Sep 17 00:00:00 2001 From: Liudmila Molkova Date: Thu, 5 Sep 2024 20:31:15 -0700 Subject: [PATCH 11/17] clarify roles --- projects/gen-ai.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/projects/gen-ai.md b/projects/gen-ai.md index e3652b32..9117a1e6 100644 --- a/projects/gen-ai.md +++ b/projects/gen-ai.md @@ -101,7 +101,7 @@ The initial set of approvers for each language must include: - @drewby - @karthikscale3 - @nirga - - @xrmx- + - @xrmx - contributors: - @AmanAgarwal041 - @alizenhom From 8a6866a48eb7ee8b9615b9c2fb8a9f85046d81dd Mon Sep 17 00:00:00 2001 From: Liudmila Molkova Date: Mon, 9 Sep 2024 11:32:39 -0700 Subject: [PATCH 12/17] refactor contributors, apply suggestions --- .cspell.yaml | 1 + projects/gen-ai.md | 62 ++++++++++++++++++++-------------------------- 2 files changed, 28 insertions(+), 35 deletions(-) diff --git a/.cspell.yaml b/.cspell.yaml index b5d0853d..637fd56b 100644 --- a/.cspell.yaml +++ b/.cspell.yaml @@ -169,6 +169,7 @@ words: - nikimanoledaki - novotny - observiq + - OpenLIT - opentelemetry - opentelemetrybot - otel diff --git a/projects/gen-ai.md b/projects/gen-ai.md index 9117a1e6..3c6a2fb1 100644 --- a/projects/gen-ai.md +++ b/projects/gen-ai.md @@ -31,7 +31,7 @@ Example Use Cases: Existing GenAI observability projects are driven independently by individual vendors which don't have a common space to collaborate on common instrumentation libraries. -[OpenLLMetry](https://github.com/traceloop/openllmetry) project was created with the intention [to create space under CNCF](https://github.com/cncf/sandbox/issues/67) and gave a start to OTel [LLM Semantic Convention](https://github.com/open-telemetry/community/blob/main/projects/llm-semconv.md) project. +[OpenLLMetry](https://github.com/traceloop/openllmetry) project was created with the intention [to create space under CNCF](https://github.com/cncf/sandbox/issues/67) and gave a start to OTel [LLM Semantic Convention](https://github.com/open-telemetry/community/pull/1928) project. GenAI Instrumentation libraries like OpenLLMetry, OpenLIT, LangTrace and other provides general-purpose OTel instrumentations for GenAI clients, frameworks and related components. @@ -49,7 +49,7 @@ GenAI observability project is a new OTel cross-cutting SIG that intends to prov It includes: -- LLM Semantic Conventions +- GenAI Semantic Conventions - Instrumentations for GenAI client libraries, frameworks, or other components. The eventual goal is to contribute instrumentation libraries to upstream components or to otel-contrib repos when upstream contributions are not possible. @@ -71,7 +71,7 @@ As a result we should have feature parity with the instrumentations of existing instrumentation libraries that all vendors can depend upon. Long term: -- Implement instrumentations for GenAI orchestrators and LLM frameworks for popular libraries in different languages +- Implement instrumentations for GenAI orchestrators and GenAI frameworks for popular libraries in different languages - Evolve GenAI and other relevant conventions (DB) to cover complex multi-step scenarios such as RAG - Propose mature instrumentations to upstream libraries/frameworks @@ -79,7 +79,7 @@ Long term: The initial set of approvers for each language must include: -- representatives of at least 2 different observability vendors that are active members of OTel LLM SemConv WG +- representatives of at least 2 different observability vendors that are active members of OTel GenAI SemConv WG - at least one existing approver or maintainer of the contrib, instrumentation or core OTel repo in the corresponding language **Project Leads:** @@ -93,39 +93,31 @@ The initial set of approvers for each language must include: - TBD **Engineers:** -- Python - - maintainers: + +- **initial** maintainers (based on the existing membership in the corresponding repos) + - Python - @lzchen - - TBD - - approvers/codeowners: - - @drewby - - @karthikscale3 - - @nirga - @xrmx - - contributors: - - @AmanAgarwal041 - - @alizenhom - - @galkleinman - - @gyliu513 - - @lmolkova - - @patcher9 - - *looking for more contributors* -- JavaScript - - maintainers: + - TBD + - JavaScript - @trentm + - *looking for existing OTel JS maintainers for general JS & OTel feedback* - TBD - - approvers/codeowners: - - @drewby - - @karthikscale3 - - @nirga - - *looking for existing OTel JS approvers/maintainers for general JS & OTel feedback* - - contributors: - - @AmanAgarwal041 - - @alizenhom - - @galkleinman - - @patcher9 - - *looking for more contributors* -- Other languages: not in the scope (yet). +- **initial** JS and Python approvers/codeowners (based on LLM Semantic Convention project involvement): + - @drewby + - @karthikscale3 + - @nirga +- contributors: + - @AmanAgarwal041 (JS & Python) + - @alizenhom (JS & Python) + - @codefromthecrypt (JS & Python) + - @galkleinman (JS & Python) + - @gyliu513 (JS & Python) + - @lmolkova (Python) + - @patcher9 (JS & Python) + - *looking for more contributors* + +Other languages are not in the scope (yet). ## Meeting Times @@ -134,7 +126,7 @@ To be determined (will run a vote). We'll use LLM Semantic Convention meetings until more suitable time for new participants is determined: - Week 1: Wednesdays at 10 AM PST -- Week 2: Wednesdays at 11 PM PST +- Week 2: Thursday 14:00 UTC+8 ### Meeting Links @@ -153,7 +145,7 @@ We'll use LLM Semantic Convention meetings until more suitable time for new part - Stable conventions: Date TBD - Official OTel Instrumentation libraries for popular clients in JS and Python: - OpenAI instrumentation library shipped: Nov 2024 - - Other popular client instrumentations supported by OpenLLMetry: Apr 2025 + - Other popular client instrumentations supported by GenAI observability vendors: Apr 2025 ## Labels From 58e32863ebbf6e969305423983bc2775340bfd56 Mon Sep 17 00:00:00 2001 From: Liudmila Molkova Date: Mon, 9 Sep 2024 12:01:19 -0700 Subject: [PATCH 13/17] up --- projects/gen-ai.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/projects/gen-ai.md b/projects/gen-ai.md index 3c6a2fb1..eb482d1d 100644 --- a/projects/gen-ai.md +++ b/projects/gen-ai.md @@ -94,7 +94,7 @@ The initial set of approvers for each language must include: **Engineers:** -- **initial** maintainers (based on the existing membership in the corresponding repos) +- **initial** maintainers (based on the existing membership in the corresponding repo) - Python - @lzchen - @xrmx @@ -103,7 +103,7 @@ The initial set of approvers for each language must include: - @trentm - *looking for existing OTel JS maintainers for general JS & OTel feedback* - TBD -- **initial** JS and Python approvers/codeowners (based on LLM Semantic Convention project involvement): +- **initial** JS and Python GenAI component/code owners (based on LLM Semantic Convention project involvement): - @drewby - @karthikscale3 - @nirga From 3677fed709ff339a2abbfe49b382256436316228 Mon Sep 17 00:00:00 2001 From: Liudmila Molkova Date: Tue, 24 Sep 2024 16:11:50 -0700 Subject: [PATCH 14/17] up --- projects/gen-ai.md | 39 ++++++++++++++++++--------------------- 1 file changed, 18 insertions(+), 21 deletions(-) diff --git a/projects/gen-ai.md b/projects/gen-ai.md index eb482d1d..410e6de2 100644 --- a/projects/gen-ai.md +++ b/projects/gen-ai.md @@ -52,11 +52,18 @@ It includes: - GenAI Semantic Conventions - Instrumentations for GenAI client libraries, frameworks, or other components. -The eventual goal is to contribute instrumentation libraries to upstream components or to otel-contrib repos when upstream contributions are not possible. +The eventual goal is to contribute instrumentation libraries to upstream components. When it's not possible or when libraries need some time to mature, +these libraries will evolve in contrib repositories. -During initial development and experimentation, we may need to release libraries more frequently than it's possible for contrib and then may temporarily host instrumentation libraries in a `opentelemetry-{language}-genai` repository(ies). +JS contrib provides component-ownership model with high level of [autonomy](https://github.com/open-telemetry/opentelemetry-js-contrib/blob/main/CONTRIBUTING.md#component-ownership). +We can explore if this can be applied to Python. -This would help GenAI observability vendors that will depend on these libraries to patch issues and adapt to external changes. +> [!NOTE] +> The release cadence of contrib repo needs to support releasing components relatively frequently to +> patch issues and adapt to upstream changes so that GenAI observability vendors could depend on them. + +Python contrib repo is going to support [per-package release](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/2875) soon. +JS contrib release is automated with Release Please - TBD. ## Deliverables @@ -77,10 +84,9 @@ Long term: ## Staffing / Help Wanted -The initial set of approvers for each language must include: +The development will start in the OTel contrib repositories and/or in upstream libraries following ownership model that exist in this repo. -- representatives of at least 2 different observability vendors that are active members of OTel GenAI SemConv WG -- at least one existing approver or maintainer of the contrib, instrumentation or core OTel repo in the corresponding language +Some of the repositories (such as opentelemetry-js-contrib) provide a lot of autonomy to the [component owners](https://github.com/open-telemetry/opentelemetry-js-contrib/blob/main/CONTRIBUTING.md#component-ownership). **Project Leads:** @@ -93,28 +99,19 @@ The initial set of approvers for each language must include: - TBD **Engineers:** - -- **initial** maintainers (based on the existing membership in the corresponding repo) - - Python - - @lzchen - - @xrmx - - TBD - - JavaScript - - @trentm - - *looking for existing OTel JS maintainers for general JS & OTel feedback* - - TBD -- **initial** JS and Python GenAI component/code owners (based on LLM Semantic Convention project involvement): - - @drewby - - @karthikscale3 - - @nirga -- contributors: - @AmanAgarwal041 (JS & Python) - @alizenhom (JS & Python) - @codefromthecrypt (JS & Python) + - @drewby (JS & Python) - @galkleinman (JS & Python) - @gyliu513 (JS & Python) + - @karthikscale3 (JS & Python) - @lmolkova (Python) + - @lzchen (Python) + - @nirga (JS & Python) - @patcher9 (JS & Python) + - @trentm (JS) + - @xrmx (Python) - *looking for more contributors* Other languages are not in the scope (yet). From c757f9e0177457f7143b9dce42823c1dc179ce32 Mon Sep 17 00:00:00 2001 From: Liudmila Molkova Date: Tue, 24 Sep 2024 16:29:16 -0700 Subject: [PATCH 15/17] Remove approvers/maintainers, add time commitment, add meeting poll --- projects/gen-ai.md | 41 ++++++++++++++++++++++------------------- 1 file changed, 22 insertions(+), 19 deletions(-) diff --git a/projects/gen-ai.md b/projects/gen-ai.md index 410e6de2..d0fd9a18 100644 --- a/projects/gen-ai.md +++ b/projects/gen-ai.md @@ -99,31 +99,33 @@ Some of the repositories (such as opentelemetry-js-contrib) provide a lot of aut - TBD **Engineers:** - - @AmanAgarwal041 (JS & Python) - - @alizenhom (JS & Python) - - @codefromthecrypt (JS & Python) - - @drewby (JS & Python) - - @galkleinman (JS & Python) - - @gyliu513 (JS & Python) - - @karthikscale3 (JS & Python) - - @lmolkova (Python) - - @lzchen (Python) - - @nirga (JS & Python) - - @patcher9 (JS & Python) - - @trentm (JS) - - @xrmx (Python) - - *looking for more contributors* + +| Alias | Language | Company/Project | Time commitment | +| ----- | -------- | ----------------|---------------- | +| @AmanAgarwal041 | JS & Python | OpenLIT | | +| @alizenhom | JS & Python | Langtrace | | +| @codefromthecrypt | JS & Python | Elastic | | +| @drewby | JS & Python | Microsoft | | +| @galkleinman | JS & Python | Traceloop | | +| @gyliu513 | JS & Python | IBM | | +| @karthikscale3 | JS & Python | Langtrace | | +| @lmolkova | Python | Microsoft | | 16 hours/week | +| @lzchen | Python | Microsoft | | | +| @nirga | JS & Python | Traceloop | | +| @patcher9 | JS & Python | OpenLIT | | +| @trentm | JS | Elastic | | +| @xrmx | Python | Elastic | | + +*looking for more contributors* Other languages are not in the scope (yet). ## Meeting Times -To be determined (will run a vote). - -We'll use LLM Semantic Convention meetings until more suitable time for new participants is determined: +We'll use GenAI LLM Semantic Convention meetings until more suitable time for new participants is determined: -- Week 1: Wednesdays at 10 AM PST -- Week 2: Thursday 14:00 UTC+8 +- **Every week**: Wednesdays at 10 AM PST - Looking for alternative time to accommodate more time zones - https://strawpoll.com/e6Z2ApEKXgN +- **Every other week**: Thursday 14:00 UTC+8 ### Meeting Links @@ -155,6 +157,7 @@ We'll use LLM Semantic Convention meetings until more suitable time for new part - PR: [Add LLM semantic conventions #639](https://github.com/open-telemetry/semantic-conventions/pull/639) - Closed/Replaced with 825 - PR: [[WIP] Add Initial Support for Instrumenting OpenAI Python Library - Chat Completion Create](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/2759) - PR: [[WIP] Add Initial Support for Instrumenting OpenAI JS Library - Chat Completion Create](https://github.com/open-telemetry/opentelemetry-js-contrib/pull/2402) + ## Project Board * Project Board: [LLM Client Semantic Conventions](https://github.com/orgs/open-telemetry/projects/82) From 2c564af2de1939beadd3a70b582c15b790006eb3 Mon Sep 17 00:00:00 2001 From: Liudmila Molkova Date: Thu, 3 Oct 2024 17:42:55 -0700 Subject: [PATCH 16/17] Update projects/gen-ai.md Co-authored-by: Adrian Cole <64215+codefromthecrypt@users.noreply.github.com> --- projects/gen-ai.md | 1 + 1 file changed, 1 insertion(+) diff --git a/projects/gen-ai.md b/projects/gen-ai.md index d0fd9a18..495a836e 100644 --- a/projects/gen-ai.md +++ b/projects/gen-ai.md @@ -115,6 +115,7 @@ Some of the repositories (such as opentelemetry-js-contrib) provide a lot of aut | @patcher9 | JS & Python | OpenLIT | | | @trentm | JS | Elastic | | | @xrmx | Python | Elastic | | +| @codefromthecrypt | JS & Python | Elastic | 4 hours/week | *looking for more contributors* From 0b22f8bec7a336fbc7d330e13f09738a80887bad Mon Sep 17 00:00:00 2001 From: Liudmila Molkova Date: Tue, 8 Oct 2024 20:25:48 -0700 Subject: [PATCH 17/17] Add Leighton's time estimate --- projects/gen-ai.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/projects/gen-ai.md b/projects/gen-ai.md index 495a836e..be92ac40 100644 --- a/projects/gen-ai.md +++ b/projects/gen-ai.md @@ -96,7 +96,7 @@ Some of the repositories (such as opentelemetry-js-contrib) provide a lot of aut **Sponsoring Members:** - @lmolkova -- TBD +- @tedsuo **Engineers:** @@ -110,7 +110,7 @@ Some of the repositories (such as opentelemetry-js-contrib) provide a lot of aut | @gyliu513 | JS & Python | IBM | | | @karthikscale3 | JS & Python | Langtrace | | | @lmolkova | Python | Microsoft | | 16 hours/week | -| @lzchen | Python | Microsoft | | | +| @lzchen | Python | Microsoft | | 4 hours/week | | @nirga | JS & Python | Traceloop | | | @patcher9 | JS & Python | OpenLIT | | | @trentm | JS | Elastic | |