Tracing in ML-Commons #4084

chriswlai · 2025-08-09T00:23:28Z

Description

This feature adds tracing to ML-Commons. More details can be found here: https://quip-amazon.com/Nc4RA8w6XIwa/Design-Doc-ML-Commons-Agent-Tracing-Observability

Related Issues

Resolves #3971

Check List

New functionality includes testing.
New functionality has been documented.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

* adding agent tracing to mlplugin Signed-off-by: chrislai <[email protected]> * add settings and clean code Signed-off-by: chrislai <[email protected]> * add tests Signed-off-by: chrislai <[email protected]> * cr fixes Signed-off-by: chrislai <[email protected]> * tests message fix and spotlessApply Signed-off-by: chrislai <[email protected]> * spotlessApply Signed-off-by: chrislai <[email protected]> * add tests Signed-off-by: chrislai <[email protected]> * fix comments Signed-off-by: chrislai <[email protected]> * add visiblefortesting Signed-off-by: chrislai <[email protected]> * address comments Signed-off-by: chrislai <[email protected]> * javadoc Signed-off-by: chrislai <[email protected]> * more tests Signed-off-by: chrislai <[email protected]> * more class fixes Signed-off-by: chrislai <[email protected]> * overloaded method Signed-off-by: chrislai <[email protected]> --------- Signed-off-by: chrislai <[email protected]>

* adding agent tracing to mlplugin Signed-off-by: chrislai <[email protected]> * add tests Signed-off-by: chrislai <[email protected]> * cr fixes Signed-off-by: chrislai <[email protected]> * spotlessApply Signed-off-by: chrislai <[email protected]> * add tests Signed-off-by: chrislai <[email protected]> * P-E-R tracing Signed-off-by: chrislai <[email protected]> * add input/output to all spans Signed-off-by: chrislai <[email protected]> * per agent runner updates Signed-off-by: chrislai <[email protected]> * remove conversational agent changes Signed-off-by: chrislai <[email protected]> * rebase to main Signed-off-by: chrislai <[email protected]> * propagate tokens Signed-off-by: chrislai <[email protected]> * remove extractJsonBlock Signed-off-by: chrislai <[email protected]> * reconfigure classes and write coverage tests Signed-off-by: chrislai <[email protected]> * reconfig + tests Signed-off-by: chrislai <[email protected]> * address comments Signed-off-by: chrislai <[email protected]> * javadoc and avoid import* Signed-off-by: chrislai <[email protected]> * remove index mapping Signed-off-by: chrislai <[email protected]> * large refactor + testing Signed-off-by: chrislai <[email protected]> * avoid default null Signed-off-by: chrislai <[email protected]> * less null checks Signed-off-by: chrislai <[email protected]> --------- Signed-off-by: chrislai <[email protected]>

* adding agent tracing to mlplugin Signed-off-by: chrislai <[email protected]> * add tests Signed-off-by: chrislai <[email protected]> * cr fixes Signed-off-by: chrislai <[email protected]> * spotlessApply Signed-off-by: chrislai <[email protected]> * add tests Signed-off-by: chrislai <[email protected]> * P-E-R tracing Signed-off-by: chrislai <[email protected]> * add input/output to all spans Signed-off-by: chrislai <[email protected]> * conversational agent tracing Signed-off-by: chrislai <[email protected]> * allow flexability in tool return format Signed-off-by: chrislai <[email protected]> * directly access singleton + good practice coding Signed-off-by: chrislai <[email protected]> * remove null checks Signed-off-by: chrislai <[email protected]> * rebase off main Signed-off-by: chrislai <[email protected]> * spacing Signed-off-by: chrislai <[email protected]> * propagate tokens Signed-off-by: chrislai <[email protected]> * refactor and add tests Signed-off-by: chrislai <[email protected]> * fix rebase issues Signed-off-by: chrislai <[email protected]> * huge refactor Signed-off-by: chrislai <[email protected]> * clean code Signed-off-by: chrislai <[email protected]> * spotlessApply Signed-off-by: chrislai <[email protected]> * more refactoring Signed-off-by: chrislai <[email protected]> * more helper functions Signed-off-by: chrislai <[email protected]> * reformat listener Signed-off-by: chrislai <[email protected]> * major refactor + address comments Signed-off-by: chrislai <[email protected]> --------- Signed-off-by: chrislai <[email protected]>

Signed-off-by: chrislai <[email protected]>

…pensearch-project#3976) * adding agent tracing to mlplugin Signed-off-by: chrislai <[email protected]> * add tests Signed-off-by: chrislai <[email protected]> * cr fixes Signed-off-by: chrislai <[email protected]> * spotlessApply Signed-off-by: chrislai <[email protected]> * add tests Signed-off-by: chrislai <[email protected]> * P-E-R tracing Signed-off-by: chrislai <[email protected]> * add input/output to all spans Signed-off-by: chrislai <[email protected]> * conversational agent tracing Signed-off-by: chrislai <[email protected]> * allow flexability in tool return format Signed-off-by: chrislai <[email protected]> * directly access singleton + good practice coding Signed-off-by: chrislai <[email protected]> * convflow and flow agent tracing Signed-off-by: chrislai <[email protected]> * new settings and noop update Signed-off-by: chrislai <[email protected]> * overloading method Signed-off-by: chrislai <[email protected]> * propagate tokens Signed-off-by: chrislai <[email protected]> * rebase cleaning Signed-off-by: chrislai <[email protected]> * clean Signed-off-by: chrislai <[email protected]> * clean Signed-off-by: chrislai <[email protected]> * javadoc Signed-off-by: chrislai <[email protected]> * rebase to main Signed-off-by: chrislai <[email protected]> * refactoring code Signed-off-by: chrislai <[email protected]> * refactoring Signed-off-by: chrislai <[email protected]> * address comments Signed-off-by: chrislai <[email protected]> * fixed numbering Signed-off-by: chrislai <[email protected]> * address comments Signed-off-by: chrislai <[email protected]> --------- Signed-off-by: chrislai <[email protected]>

Signed-off-by: chrislai <[email protected]>

* adding agent tracing to mlplugin Signed-off-by: chrislai <[email protected]> * add settings and clean code Signed-off-by: chrislai <[email protected]> * add tests Signed-off-by: chrislai <[email protected]> * cr fixes Signed-off-by: chrislai <[email protected]> * tests message fix and spotlessApply Signed-off-by: chrislai <[email protected]> * spotlessApply Signed-off-by: chrislai <[email protected]> * add tests Signed-off-by: chrislai <[email protected]> * fix comments Signed-off-by: chrislai <[email protected]> * add visiblefortesting Signed-off-by: chrislai <[email protected]> * address comments Signed-off-by: chrislai <[email protected]> * javadoc Signed-off-by: chrislai <[email protected]> * more tests Signed-off-by: chrislai <[email protected]> * more class fixes Signed-off-by: chrislai <[email protected]> * overloaded method Signed-off-by: chrislai <[email protected]> --------- Signed-off-by: chrislai <[email protected]>

Signed-off-by: chrislai <[email protected]>

Agent Tracing Storage and Visualization Tutorial

pyek-bot · 2025-09-02T19:30:53Z

.../main/java/org/opensearch/ml/engine/algorithms/agent/MLPlanExecuteAndReflectAgentRunner.java

+                    long llmLatency = System.currentTimeMillis() - llmStartTime;
+
+                    Map<String, Integer> extractedTokens = MLAgentTracer.extractTokensFromModelOutput(modelTensorOutput, allParams);


token calculation

chriswlai and others added 30 commits August 6, 2025 16:27

tracing rebased to main

aa7952a

Signed-off-by: chrislai <[email protected]>

fix tests and use 3.1

e33c8d4

Signed-off-by: chrislai <[email protected]>

Connector and Model Tracing (opensearch-project#4004)

e9ff3cd

rebase with all tracing

718f4a9

Signed-off-by: chrislai <[email protected]>

revert to 3.2

85403d2

Signed-off-by: chrislai <[email protected]>

adding agent tracing to mlplugin

442879f

Signed-off-by: chrislai <[email protected]>

add tests

c263e19

Signed-off-by: chrislai <[email protected]>

cr fixes

2c39700

Signed-off-by: chrislai <[email protected]>

spotlessApply

c34f040

Signed-off-by: chrislai <[email protected]>

add tests

548fb1a

Signed-off-by: chrislai <[email protected]>

P-E-R tracing

19c3c62

Signed-off-by: chrislai <[email protected]>

conversational agent tracing

02cecfc

Signed-off-by: chrislai <[email protected]>

storage and visualization tutorial

5019a64

Signed-off-by: chrislai <[email protected]>

clean updates

beb3fab

Signed-off-by: chrislai <[email protected]>

fixes

2515b92

Signed-off-by: chrislai <[email protected]>

rebase

12693f3

Signed-off-by: chrislai <[email protected]>

remove extra edits

bfe49f9

Signed-off-by: chrislai <[email protected]>

update docs

ce54f8c

Signed-off-by: chrislai <[email protected]>

add index mapping to tutorial

7b0eb61

Signed-off-by: chrislai <[email protected]>

updating tutorial

166f85a

Signed-off-by: chrislai <[email protected]>

improve tutorial

fff0c0b

Signed-off-by: chrislai <[email protected]>

add sec disable

4b494cc

Signed-off-by: chrislai <[email protected]>

consistent versions

842a432

Signed-off-by: chrislai <[email protected]>

add otel install for non-docker

cc41762

Signed-off-by: chrislai <[email protected]>

address mroe comments

37eac94

Signed-off-by: chrislai <[email protected]>

chriswlai and others added 2 commits August 8, 2025 17:15

update

c60fba1

Signed-off-by: chrislai <[email protected]>

Merge pull request #1 from chriswlai/feature/store-visual

21237b0

Agent Tracing Storage and Visualization Tutorial

chriswlai had a problem deploying to ml-commons-cicd-env-require-approval August 9, 2025 00:24 — with GitHub Actions Failure

pyek-bot mentioned this pull request Aug 11, 2025

Agent Tracing Storage and Visualization Tutorial #3974

Closed

5 tasks

pyek-bot reviewed Sep 2, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Tracing in ML-Commons #4084

Tracing in ML-Commons #4084

Uh oh!

chriswlai commented Aug 9, 2025 •

edited

Loading

Uh oh!

pyek-bot Sep 2, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		long llmLatency = System.currentTimeMillis() - llmStartTime;

		Map<String, Integer> extractedTokens = MLAgentTracer.extractTokensFromModelOutput(modelTensorOutput, allParams);

Tracing in ML-Commons #4084

Are you sure you want to change the base?

Tracing in ML-Commons #4084

Uh oh!

Conversation

chriswlai commented Aug 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

pyek-bot Sep 2, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

chriswlai commented Aug 9, 2025 •

edited

Loading