feat(llmobs): add span processor #13426

Kyle-Verhoog · 2025-05-15T05:59:19Z

Add capability to add a span processor. The processor can be used to mutate or redact sensitive data contained in inputs and outputs from LLM calls.

def my_processor(span):
    for message in span.output_messages:
        message["content"] = ""

LLMObs.enable(span_processor=my_processor)

LLMObs.register_processor(my_processor)

Public docs: DataDog/documentation#29365
Shared tests: TODO

Closes: #11179

Checklist

PR author has checked that all the criteria below are met
The PR description includes an overview of the change
The PR description articulates the motivation for the change
The change includes tests OR the PR description describes a testing strategy
The PR description notes risks associated with the change, if any
Newly-added code is easy to change
The change follows the library release note guidelines
The change includes or references documentation updates if necessary
Backport labels are set (if applicable)

Reviewer Checklist

Reviewer has checked that all the criteria below are met
Title is accurate
All changes are related to the pull request's stated goal
Avoids breaking API changes
Testing strategy adequately addresses listed risks
Newly-added code is easy to change
Release note makes sense to a user of the library
If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
Backport labels are set in a manner that is consistent with the release branch maintenance policy

github-actions · 2025-05-15T05:59:47Z

CODEOWNERS have been resolved as:

releasenotes/notes/llmobs-processor-d5cb47b12bc3bbd1.yaml               @DataDog/apm-python
ddtrace/llmobs/__init__.py                                              @DataDog/ml-observability
ddtrace/llmobs/_llmobs.py                                               @DataDog/ml-observability
ddtrace/llmobs/_telemetry.py                                            @DataDog/ml-observability
tests/llmobs/_utils.py                                                  @DataDog/ml-observability
tests/llmobs/conftest.py                                                @DataDog/ml-observability
tests/llmobs/test_llmobs.py                                             @DataDog/ml-observability

Kyle-Verhoog · 2025-05-15T06:00:26Z

tests/llmobs/test_llmobs.py

+        traces = events[0]
+        assert len(traces) == 2
+        assert "scrub_values:1" in traces[0]["spans"][0]["tags"]
+        assert traces[0]["spans"][0]["meta"]["input"]["messages"][0]["content"] == "scrubbed"


gonna follow up to make fetching the spans / traces from events less implementation dependent

tests/llmobs/_utils.py

Kyle-Verhoog · 2025-05-15T06:02:25Z

tests/llmobs/test_llmobs.py

+        env["DD_LLMOBS_AGENTLESS_ENABLED"] = "0"
+        env["DD_TRACE_ENABLED"] = "0"
+        env["DD_TRACE_AGENT_URL"] = llmobs_backend.url()
+        env["DD_TRACE_LOGGING_RATE"] = "0"


took me way too long to learn that

1 log record per name/level/pathname/lineno every 60 seconds by default

is the default logging configuration for the library 🙂

github-actions · 2025-05-15T06:19:54Z

Bootstrap import analysis

Comparison of import times between this PR and base.

Summary

The average import time from this PR is: 232 ± 3 ms.

The average import time from base is: 233 ± 2 ms.

The import time difference between this PR and base is: -0.9 ± 0.1 ms.

Import time breakdown

The following import paths have shrunk:

ddtrace.auto 1.855 ms (0.80%)

ddtrace.bootstrap.sitecustomize 1.186 ms (0.51%)

ddtrace.bootstrap.preload 1.186 ms (0.51%)

ddtrace.internal.remoteconfig.client 0.611 ms (0.26%)

ddtrace 0.669 ms (0.29%)

pr-commenter · 2025-05-15T06:49:46Z

Benchmarks

Benchmark execution time: 2025-05-15 06:49:43

Comparing candidate commit 35b25f4 in PR branch kylev/io-processor with baseline commit 83dea4c in branch main.

Found 0 performance improvements and 2 performance regressions! Performance is the same for 525 metrics, 9 unstable metrics.

scenario:iast_aspects-ospathjoin_aspect

🟥 execution_time [+871.486ns; +942.201ns] or [+14.380%; +15.546%]

scenario:iast_aspects-ospathnormcase_aspect

🟥 execution_time [+431.795ns; +503.884ns] or [+12.722%; +14.847%]

Add capability to add a span processor. The processor can be used to mutate or redact sensitive data contained in inputs and outputs from LLM calls. ```python def my_processor(span): for message in span.output_messages: message["content"] = "" LLMObs.enable(span_processor=my_processor) LLMObs.add_processor(my_processor) ```

sabrenner

the api, logic, and cases tested LGTM! nice job on telemetry as well 😎 just a couple questions, will approve after resolving them 😄

also - i think it should be register_processor instead of add_processor the PR description code block for clarity for folks who come to the PR looking at the changes

ddtrace/llmobs/_llmobs.py

sabrenner · 2025-05-15T16:00:41Z

ddtrace/llmobs/_llmobs.py

+        if llmobs_span.input_messages is not None:
+            meta["input"]["messages"] = llmobs_span.input_messages
+        if llmobs_span.output_messages is not None:
+            meta["output"]["messages"] = llmobs_span.output_messages


also just want to double check that we don't need to apply the functions on the value/document fields (or if we will but just not in scope for now).

it seems like the value field is implicitly deprecated (from what i can tell) @Yun-Kim can you confirm? for documents i think we can follow up to add them. I omitted for the sake of brevity of the PR size and scope.

yeah will let Yun chime in - we still set them here and here for non-llm spans (similarly for Node.js), although it could be the case that they are deprecated/not used in our pipelines outside of the SDKs (i would need to verify tho)

will Yun weigh in but will otherwise treat this as resolved regardless bc i think the main use case is LLM spans 👍

Yeah, the input.value/output.value fields aren't deprecated but only used on non-llm-kind spans, although our backend/UI depends on the value fields for llm spans if messages aren't provided. Since some of our integrations generate I/O info for non-LLM span kinds (crewai, langchain, langgraph, openai agents), it might be worth adding the span processing for the value fields as well (none of our integrations generate documents fields so no need to worry here about it for now)

none of our integrations generate documents fields so no need to worry here about it for now

i think langchain retrieval operations are the exception for generating output documents, and we also capture input documents for embedding spans as the input to whatever operation we trace (eg openai.embeddings.create("some text that we'll turn into a document")) - but agree that if we do need/want to add it, it can be done in a follow up.

sabrenner · 2025-05-15T16:02:02Z

ddtrace/llmobs/_llmobs.py

@@ -97,14 +102,32 @@
 }


+@dataclass
+class LLMObsSpan:


not sure if this would benefit from some typedocs here - i do feel it's pretty self-explanatory so might not be necessary, but up to you!

yeah i was midway through documenting and questioned what i would meaningfully write that isn't described by the fields. Maybe we could include an example usage? Might be worth saying that the fields are mutable 🤷‍♂️

yeah i think an example usage would be good enough for this, yeah the fields themselves are self-explanatory 😂

Kyle-Verhoog commented May 15, 2025

View reviewed changes

Kyle-Verhoog force-pushed the kylev/io-processor branch from 35b25f4 to 230b1ca Compare May 15, 2025 15:28

sabrenner reviewed May 15, 2025

View reviewed changes

sabrenner approved these changes May 15, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(llmobs): add span processor #13426

feat(llmobs): add span processor #13426

Kyle-Verhoog commented May 15, 2025 •

edited

Loading

github-actions bot commented May 15, 2025

Kyle-Verhoog May 15, 2025

Kyle-Verhoog May 15, 2025

github-actions bot commented May 15, 2025 •

edited

Loading

pr-commenter bot commented May 15, 2025

sabrenner left a comment

sabrenner May 15, 2025

Kyle-Verhoog May 15, 2025

sabrenner May 15, 2025

Yun-Kim May 15, 2025 •

edited

Loading

sabrenner May 15, 2025

sabrenner May 15, 2025

Kyle-Verhoog May 15, 2025

sabrenner May 15, 2025

feat(llmobs): add span processor #13426

Are you sure you want to change the base?

feat(llmobs): add span processor #13426

Conversation

Kyle-Verhoog commented May 15, 2025 • edited Loading

Checklist

Reviewer Checklist

github-actions bot commented May 15, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented May 15, 2025 • edited Loading

Bootstrap import analysis

Summary

Import time breakdown

pr-commenter bot commented May 15, 2025

Benchmarks

scenario:iast_aspects-ospathjoin_aspect

scenario:iast_aspects-ospathnormcase_aspect

sabrenner left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Yun-Kim May 15, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Kyle-Verhoog commented May 15, 2025 •

edited

Loading

github-actions bot commented May 15, 2025 •

edited

Loading

Yun-Kim May 15, 2025 •

edited

Loading