feat(llmobs): add span processor

[Kyle-Verhoog]

Add capability to add a span processor. The processor can be used to mutate or redact sensitive data contained in inputs and outputs from LLM calls.

from ddtrace.llmobs import LLMObsSpan

def my_processor(span: LLMObsSpan):
    for message in span.output:
        message["content"] = ""

LLMObs.enable(span_processor=my_processor)

LLMObs.register_processor(my_processor)

Public docs: DataDog/documentation#29365
Shared tests: TODO

Closes: #11179

Checklist

• PR author has checked that all the criteria below are met
• The PR description includes an overview of the change
• The PR description articulates the motivation for the change
• The change includes tests OR the PR description describes a testing strategy
• The PR description notes risks associated with the change, if any
• Newly-added code is easy to change
• The change follows the library release note guidelines
• The change includes or references documentation updates if necessary
• Backport labels are set (if applicable)

Reviewer Checklist

• Reviewer has checked that all the criteria below are met
• Title is accurate
• All changes are related to the pull request's stated goal
• Avoids breaking API changes
• Testing strategy adequately addresses listed risks
• Newly-added code is easy to change
• Release note makes sense to a user of the library
• If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
• Backport labels are set in a manner that is consistent with the release branch maintenance policy

[github-actions]

CODEOWNERS have been resolved as:

releasenotes/notes/llmobs-processor-d5cb47b12bc3bbd1.yaml               @DataDog/apm-python
ddtrace/llmobs/__init__.py                                              @DataDog/ml-observability
ddtrace/llmobs/_llmobs.py                                               @DataDog/ml-observability
ddtrace/llmobs/_telemetry.py                                            @DataDog/ml-observability
tests/llmobs/_utils.py                                                  @DataDog/ml-observability
tests/llmobs/conftest.py                                                @DataDog/ml-observability
tests/llmobs/test_llmobs.py                                             @DataDog/ml-observability
tests/llmobs/test_llmobs_service.py                                     @DataDog/ml-observability

[github-actions]

Bootstrap import analysis

Comparison of import times between this PR and base.

Summary

The average import time from this PR is: 229 ± 2 ms.

The average import time from base is: 231 ± 1 ms.

The import time difference between this PR and base is: -2.37 ± 0.07 ms.

Import time breakdown

The following import paths have shrunk:

ddtrace.auto 2.041 ms (0.89%)

ddtrace.bootstrap.sitecustomize 1.340 ms (0.59%)

ddtrace.bootstrap.preload 1.340 ms (0.59%)

ddtrace.internal.remoteconfig.client 0.655 ms (0.29%)

ddtrace 0.701 ms (0.31%)

ddtrace._logger 0.028 ms (0.01%)

logging 0.028 ms (0.01%)

traceback 0.028 ms (0.01%)

contextlib 0.028 ms (0.01%)

[pr-commenter]

Benchmarks

Benchmark execution time: 2025-05-22 04:10:11

Comparing candidate commit 50f059f in PR branch kylev/io-processor with baseline commit 8ee6868 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 507 metrics, 5 unstable metrics.

[sabrenner]

the api, logic, and cases tested LGTM! nice job on telemetry as well 😎 just a couple questions, will approve after resolving them 😄

also - i think it should be register_processor instead of add_processor the PR description code block for clarity for folks who come to the PR looking at the changes

[sabrenner]

lgtm, thanks for changing the field names, i think it unblocks us from adding the other i/o types down the road!