feat(llmobs): track prompt caching for anthropic sdk #13757
Conversation
|
|
![datadog-datadog-prod-us1[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
Bootstrap import analysisComparison of import times between this PR and base. SummaryThe average import time from this PR is: 288 ± 3 ms. The average import time from base is: 290 ± 3 ms. The import time difference between this PR and base is: -1.9 ± 0.1 ms. Import time breakdownThe following import paths have shrunk:
|
BenchmarksBenchmark execution time: 2025-06-27 21:26:38 Comparing candidate commit 277d942 in PR branch Found 0 performance improvements and 3 performance regressions! Performance is the same for 558 metrics, 3 unstable metrics. scenario:iastaspects-replace_aspect
scenario:iastaspectsospath-ospathsplit_aspect
scenario:iastaspectssplit-split_aspect
|
…/dd-trace-py into evan.li/anthropic-prompt-caching
| --- | ||
| features: | ||
| - | | ||
| LLM Observability: This introduces the ability to track the number of tokens read and written to the cache for Anthropic prompt caching. |
There was a problem hiding this comment.
| LLM Observability: This introduces the ability to track the number of tokens read and written to the cache for Anthropic prompt caching. | |
| LLM Observability: Introduces capturing cache input read/write token counts for Anthropic prompt caching use cases. |
There was a problem hiding this comment.
how about " This introduces capturing the number of input tokens read and written to the cache for Anthropic prompt caching use cases."
am hesistant ab "cache input read/write token counts", think it is a bit confusing
| "cache_write_input_tokens": 0, | ||
| "cache_read_input_tokens": 0, |
There was a problem hiding this comment.
From your implementation, aren't we only adding these metrics if they exist? Should we be expecting these to always be present?
There was a problem hiding this comment.
this test contains
{"type": "text", "text": "only respond in all caps", "cache_control": {"type": "ephemeral"}},
which causes anthropic to return cache read/write token metrics as 0
since other tests don't have the cache control arg passed in, the metrics don't exist
| "cache_write_input_tokens": 2055, # Cache write tokens | ||
| "cache_read_input_tokens": 0, # Should be 0 for first request |
There was a problem hiding this comment.
I think the comments are unnecessary
| "cache_write_input_tokens": 0, # Should be 0 for cache read | ||
| "cache_read_input_tokens": 2055, # Cache read tokens |
| "cache_write_input_tokens": 1031, # Cache write tokens | ||
| "cache_read_input_tokens": 0, # Should be 0 for first request |
| "cache_write_input_tokens": 0, # Should be 0 for cache read | ||
| "cache_read_input_tokens": 1031, # Cache read tokens |
Tracks number of tokens read from and written to the prompt cache for anthropic
https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching
anthropic returns
cache_creation/read_input_tokensin their usage field.We map these to
cache_write/read_input_tokenskeys in ourmetricsfield.Testing is blocked on DataDog/dd-apm-test-agent#217
implementation note
Right now, we are using
get_llmobs_metrics_tagsto set metrics for anthropic, which depends on usingset_metricandget_metric. We do not want to continue this pattern for prompt caching, so we instead directly extract it out fromresponse.usagefield.The caveat is that for the streamed case, the
usagefield is a dictionary that is manually constructed by us when parsing out streamed chunksFollow ups
llmobs_eventsfixturemetricsparsing from set/get metrics completelyChecklist
Reviewer Checklist