feat(observability): Refactor content observation mechanism to support both logging and tracing #3612

traburiss · 2025-06-19T12:56:55Z

This commit refactors the content observation mechanism by adding support for trace recording without altering the original logging functionality. The motivation for this change is to ensure complete context availability in scenarios such as integration with Langfuse, which is essential for proper functionality.

Main changes:

New configuration options:
1. trace-prompt: Enables recording prompts to trace
2. trace-completion : Enables recording completions to trace
3. trace-prompt-size : Limits the length of prompt context
4. content-formatter : Handles content formatting, currently supporting both 'text' and 'langfuse' modes
Added two handlers, ChatModelCompletionObservationTraceHandler and ChatModelPromptContentObservationTraceHandler, to support recording context to trace.
Introduced the MessageFormatter interface and its subclasses to support formatting for prompt and completion content.
Rolled back parts of the code and dependencies from commit ca843e8.

…t both logging and tracing This commit refactors the content observation mechanism by adding support for trace recording without altering the original logging functionality. The motivation for this change is to ensure complete context availability in scenarios such as integration with Langfuse, which is essential for proper functionality. Main changes: 1. New configuration options: 1. `trace-prompt`: Enables recording prompts to trace 2. `trace-completion` : Enables recording completions to trace 3. `trace-prompt-size` : Limits the length of prompt context 4. `content-formatter` : Handles content formatting, currently supporting both 'text' and 'langfuse' modes 2. Added two handlers, ChatModelCompletionObservationTraceHandler and ChatModelPromptContentObservationTraceHandler, to support recording context to trace. 3. Introduced the MessageFormatter interface and its subclasses to support formatting for prompt and completion content. 4. Rolled back parts of the code and dependencies from commit ca843e8. Signed-off-by: tingchuan.li <[email protected]>

ilayaperumalg · 2025-06-24T15:52:07Z

@jonatan-ivanov Could you take a look a this PR? Thanks!

shakuzen · 2025-06-25T01:38:39Z

It would help in reviewing this if you could share a link to the relevant parts of Langfuse's documentation that specifies what they're expecting for the desired integration. How likely is it to change? This is reintroducing using the deprecated API on OTel Spans to add Event Attributes, which was removed because it's deprecated and the suggested alternative was Log Events.

traburiss · 2025-06-25T12:59:06Z

@shakuzen

https://langfuse.com/docs/integrations/spring-ai

Langfuse doc still uses include-prompt and include-completion, but these parameters are now deprecated. However, the new parameters log-prompt and log-completion cannot work with Langfuse.
Langfuse need both prompt and completion data included within spans.

jonatan-ivanov · 2025-06-25T23:31:47Z

...pringframework/ai/model/chat/observation/autoconfigure/ChatObservationAutoConfiguration.java

+				name = "chatModelPromptContentObservationTraceHandler")
+		@ConditionalOnProperty(prefix = ChatObservationProperties.CONFIG_PREFIX, name = "trace-prompt",
+				havingValue = "true")
+		TracingAwareLoggingObservationHandler<ChatModelObservationContext> chatModelPromptContentObservationTraceHandler(


Why are these wrapped into TracingAwareLoggingObservationHandler?
Please check what it does, it is used to make the tracing data available for log entries (log correlation). If I understand correctly, this is not needed in this use-case.

Yes, the wrapping of TracingAwareLoggingObservationHandler should be removed.

jonatan-ivanov · 2025-06-25T23:36:02Z

spring-ai-commons/pom.xml

@@ -61,7 +61,7 @@

 		<dependency>
 			<groupId>io.micrometer</groupId>
-			<artifactId>micrometer-tracing</artifactId>
+			<artifactId>micrometer-tracing-bridge-otel</artifactId>


spring-ai-commons should not depend on a bridge (implementation instead of the abstraction).

jonatan-ivanov · 2025-06-25T23:38:17Z

spring-ai-commons/src/main/java/org/springframework/ai/observation/tracing/TracingHelper.java

+	}
+
+	@Nullable
+	public static Span extractOtelSpan(@Nullable TracingObservationHandler.TracingContext tracingContext) {


I think we should depend on the abstraction (tracing API) instead of a single implementation (OTel API/SDK).

jonatan-ivanov · 2025-06-25T23:38:50Z

spring-ai-model/pom.xml

@@ -60,7 +60,7 @@

 		<dependency>
 			<groupId>io.micrometer</groupId>
-			<artifactId>micrometer-tracing</artifactId>
+			<artifactId>micrometer-tracing-bridge-otel</artifactId>


spring-ai-model should not depend on a bridge (implementation instead of the abstraction).

jonatan-ivanov · 2025-06-25T23:42:15Z

...java/org/springframework/ai/chat/observation/ChatModelCompletionObservationTraceHandler.java

+			.getRequired(TracingObservationHandler.TracingContext.class);
+		Span currentSpan = TracingHelper.extractOtelSpan(tracingContext);
+		if (currentSpan != null) {
+			currentSpan.addEvent(AiObservationEventNames.CONTENT_COMPLETION.value(),


It seems this API will be deprecated (maybe already is?), see: micrometer-metrics/micrometer#5238 (comment)

I wonder if it would be possible to write directly to Span.Tag.

like

tracingContext.getSpan().tagOfStrings(AiObservationAttributes.CONTENT_COMPLETION.value(), completion);

As I’ve tested, Langfuse properly captures the data when using this approach.

jonatan-ivanov · 2025-06-25T23:46:03Z

Langfuse doc still uses include-prompt and include-completion, but these parameters are now deprecated. However, the new parameters log-prompt and log-completion cannot work with Langfuse.

This makes me a bit confused. Why are we doing this if one is already deprecated and another is not supported?

markpollack · 2025-06-26T16:06:57Z

Basically my view is that our observability needs to support products like langfuse. How to achieve this goal.

wrt to this PR

Rolled back parts of the code and dependencies from commit ca843e8.

that change in behavior can't occur in 1.0.x - and not sure about the strategy for 1.1 on this front. Hoping to use this issue to drive a strategy.

I thought there is flexibility to override behavior - we can perhaps create ready to use repos for products that customize observability in our https://github.com/orgs/spring-ai-community/repositories org

@ThomasVitale suggestions on how to structure the convo and go forward - a working google doc would seem to be a good start.

markpollack · 2025-06-27T14:18:42Z

Also related discussion is here - #3151 (comment)

traburiss · 2025-06-30T08:24:46Z

Basically my view is that our observability needs to support products like langfuse. How to achieve this goal.

wrt to this PR

Rolled back parts of the code and dependencies from commit ca843e8.

that change in behavior can't occur in 1.0.x - and not sure about the strategy for 1.1 on this front. Hoping to use this issue to drive a strategy.

I thought there is flexibility to override behavior - we can perhaps create ready to use repos for products that customize observability in our https://github.com/orgs/spring-ai-community/repositories org

@ThomasVitale suggestions on how to structure the convo and go forward - a working google doc would seem to be a good start.

@markpollack

In my opinion, if the intention is to prevent large amount of data within span, constraints should be imposed through parameters to limit the length/byte size rather than directly removing these two fields from the span.

shakuzen · 2025-07-01T05:00:34Z

Given the lack of stability around what to send in what format to what observability backend, I think it would be most maintainable and usable to make separate modules that customize the obserability data in the way that a specific use case needs for well-known use cases, such as Langfuse. I think this is what @markpollack suggested in his previous comment. From the user's perspective this looks like adding a dependency on a Spring AI module when they want to ship data in the way Langfuse expects. From a maintainability perspective, it means we don't need to keep making changes to core modules when things change in Langfuse. This should all be possible with auto-configuring different ObservationHandler implementations, which is what this PR does. In fact, any user can configure an implementation of ObservationHandler that does this or anything else. So it's already possible for users to achieve what this PR is doing without changes in Spring AI, right? I agree it would be best if it were easier to achieve, and that's the reason for a module that does what's needed that people can use.

I suspect there are other integrations out there for AI observability and what they expect may be different than Langfuse and may change at different times and in different ways than other integrations. Trying to make the observability workable with everything from core modules sounds like a losing proposition. If there were standardization and stability around this, it would be feasible to do this from common modules, but it seems weren't not there yet. Perhaps an alternative is to make the separate module based around OTel semantic conventions for Gen AI and put the onus on observability backends to support the latest version of that (I would suspect there's some pressure for this anyway).

That said, my reading of the linked documentation to Langfuse is that they intend to support what Spring AI produces, so we may not need to try so hard to send observability data in the way we used to just because that is what Langfuse supported (because that's what we produced... notice the chicken and egg problem).

traburiss · 2025-07-01T06:54:45Z

I agree with @shakuzen suggested in his previous comment, it is a better solution..

In fact, this is exactly how I've implemented it in my production code
I developed a dedicated autoconfiguration and starter, which provide additional configuration options and a custom handler to support Langfuse

I’ll create an issue in the spring-ai-community repository at https://github.com/orgs/spring-ai-community/repositories.

traburiss force-pushed the feature/content-trace branch from b212a6f to c485721 Compare June 19, 2025 13:12

traburiss force-pushed the feature/content-trace branch from c485721 to ef2ad4d Compare June 19, 2025 13:18

jonatan-ivanov reviewed Jun 25, 2025

View reviewed changes

traburiss added 2 commits June 27, 2025 22:36

Merge branch 'spring-projects:main' into feature/content-trace

ca88bc9

Merge branch 'spring-projects:main' into feature/content-trace

807de80

feat(observability): Refactor content observation mechanism to support both logging and tracing #3612

Are you sure you want to change the base?

feat(observability): Refactor content observation mechanism to support both logging and tracing #3612

Uh oh!

Conversation

traburiss commented Jun 19, 2025

Uh oh!

ilayaperumalg commented Jun 24, 2025

Uh oh!

shakuzen commented Jun 25, 2025

Uh oh!

traburiss commented Jun 25, 2025

Uh oh!

jonatan-ivanov Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

traburiss Jun 30, 2025

Choose a reason for hiding this comment

Uh oh!

jonatan-ivanov Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

jonatan-ivanov Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

traburiss Jun 30, 2025

Choose a reason for hiding this comment

Uh oh!

jonatan-ivanov Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

jonatan-ivanov Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

traburiss Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jonatan-ivanov commented Jun 25, 2025

Uh oh!

markpollack commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

markpollack commented Jun 27, 2025

Uh oh!

traburiss commented Jun 30, 2025

Uh oh!

shakuzen commented Jul 1, 2025

Uh oh!

traburiss commented Jul 1, 2025

Uh oh!

Uh oh!

traburiss Jun 30, 2025 •

edited

Loading

markpollack commented Jun 26, 2025 •

edited

Loading