Support OpenTelemetry-based instrumentation (openllmetry and others)

[gyliu513]

Describe the feature or potential improvement

openllmetry can provide many instrumentation code for vectordb, LLMs, LLM orchestration platforms etc, and traceloop sdk is kind of otel collector and can expose metrics, logs(did not implement yet), tracing to a third party observability platform. Langfuse is a good candidate to act as a 3rd party LLM observability platform.

Additional information

No response

[marcklingen]

Thanks for sharing! I think generally supporting the forming opentelemetry conventions (genai/llm) is super interesting as they mature. There are currently a number of instrumentation implementations with slightly different conventions as far as I understand the notes of the working group(s). I'd love to understand your perspective on this!

I think adding an OTel collector to the langfuse platform seems the most promising way to support OTel-based instrumentation as long as the formats/semantics across instrumentation libraries are starting to be standardized/stable enough.

[gyliu513]

I think adding an OTel collector to the langfuse platform seems the most promising way to support OTel-based instrumentation as long as the formats/semantics across instrumentation libraries are starting to be standardized/stable enough.

@marcklingen Yes, OTel collector will be the best option for this. I was wondering maybe even before the formats/semantics across instrumentation libraries become mature, you can probably still able to build some OTel collectors to integrate with langfuse, as the formats/semantics across instrumentation libraries did not impact much for the implementation. Embracing OTEL can also help enlarge the ecosystem for langfuse as well.

There are currently a number of instrumentation implementations with slightly different conventions as far as I understand the notes of the working group(s). I'd love to understand your perspective on this!

Yes, but the OTEL community is trying to unify this via the LLM working group, there is already a initial draft spec for LLM semantic convention at https://github.com/open-telemetry/semantic-conventions/tree/main/docs/gen-ai

[marcklingen]

I was wondering maybe even before the formats/semantics across instrumentation libraries become mature, you can probably still able to build some OTel collectors to integrate with langfuse, as the formats/semantics across instrumentation libraries did not impact much for the implementation

I am unsure about this one as

• we will not be able to support all Langfuse features while they are not yet covered by the conventions
• depending on the speed of the development of the conventions, once we add an initial otel collector (with basic features), there will probably be great interest to support vendor-specific implementation aspects while the standards still develop. This leads to potential maintenance overhead and breaking backwards compatibility once conventions are in place for everyone who relied on the 3rd party vendor instrumentation.

Yes, but the OTEL community is trying to unify this via the LLM working group, there is already a initial draft spec for LLM semantic convention at https://github.com/open-telemetry/semantic-conventions/tree/main/docs/gen-ai

I'll try to join the working group to keep in the loop with these changes, very interested in this. I am especially excited about this proposal: open-telemetry/community#2326

[gyliu513]

Thanks @marcklingen , I totally understand your concern for backward compatibility.

we will not be able to support all Langfuse features while they are not yet covered by the conventions
depending on the speed of the development of the conventions, once we add an initial otel collector (with basic features), there will probably be great interest to support vendor-specific implementation aspects while the standards still develop. This leads to potential maintenance overhead and breaking backwards compatibility once conventions are in place for everyone who relied on the 3rd party vendor instrumentation.

Let me share some of my understanding, usually, if a product did not reach GA milestone, the product usually in Beta or Alpha state, and customer can have a quick try and share some feedback. Due to the product is Beta or Alpha, and it will not support running in production, but just PoC. When reaching GA, the customer may need to allow some break changes. But anyway, glad to see you are interested in OTEL AI Semantic Convention, look forward to work with you in this area 👍

[marcklingen]

Fair, we might ship something like this as a quick first POC at one point to get feedback and collaborate with everyone, labeling it as beta. However, getting to GA quickly was something we focused on with all of our other integrations as it was valued by many teams using Langfuse. In this case, we need to make clear that things can still break while the conventions change.

[marcklingen]

btw, awesome to see you being interested to contribute to the new otel instrumentation libraries

[gyliu513]

Fair, we might ship something like this as a quick first POC at one point to get feedback and collaborate with everyone

I'm excited for this, look forward to see this :)

[mjrusso]

Just adding a note here that OTel support would also enable Langfuse to support Magentic (cc @jackmpcollins) and any library that is itself instrumented with Pydantic Logfire. For Magentic in particular, see details here: https://magentic.dev/logging-and-tracing/

[marcklingen]

thanks!

[baggiponte]

logfire is superb, would be awesome to get the integration (also given that pydantic is ubiquitous).

[marcklingen]

OTel support would also allow to integrate with Firebase GenKit, see instrumentation docs and thread: https://github.com/orgs/langfuse/discussions/3351

cc @debkanchan

[marcklingen]

Merging thread #2043 by @baggiponte

openllmetry by Traceloop allows instrumenting your LLM applications and send data to any collector such as splunk, datadog, signoz and so on. It's a standardisation effort that's becoming quite popular and successful and I think langfuse would benefit from supporting that.
This would mean that most applications already using openllmetry/opentelemetry could seamlessly send data to langfuse (or a different product, like the APM posthog). This could tie langfuse closer to the telemetry ecosystem. It would also reduce the duplication effort of instrumenting the same app for two different collectors.

[johnson-liang]

Maybe merge this issue too?
#199

[marcklingen]

makes sense, let's merge all threads on this here. thank you!

[marcklingen]

Merging #199

pamelafox commented on Mar 5
Did you all make a decision regarding OTel integration? My team at Microsoft is curious (since Azure Monitor brings in OTel traces).

@marcklingen marcklingen commented on Mar 5 •
Currently we see otel to be promising for some of our users (just monitor/log production traffic), whereas it does not work for others (also want to quickly iterate locally / in batches / in notebooks). Thus, we cannot yet shift all of our tracing/ingestion to OTel.

The best way forward here is most likely to build an OTel compatible receiver/processor as an alternative way to ingest data into langfuse for those who want it for their production deployments.

As generally building integrations for Langfuse is very simple, we are very open to collab on this, right now, there are higher impact features to ship for most of our users especially with a focus on evaluation and testing of LLM applications.

If you have a view on this, happy to chat with you and your team!

@pamelafox
pamelafox commented on Mar 5
Okay, that makes sense. I don't have a strong opinion currently, I'll look out for feedback from our Microsoft customers.

@sudivate
sudivate commented on Mar 8 •
Hi @marcklingen, it's great to know that you're thinking about integrating OTEL into Langfuse.

A new Working Group (WG) was formed recently as part of the OpenTelemetry Semantic Conventions project, with a specific focus on LLM applications and frameworks.

If you're interested, I'd like to extend an invitation for you to join our community meeting. You can share any insights you have from your experience with OTEL adoption and provide feedback on the conventions we're developing for this project.

LLM Semantic Conventions Working Goup
OTEL Semantic Conventions Sigs Index
Slack Channel

@drewby @pamelafox

@gingters
gingters commented on Apr 9
I would also love to see support of OpenTelemetry ingestion.
My specific reason would be to enable observablity of Semantic Kernel based applications in langfuse. It would be awesome if we could see the single steps of SK applications in the same way we can currently see langchain-based application traces.

@benjamincudi
benjamincudi commented on Jul 11
I'd also love to see support in some kind for this - in particular, I'd love to be able to incorporate (or leverage) Langfuse tracing as part of broader application tracing, and send those traces through an OpenTelemetry Collector rather than directly to LANGFUSE_HOST

Is this something that has been tested or attempted previously with any success?

@athewsey
athewsey commented on Aug 19 •
Also interested in this feature - my (customers') use-case would be similar to the above: Integrate Langfuse's traces with those from other microservices/apps in AWS X-Ray, for end-to-end debugging & visibility.

@johnson-liang
johnson-liang commented on Sep 13
When it comes to Langfuse ingesting OpenTelemetry, I think it is more straightforward to implement an OpenTelemetry exporter that calls Langfuse SDK for each OpenTelemetry trace.

[cubxxw]

I am wondering whether the semantic standards and compatibility of OTel have reached a level of maturity to meet practical needs. Considering the potential differences between various implementations, can this standardization progress provide sufficient stability and consistency to support a wide range of use cases?

[codefromthecrypt]

I'm a member of the community, but not yet a spec approver or lead. Here's my 2p for what it's worth:

• for observing basic chat completions and embeddings traces, metrics and logs, there's some value from system POV (think dashboards that count tokens).
• there's a cavaet in logs as many eval products do not accept otel's log format, yet.
• specs have focused so far on LLM access, while agentic use or evaluations are a work in progress
• things could be driven more closely to end requirements with more participation not just in issues, but also in zoom meetings, so please help if you can.
• the spec will call itself experimental for the foreseeable future, but that doesn't mean people won't use it.

I think LLMOps is tricky as many people think of observability just like distributed systems with certain types of metrics etc, and think the problem is solved, while folks on the MLOps side may see lack of quality metrics, or specs around training pipelines, etc and feel it isn't.

Hope I was fair and balanced in above. In summary, folks working on the spec anchor what ends up there. With more folks saying "what is practical", it is possible to have specs achieve that.