Add Embedding observation type

[marcklingen]

Langfuse tracing currently has 3 distinct observation types (docs): spans, events, generations

Currently embedding jobs can be tracked via generations (add all to input, cost tracking works as expected).

I'd consider to add an Embedding observation object

• Enables building dedicated UI to visualize embedding jobs and metrics
• Makes Langfuse easier to understand. Using a generation object to track an embedding job does not make intuitive sense.

[jasonslyvia]

It's kind of surprising this discussion has so many upvotes but no response, given that langfuse is in rather active development.

[marcklingen]

I am kind of thinking the same. Do you have any additions to the notes above? What would you like to see in Langfuse with regards to Embeddings?

[jasonslyvia]

A few things come into my mind:

• only count prompt tokens when calculating costs
• do not store the exact output embeddings even if it's passed to generation.end({ output }) (this is opionated for my own use case)

That's how I managed to trace embedding using trace and generation.

Btw, it will be super useful if langfuse playground is able to run some vector distance calculation between embeddings, to help me debug why some of my semantic searches not working as expected. If this is viable, then my second opinion is invalid.

[marcklingen]

thanks for sharing

[aiakubovich]

My dream would be to have the option to see visualizations from which clusters the embeddings we used for each trace and where those embeddings located. So, when I click on trace and go to embeddings, I would see something like this (in this example ranges of embeddings A, B, C and D were taken for that trace):

[marcklingen]

Agree, this'd be useful, thanks for sharing.

On another note, this'd be also helpful for embedding-based intent classification of regular traces (unrelated to actual embeddings used by the application traced with langfuse)

[EBazarov]

Datadog has a similar feature in their LLM Observability and I think it's pretty close to what most of us will be looking when dealing with analysis on top of embeddings.
https://www.datadoghq.com/product/llm-observability/#enhance-response-quality

[simonwh]

Embedding based intent classification of traces would be extremely helpful, similar to Datadog. Do we have an seperate idea that tracks this?