feat: Add custom Langfuse span handling support

[vblagoje]

Why:

The main motivation behind this change is to enable developers to easily customize Langfuse traces without having to rewrite the entire LangfuseConnector. By introducing the SpanHandler interface and its default implementation, DefaultSpanHandler, developers can now extend SpanHandler or, even better, override the handle method in DefaultSpanHandler to tailor trace processing to their needs. This simplifies the process of customizing what data is logged to Langfuse, providing a more flexible and developer-friendly integration.

• fixes Add SpanHandler abstraction to enable custom LLM tracing in Langfuse #1310

Enables developers to add further customization to spans as noted in these issues:

• Langfuse Integration: add support for log levels #1265
• Tracing integrations: allow for custom trace/run_ids #1264
• Langfuse Integration: expose the trace_id in the response #1263
• Extend the list of supported (chat) generators in langfuse for custom generators  #1153

What:

• Added a new SpanHandler class and its default implementation DefaultSpanHandler for customizing span creation and processing.
• Grouped all span parameters into a SpanContext dataclass.
• Updated the LangfuseConnector and LangfuseTracer to support custom span handlers.
• Modified tests to verify the new functionality and its impact on span handling.

How can it be used:

Developers can utilize custom span handlers by providing their implementations that extend the SpanHandler class:

class CustomSpanHandler(DefaultSpanHandler):
    def handle(self, span: LangfuseSpan, component_type: Optional[str]) -> None:
        # Custom span handling logic, customize Langfuse spans however it fits you
        pass

connector = LangfuseConnector(span_handler=CustomSpanHandler())

How did you test it:

Tests were conducted using the QualityCheckSpanHandler to ensure span WARNING levels are set correctly. Different response scenarios were simulated to check the handler's response, and the Langfuse API was polled to verify the output.

Notes for the reviewer:

Pay attention to the SpanHandler's implementation to ensure it meets necessary specifications and seamlessly integrates with the existing LangfuseTracer. Additionally, confirm that the new test scenarios cover diverse usage cases adequately.

[vblagoje]

@alex-stoica @ggdupont @lc-bo @LastRemote would this approach fit all of your Langfuse customization needs?

[alex-stoica]

@vblagoje I have to think deeper, but at the first glance, my answer is "partially", i.e.

1. I can create a CustomSpanHandler that would trace some prompt information about a generation ✅
2. I still need to build a langfuse prompt builder, quite similar with something proposed in Connect the generators to prompts in langfuse  #1154, that pushes the prompt name and a custom generator outputs this prompt name so that I can log it further with my current span ❌(This part means this extra component is still needed)

So, I'd say it's a solid and elegant step in the right direction, but not the full solution.

[vblagoje]

@vblagoje I have to think deeper, but at the first glance, my answer is "partially", i.e.

1. I can create a CustomSpanHandler that would trace some prompt information about a generation ✅
2. I still need to build a langfuse prompt builder, quite similar with something proposed in Connect the generators to prompts in langfuse  #1154, that pushes the prompt name and a custom generator outputs this prompt name so that I can log it further with my current span ❌(This part means this extra component is still needed)

So, I'd say it's a solid and elegant step in the right direction, but not the full solution.

@alex-stoica thanks for the feedback! What you describe seems a bit orthogonal to this issue, no? Here we enable devs to customize these spans as they see fit - that's all, good first step. This prompt injection seems something different - and that's ok.

[alex-stoica]

@vblagoje
short answer: Yes, it's independent but aligned with the same goal, just wanted to highlight this to ensure we continue making progress in that direction too
longer explanation: to address the issue I was describing, I previously had to:

1. Modify legacy code from the Haystack repository to make sure I can log extra fields from a generator - an undesirable and messy solution.
2. Develop a custom LangfusePromptBuilder to forward the prompt name, along with a CustomGenerator to output the prompt name as well.

With this new modification, step 1 is resolved (not required anymore). In terms of this specific issue, the codebase moves closer to being open/closed (SOLID principles), I now only need to extend functionality (e.g., create a custom span, prompt builder, or generator) without modifying existing code. As mentioned earlier, this is a good step forward.

[vblagoje]

@alex-stoica thanks a lot for the feedback. I added a small change to encapsulate all the parameters for span creation into SpanContext. SpanHandler.create_span had just too many parameters and looked fragile in terms of future maintenance. I'll keep this PR in draft state for this week for you and others to try it out and kick the tires - next week we can move to PR integration if there are no major changes required. 🙏

[LastRemote]

Thanks for the ping. Unfortunately it does not help with my current problems with Langfuse (which are unrelated to this PR), but I do think this is a more elegant approach and a good step forward.

To be clear, I already implemented a slightly modified version of LangfuseTracer that is able to register new generators on the fly (this part can be done in a better way by using CustomSpanHandler) and automatically translate Anthropic / Bedrock-Anthropic usage to OpenAI format.

Things that trouble me right now, for a more transparent feedback:

• LangfuseSpan.set_content_tag does some fancy translations to chat messages which allows pretty-printing in Langfuse. However this is causing all other inputs/output information to be dropped.
• Context variables have a rather complicated interaction with Langfuse when deploying the pipeline as an API service.
• Both Langfuse and Haystack have poor support for o1-like reasoning models regarding how to track and handle reasoning contents. This is a big problem and it is worth opening a new thread for this.
• When there is a runtime error, Langfuse is not capturing any information. I think this is better handled by doing some try and except clauses in haystack pipeline executions. This PR partially fixes this problem.
• [just a thought, not bugging me right now] Some components might need to store additional information in a trace, such as the intermediate steps in a complicated component. Currently there is no way to do this because the component does not have direct access to the tracing span, and we cannot handle the data if they are not captured in the first place.

[alex-stoica]

Both Langfuse and Haystack has poor support for o1-like reasoning models regarding how to track and handle reasoning contents. This is a big problem and it is worth opening a new thread for this.

Totally agree with that. Also, the trend is towards more "LRM"s, whatever they are

[LastRemote]

For reference I just created a new issue for LRMs: deepset-ai/haystack#8760

[vblagoje]

Thanks for the feedback @alex-stoica and @LastRemote. The PR is now opened and will go into review next week, with a new langfuse integration released shortly after - if all goes as planned. If you see anything else that can be adjusted now - let us know.

[anakin87]

I've taken a look and in principle I can't find anything wrong with the implementation....

I think we should mention this development in our docs. Can you take care of that?

Since I am not an expert on this part, another pair of eyes might be helpful (I would propose involving @mpangrazzi).

[anakin87]

could you please explain this change?

[vblagoje]

Thanks for asking @anakin87 . It is a mistake! StatefulClient is a superclass of all these spans and one can create other spans from StatefulClient superclass API using generation/span methods but we need to keep the previous version because we also call update on these spans which is only declared in subclasses: StatefulSpanClient, StatefulTraceClient, and StatefulGenerationClient. I guess IDE/linters didn't pick these up because of forward references. I'll make corrections. 🙏

[mpangrazzi]

I am also not a Langfuse expert too, but after looking at the why and the changes, I would say the look good to me.

I agree with @anakin87, we should definitely mention this additional feature in the docs.

[vblagoje]

@dfokina I updated https://docs.haystack.deepset.ai/docs/langfuseconnector but can't somehow get the formatting right for the code snippet. It seems like Python option is gone from readme. Please investigate.

@anakin87 please review a743f15 pydoc additions. Daria and I will fix the doc to be just right.

[anakin87]

Feel free to merge once tests are green

[vblagoje]

@anakin87 this change deserve a jump to 0.8.0 in https://pypi.org/project/langfuse-haystack/#history , thumbs up/down?

[vblagoje]

@dfokina I also noticed that we should perhaps, in our LangfuseConnector doc, describe an additional init param:

httpx_client: Optional[httpx.Client] = None,

As this parameter has also been added recently. It is a small change to docs:

:param httpx_client: Optional custom httpx.Client instance to use for Langfuse API calls. Note that when
            deserializing a pipeline from YAML, any custom client is discarded and Langfuse will create its own default
            client, since HTTPX clients cannot be serialized.

Perhaps you can mention both httpx_client and span_handler params somehow. I'll let you do that in a way you see fit.

[vblagoje]

Fixed and included in https://pypi.org/project/langfuse-haystack/0.8.0/ thank you all @alex-stoica @LastRemote @lc-bo @marcklingen @davidsbatista @anakin87 @mpangrazzi @dfokina

[dfokina]

Sorry for the delay @vblagoje. I fixed the code snippet and edited the text a bit. Will add that new param in a moment, too.