Auto-patching Langchain to include Langfuse callback handler

[kongzii]

Describe the feature or potential improvement

I have been integrating Langfuse into our agents lately, and it's quite cumbersome to always initialize and include Langfuse's callback before calling Langchain's invoke method:

@observe()
def func():
  config =  {"callbacks":[CallbackHandler()]}
  ...
  llm.invoke(..., config=config)

It's even more cumbersome if someone needs to have both observed and non-observed versions of the function:

def func(add_langfuse: bool):
  config =  {"callbacks":[CallbackHandler()]} if add_langfuse else None
  ...
  llm.invoke(..., config=config)

@observe()
def func_observed():
  return func(add_langfuse=True)

As an inspiration, we can take MLFlow, which is quite popular, they provide an auto_log feature that will "magically" include mlflow logging in supported libraries: https://mlflow.org/docs/latest/tracking/autolog.html

I think something similar shouldn't be too hard to achieve, I can see two approaches:

1. Only @observed function are monkey-patched with auto-tracking, if autolog is enabled
2. The whole codebase is monkey-patched, if autolog is enabled

I think (1) is way better, but I'm not sure it would be possible, but still (2) would be great too.

If this sounds reasonable to you as well, I'm open to kicking off the first implementation.

We could start with Langchain (as that's the library I'm working most atm), and then add support for others if all goes well.

Additional information

No response

[hassiebp]

Hi @kongzii - thanks a lot for opening this discussion!

Let me first understand the problem we are trying to solve here: reducing the required instrumentation code for each Langchain. Before discussing monkey patching the Langchain module, one could abstract away the callback handler init into a function that is called at config=get_langchain_config().

Also, one could try to push for a global callback handler in the Langchain repos similar as in LlamaIndex.

In the past, we have mixed experiences with monkey patching modules such as openai as in Python the module is patched at the global scope, which may lead to unexpected behavior when other third party modules require the patched module.

How would you trade off the cost / benefits here? What could I be missing here?

[kongzii]

Hi, thanks for the reply. Agreed that monkey-patching is definitely tricky. I did something similar to config=get_langchain_config(), and that simplifies the code in our repository for sure.

However, the issue I'm also facing is that there are more and more third-party libraries that use LLM in the background as well. With monkey-patching, at least the basics would be tracked from them (costs, prompts, completions), which I think would often be enough; right now, it's needed to do a PR and add support for Langfuse manually.

I guess the question to answer is, should every library have built-in support for Langfuse (by library I don't mean only big libraries as Langchain, but also small ones that use Langchain/OpenAI/something to execute any LLM stuff), or should Langfuse provide some universal monkey patch for OpenAI (and others) to enable at least basic tracking everywhere?

(In the first post, I proposed auto-patching of Langchain, but now as I was thinking about it, it might be more universal and useful to patch the LLM provider library, but anyway, that's a secondary discussion).

[hassiebp]

Got it, thanks @kongzii ! Since we are already monkey-patching the OpenAI module when running from langfuse.openai import openai, one could simply execute that line at the top of a application that uses a third-party library which in turn uses OpenAI to get basic tracking in Langfuse.

We will monitor this Github Idea here to see whether further monkey-patch cases are requested with popular support :)