LLM calls captured via Langchain integration should match what’s sent to the LLM

[benm5678]

The current input field shown in trace hides/changes some of the data structure that is actually passed to LLM. It would be nice to have option to see the raw input sent (including functions/tools params):

For example, in Langfuse I see:

12: {
content: ""
example: false
additional_kwargs: {
function_call: {
name: "forwardUserToSupportPortalLink"
arguments: {
friendlyMessage: "Please follow this link."
}
}
}
}
13: {
name: "forwardUserToSupportPortalLink"
content: "{"Please follow this link."
}

But in HTTP trace, we see this in the messages param (so we can see it sent role and slightly different structure to show the function call):

{
    "role": "assistant",
    "content": null,
    "function_call": {
        "arguments": "{\n  \"friendlyMessage\": \"Please follow this link.\",
        "name": "forwardUserToSupportPortalLink"
    }
},
{
    "role": "function",
    "content": "{\"Please follow this link.\",
    "name": "forwardUserToSupportPortalLink"
}

[marcklingen]

Do you use the OpenAI integration for Python?

[benm5678]

Yes...I have latest v2.71. It's hooked up this way in the code:

callbacks.append(langfuse.callback.CallbackHandler(
              host=self.LANGFUSE_HOST,
              public_key=self.LANGFUSE_PUBLIC_KEY,
              secret_key=self.LANGFUSE_SECRET_KEY
              )

[marcklingen]

This looks like the langchain Python integration as you use the callback handler.

Agree that I’d be nice to transform the values the provided by the langchain callback back to the message sent to the LLM. Definitely open to support PRs on this.

[marcklingen]

Updated the title to reflect this

[jjczopek]

@benm5678 @marcklingen
Correct me if I'm wrong, but it might also be similar to something that I have spotted - when looking at OpenAI traces in Langfuse where call to LLM contained information about the functions (function definitions) sent to the model - it's impossible to find those in the the trace - so I imagine it's not being sent.

And it would be nice to see those.

[benm5678]

Those are also very useful; they are sent via the 'functions' or 'tools' params in the open ai calls -- I resort to utilizing httpdbg currently to see the raw HTTP call in case I need some deeper troubleshooting or validation in how my functions are being sent to the LLM. I'm not sure how feasible it would be to integrate the raw HTTP call in each Langfuse logged entry.... there might be a more efficient approach, or perhaps it can be an optional switch.

[marcklingen]

agree, this should be tracked. can you share a public url to an example trace?

[jjczopek]

I created two, one with openai and second with langchain from examples, around the chatting with sample chinook sqlite database.

https://cloud.langfuse.com/project/cltwnqmwh0004h323w13gsnhb/traces/4c36af21-07ba-4d59-99c3-c5c76afc88ed?observation=f09d14e5-2c2b-427b-bd09-7917b333f2eb
https://cloud.langfuse.com/project/cltwnqmwh0004h323w13gsnhb/traces/3f1aa3e6-0781-4112-8366-1517a3538330

I'm trying to understand how we could extract the functions definitions in the callback handler, but I need some time spin up some tests.

[jjczopek]

@benm5678 @marcklingen I did look around for a while and it seems to be all over the place. Depending if it's a chain on LCEL runnable, function information can be found on serialized.kwargs.llm_kwargs or serialized.kwargs. And it seems to be available (at least from my current observations) on the on_chain_start callback function.

I might try just looking for those two places and extracting the functions from there. Do you have any advice how to hook it up eventually with the generation? Because from what I can see, due to the fact functions are available on the on_chain_start, functions might end up on span not on generation, unless I'm mistaken.