Take chat history in consideration when running a prompt experiment

[benjtlv]

I am trying to test the output of my agent but after a certain point in the discussion with the user. For example, I ask my agent to ask a bunch of questions and then try to schedule an appointment with the user.
I Want to create a dataset item that contains the chat history, so that when I run the test I can actually get the expected output
So my question is, is there a standardized way to do it in langfuse, or should I include chat_history as an additional variables in my prompt?

[dosubot[bot]]

Hey @benjtlv! 👋 I'm here to help you with any bugs, questions, or contributions you have. Let me know how I can assist you!

To include chat history in a dataset item for testing purposes in Langfuse, you can use the create_dataset_item function. This allows you to specify the input and expected output, including the chat history as part of the input. Here's an example of how you can structure it:

langfuse.create_dataset_item(
    dataset_name="<dataset_name>",
    input={
        "chat_history": [
            {"role": "user", "content": "Hello, how are you?"},
            {"role": "assistant", "content": "I'm good, thank you! How can I assist you today?"}
        ]
    },
    expected_output={
        "response": "I'm good, thank you! How can I assist you today?"
    },
    metadata={
        "model": "llama3",
    }
)

This approach allows you to include the chat history as part of the input, which can be useful for testing the output of your agent at a specific point in the conversation ^[1].

To continue talking to Dosu, mention @dosu.

[marcklingen]

Great question and related to history placeholder in prompt management: https://github.com/orgs/langfuse/discussions/2210

For now, I'd suggest to add a {{history}} placeholder in your prompt template to add the message history via a variable in the dataset item input

[j10sanders]

Is that possible to implement in the UI for prompt experiments?

[j10sanders]

And to expand on that, something like this did not work for me (since some dataset items only have one user message):

Can you expand on how the chat_history variable would be used?

[benjtlv]

@marcklingen thank you for your response. But what about the response from dosubot? Good enough or misleading?

[marcklingen]

It is correct as well as it suggests to store the history on chat_history variable within the input. You will need to include the chat_history variable in the prompt as well.

[j10sanders]

Can you show an example of how this would be implemented in the UI?

[marcklingen]

Currently this would insert the history as a json object within for example a user message. This is a workaround and we plan to add chat_history/placeholder support more natively (see below)

[marcklingen]

I've added this thread to track adding support for these "special variables" more natively: https://github.com/orgs/langfuse/discussions/4454