Add "image_url" to Langfuse Prompt types.

[grobruegge]

Describe the feature or potential improvement

Prompts within Langfuse currently support the types "system", "user" and "assistant". While this is sufficient within the context of pure language generation, modern "language" models such as "GPT-4o" are multi-model and also support images. Within the Python SDK, images can be passed to the model using type=image_url within messages as role user.

Langfuse does not natively support image urls within their prompt management, which requires some boilerplate code. For example, adding an image url in Python will look as follows:

compiled_prompt = langfuse_prompt.compile(
    attribute1=somevalue, attribute2=someothervalue
)

response = openai_client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {
            "role": "system",
            "content": compiled_prompt[0]["content"],
        },
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": compiled_prompt[1]["content"],  # user prompt
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": some_url,
                        "detail": "low",
                    },
                },
            ],
        },
    ],
)

This adds quiet a lot of code. It would be nice if one could simple add the image_url during the compile step which would then look as follows (only a suggestion):

compiled_prompt = langfuse_prompt.compile(
    attribute1=somevalue, attribute2=someothervalue, img_url=someurl
)
response = openai_client.chat.completions.create(
    model="gpt-4o",
    messages=compiled_prompt
)

Thank you so much for this framework and I am exited to see this in future versions of Langfuse!

Additional information

No response

[marliessophie]

Hi @grobruegge, thanks so much for providing this feedback.

Agree, the DX here should be improved. We'll look into this in the next few weeks to provide a simpler solution. For now, your implementation is the way to go. We'll keep you posted here when we make the corresponding changes 👍

[squarebridges]

Just following up on this as I've come across a similar use case.

Any plans to support media (images, PDFs, anything that can be base64 encoded) as part of prompts?

[marcklingen]

on the roadmap, thanks for adding your +1 to the conversation!

[zachkirsch]

+1 as well! thanks langfuse team!