Support for more modalities and base64 content

[marliessophie]

Describe the feature or potential improvement

After our latest release supporting images (passed via url) in Langfuse, we're looking to support other modalities soon.

We're committed to supporting base64 images in the near term. We're further planning support for audio, video, and attachments (e.g. PDFs).

We'd love to gauge interest across the modalities to help with most efficient prioritisation.

[chai3]

+1 base64 image (most wanted)
+1 audio
+1 attachments(pdf, xlsx, html)

I dont need to support video in my use-case

[badbye]

Great! The dataset also need to support Multi-Modal input. Looking forward!

[Alpacolypse]

Eagerly awaiting base64 support, especially in markdown

[clemra]

A few double clicks on base64 as reactions to our most recent newsletter

[Hallicopter]

base64 images for sure.

[bioshazard]

+1 b64 images. Seems like the very next thing to support. Schema seems well established at this point.

[bioshazard]

Update: So I actually wound up writing a span intercept to strip b64 data and replace with filename. I plan on having a batch operation and worried it would balloon my DB. While looking at supporting other modalities, consider a way to decouple attachment reference so it can maybe resolve from a single source than to risk duplicating into the traces.

[tcbonds]

Base64 images for multimodal models is the missing feature that is holding us back from using Langfuse. PDF attachments would also be very helpful.

[pleomax0730]

@tcbonds What alternative solution are you currently using? Our team is planning to add an AI observability tool, and your experience and insights would be greatly appreciated.

[tcbonds]

Arize Phoenix. So far we have had a pretty good experience with it.

[profikid]

Bas64 or image url would be great to support

[wbreilly]

Audio is mission critical for us

[zxkxyz]

just here to add my support for base64 (because claude)

[marcklingen]

this is on the near-term roadmap, @hassiebp is working on it and specs look great, can't wait for the release!

[pleomax0730]

Here to support Audio, PDF source.

[darylsew]

+1 base64 (because claude)

[alexdanilowicz]

+1 base64 image (because claude)

[hassiebp]

Good news - we have just launched multimodal support in beta! 🎨

• Changelog post
• Docs
• JS SDK release
• Python SDK release

Please let us know your feedback here or create an issue if you encounter any bugs. Thanks! 🙏🏾

[marcklingen]

huge! looking forward to the feedback

[bioshazard]

DUDE YES! Y'all decoupled it to a bucket!! This is HUGE!! LANGFUSE_S3_MEDIA_UPLOAD_, now I can undo my customer span processor I was using to prevent the b64 from going into my DB.

[recursingfeynman]

vLLM multi-modal format, please!

Here is an example from the vLLM docs:

{
    "type": "audio_url",
    "audio_url": {
        "url": audio_url
    },
}

It would also be great to add support for .ogg and .oga audios.

[marcklingen]

Thanks for sharing! I've created an issue for this. Open for contributions!
#4406

[pleomax0730]

I've encountered an issue. Please have a look.

https://github.com/orgs/langfuse/discussions/4363

[marcklingen]

thanks, already created an issue for this: #4410

[nitin302]

Does this support Anthropic models via AWS Bedrock using langchain or is there a better way?

from langchain_aws import ChatBedrockConverse

[nitin302]

@marcklingen i was able to get this working, it was an issue with my environment variable required for S3 upload. Quick question on the UI that renders the images... Is there any limits? For example, we have a single LLM call that send 10 images ... and we are not able to see it render inline, instead it shows up in the media section as attachments.

[nitin302]

however, with single images, it does render inline.

[marcklingen]

This might be the expected behavior. Can you share example traces?

cc @hassiebp

[nitin302]

you can just consider https://langfuse.com/guides/cookbook/example_multi_modal_traces#langchain-image-input but with multiple images...

[hassiebp]

@nitin302 Feel free to open a separate issue with a minimum repro example and langchain version here please: https://github.com/langfuse/langfuse/issues/new/choose