truncated input for base64 string inputs for generation tracing

[JTZ18]

i am using langfuse==2.53.9
I am trying to track the generation of the llm to get accurate input and output cost token calculation for multimodal inputs like images and text. It works well when i use the from langfuse.openai import AzureOpenAI. But using the low level SDK results in truncation

when i use the low level sdk and use

  generation = trace.generation(
      name="answer_query",
      model=AZURE_OPENAI_DEPLOYMENT_NAME,
      model_parameters={
          "temperature": TEMPERATURE,
          "max_tokens": MAX_TOKENS
      },
      start_time=start_time,
      end_time=end_time,
      input=[{'role': "system", "content": system_prompt},
             {'role': "user", "content": [{"type": "text", "text": user_prompt}]},
             {'role': "user", "content": [
                 {"type": "image_url",
                  "image_url": {
                      "url": f"data:image/jpeg;base64,{img}"
                  }} for img in context_images
             ]}
             ],
      output=response.choices[0].message.content
  )
  generation.end()

i get the following result below in this image, where input tokens are truncated and incorrectly calculated due to truncation

when i use the from langfuse.openai import AzureOpenAI, the inputs are properly handled and not truncated

I want to make it work in the low level SDK, any ideas how i could avoid truncation for trace.generation?

[jannikmaierhoefer]

Hi @JTZ18, could you try to upgrade to the latest version of our SDK (2.55.0) as we released full multi-modal support last week.

[marcklingen]

Can you share a deep link to the traces here? Would love to have a look

[JTZ18]

Can you share a deep link to the traces here? Would love to have a look

I tried it on langfuse cloud and the multimodal base64 images are showing up perfectly.

I apologise if i did not share earlier that im trying to get multimodal to show up at least in my input. i dont mind the images not showing up, but auto tracing the generation using langfuse.openai should at least not truncate my input system and text prompts.

My self hosted version is running on docker compose langfuse v2.93.4 OSS

[hassiebp]

@JTZ18 You mentioned above that all works well with the langfuse.openai integration and you see failure when using the low-level SDK only?

This sentence got me confused then, could you please summarize for which cases you see which failures?

i dont mind the images not showing up, but auto tracing the generation using langfuse.openai should at least not truncate my input system and text prompts.

My understanding so far:
Langfuse Cloud - OK
Self-hosted with langfuse.openai integration - OK
Self-hosted with low-level SDK - FAIL

All using the Python SDK version 2.55.0

[JTZ18]

@JTZ18 You mentioned above that all works well with the langfuse.openai integration and you see failure when using the low-level SDK only?

This sentence got me confused then, could you please summarize for which cases you see which failures?

i dont mind the images not showing up, but auto tracing the generation using langfuse.openai should at least not truncate my input system and text prompts.

My understanding so far: Langfuse Cloud - OK Self-hosted with langfuse.openai integration - OK Self-hosted with low-level SDK - FAIL

All using the Python SDK version 2.55.0

Apologies, let me summarise what works and what doesnt.
Langfuse Cloud - OK (SDK 2.55.0)
Self-hosted with langfuse.openai integration - FAIL (SDK 2.55.0)
Self-hosted with langfuse.openai integration - OK (SDK 2.53.9)

Self-hosted with low-level SDK - FAIL (SDK 2.53.9)
Self-hosted with low-level SDK - FAIL (SDK 2.55.0)

For the fail cases, when theres base64 images in my messages, my input will be truncated

[hassiebp]

If all works well with the 2.55.0 SDK on Langfuse Cloud I'm suprised about the truncation messages your are seeing with the same SDK on the self-hosted instances. All truncation / media management happens SDK side. Could you please verify you are indeed running 2.55.0 against your self hosted instances, and could you please share the exact truncation messages you are seeing in your self-hosted instances? (Assuming you have also verified that all works correct on Langfuse Cloud for 2.55.0)