Proxy Middleware to integrate Langfuse with Openrouter

[nikunjce]

Hi,

I have created new proxy middleware in asp net core application and trying to communicate langfuse with openrouter.
I have created custom model by using this proxy middleware and langfuse allowed me to add this custom model in their UI.
After that I have tried to call openrouter API through proxy middleware by using langfuse Playground but It didn't show me the output in "output" window.
API called successfully and I logged the result in middleware and I got below result.
{"id":"gen-1734173250-CJGLiH08NER7WmI2FYa0","object":"chat.completion","created":1734173250,"model":"openai/gpt-4o-mini","choices":[{"index":0,"message":{"role":"assistant","content":"The capital of Italy is Rome."},"finish_reason":"stop"}],"usage":{"prompt_tokens":10,"completion_tokens":7,"total_tokens":17}}

Now my question is why Langfuse didn't show this response in their output window.
Should I need to change anything in the response structure?

Thanks.

[dosubot[bot]]

Hey @nikunjce!👋 I'm here to help you with bugs, questions, and becoming a contributor. Let's tackle this issue together!

To display the output in Langfuse's "output" window, the response needs to adhere to a specific JSON schema. The expected structure is a JSON object with the following properties:

• steps: An array of objects, each containing:
  • explanation: A string explaining the step.
  • output: A string representing the output of the step.
• final_answer: A string representing the final answer.

Here's an example of the expected response structure:

{
  "steps": [
    {
      "explanation": "The assistant provides the capital of Italy.",
      "output": "The capital of Italy is Rome."
    }
  ],
  "final_answer": "The capital of Italy is Rome."
}

You need to transform your API response to match this structure for Langfuse to display it correctly in the "output" window ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[nikunjce]

Thanks for your reply.
First of all I would like to tell you that I need to pass few extra parameters with your requested parameters to openrouter so I have created custom middleware.

Currently when I add custom model directly by using Openrouter URL then it is working fine without any issue But when I add custom model by using custom middleware URL then Langfuse UI didn't show output in output window. So I would like to know that should I need to pass anything in request so Openrouter will return those parameters in response OR I need to change Openrouter response and need to add your suggested parameters in it.
Are you changing openrouter response in your code when you are calling its API?

Thanks.

[dosubot[bot]]

When using a custom middleware with Langfuse, you need to ensure that the response from Openrouter matches the expected structure for Langfuse to display it correctly. Langfuse does not automatically transform the Openrouter response; you need to handle this transformation in your middleware.

If the direct integration with Openrouter works without issues, it suggests that the response structure is already compatible when using the direct URL. However, when using a custom middleware, you must ensure that the response returned to Langfuse matches the expected structure.

Langfuse expects specific parameters in the response for successful display, such as id, object, created, model, choices, and usage ^[1]. If your middleware modifies the response, make sure it retains these parameters and their structure.

You should adjust your middleware to transform the response from Openrouter to include any additional parameters you need while maintaining the required structure for Langfuse. This way, Langfuse can process and display the response correctly in the UI.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[nikunjce]

Can you please provide me the link or description of the langfuse supported response or Please provide me the example for it so I ca create structured response which will be supported by the Langfuse.

[dosubot[bot]]

The expected JSON response structure for Langfuse to display outputs correctly in the UI includes several fields. Here's an example of the structure you should use:

{
  "id": "generation-id",
  "name": "chat-completion",
  "input": {
    "prompt": "Hello, how are you?"
  },
  "output": {
    "completion": "I'm fine, thank you!"
  },
  "metadata": {
    "attempt": 1
  },
  "version": "1.0",
  "model": "gpt-3.5-turbo",
  "modelParameters": {
    "temperature": 0.9,
    "maxTokens": 2000
  },
  "usage": {
    "promptTokens": 5,
    "completionTokens": 6,
    "totalTokens": 11
  }
}

This structure includes fields like id, name, input, output, metadata, version, model, modelParameters, and usage. Ensure your middleware transforms the response to match this structure for Langfuse to display it correctly in the UI ^[1][2].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[nikunjce]

I have used same structure that you had suggested but it still didn't show output in output window.

[nikunjce]

Thanks for your help. I think now app is working fine.

[marcklingen]

Hi @nikunjce,

Thanks for the ping via other channels that this is unresolved.

Generally, Langfuse supports the openai api schema as it is implemented by the openai api. Under the hood the Langchain JS Openai Class is used here.

When setting up a new model, Langfuse tests whether it works by requesting a test completion. Afterwards the model can be used in the UI.

First of all I would like to tell you that I need to pass few extra parameters with your requested parameters to openrouter so I have created custom middleware.

Is this a general problem when using openrouter or specific to your setup/implementation?

[nikunjce]

Is this a general problem when using openrouter or specific to your setup/implementation?
This is our requirement.

Thanks for your help. I think there was an issue in settings or something else. I don't know exactly what was the problem but now everything is working fine.