Add new integration: LLM Proxy

[marcklingen]

Current situation

Currently the Client SDKs (js, python) or native integrations allow to instrument an application to asynchronously send traces to Langfuse.

While most integrations were built to support nested traces, the current OpenAI integration makes it very easy to capture individual LLM calls (generations) by changing the import.

Proxy

Alternatively, LLM calls could be logged via a proxy as pioneered by Helicone and adapted by Cloudflare (AI Gateway).

client = OpenAI(
+  base_url="https://proxy.langfuse.com/openai",
)

Considerations

This form of integration goes into the same direction of the current OpenAI integration (very easy to get started, no support for non-LLM spans/events) but comes with some pros and cons

Pro:

• Very simple integration to capture LLM calls for all frameworks/SDKs that support changing the inference endpoint
• Support additional capabilities that are only possible when in the critical path: model routing, blocking, rate-limiting, caching

Con:

• Langfuse moves into the critical path
• Effort to build/support this as adding support for additional models gets way more complicated when Langfuse becomes a proxy

At this point I think this should not be a priority as most users get a lot of value from using deeply nested traces. Looking forward to thoughts/ideas on this.

[Manouchehri]

We've found that LiteLLM + Langfuse is a really great fit. We've got both running as a container on Google Cloud Run. It's super cool to have an OpenAI API endpoint that works with across multiple Azure OpenAI regions (load balanced), and automatically logs to Langfuse. (There's other features, that's just the thing I love the most.)

https://docs.litellm.ai/docs/proxy/logging#logging-proxy-inputoutput---langfuse

https://docs.litellm.ai/docs/observability/langfuse_integration

cc @krrishdholakia and @ishaan-jaff if you haven't already spoken with them. They're already solving the cons for us/you. =)

[marcklingen]

That's awesome and great feedback on this request. I am a big fan of LiteLLM and the team.