Allow using User-defined models in Playground & Evals

[Chinoman10]

Describe the feature or potential improvement

Currently both Playground and Evals require defining custom models under the LLM API keys feature (which doesn't support the other features under Tracing > Models, such as pricing, tokenizer settings, etc.).

This Feature Request is to ask for both PG & Evals to be able to use the models defined elsewhere (Tracing > Models).

This is somewhat related/similar to: #4726
But different in how the feature would work. #4726 would allow us to add more custom models under the LLM API keys settings, but we'd still be missing the pricing, tokenizer and other options.

So this suggestion instead is to simply define the models in Tracing and let those be used in PG and the Evals features.

Additional information

https://github.com/orgs/langfuse/discussions/4726

[marcklingen]

Thanks for sharing! These functionalities are currently quite separate. Models can be traced without API keys in Langfuse, and Langfuse supports regex matching for tracing, which does not apply to PG and evals.

Let's keep this feedback here to see if others are interested as well!