Run prompt experiments via the SDKs/API

[hspak]

Describe the feature or potential improvement

If the prompt experiment creation was exposed via the SDK, it could make automated eval runs pretty straightforward.

I was thinking something along the lines of:

1. Datasets and evals are pre-configured.
2. The prompt experiment could be programmatically kicked off (e.g by CI) to proactively catch regressions.

Let me know if there's an alternate approach I should consider as well.

Additional information

No response

[marcklingen]

Thanks for sharing! There is no near-term plan to extend the in-ui prompt experiments to the SDKs as these kinds of tests can be run via the Langfuse SDK as well in a few lines of code (happy to help figure this out if you have questions): https://langfuse.com/docs/datasets/get-started

Prompt Experiments are a way of doing everything in the UI, it is basically a fully-managed experiment run that would otherwise need to be created via the SDK.