Webhook to trigger CI/CD pipeline when new prompt version is created

[tamdoEH]

Describe the feature or potential improvement

Webhooks could be a great add-on that would attract more users. Whenever a prompt is deployed to production, it will trigger a CI/CD pipeline.

Additional information

No response

[marcklingen]

I like this idea as it is a creative use of webhooks. It could for example power the following workflow which helps overcome common limitations of playgrounds:

1. Use langfuse datasets to store a testing dataset
2. Create a new prompt version
3. Trigger a runner in CI via a Webhook that runs the application (pulling the latest prompt) in a loop on the dataset and pushed scores to langfuse
4. See evaluation metrics for prompt versions as langfuse dataset runs