Configurable dashboards

[marcklingen]

Langfuse includes a lot of interesting data (dimensions and metrics). Currently Langfuse includes a number of prebuilt charts on the main dashboard.

We consider building configurable dashboards to plot (timeseries) or pivot (table) any of the data ingested into Langfuse. Creating this issue to understand overall excitement for this and collect use cases. Please share feedback/ideas :)

Dimensions

• Model
• Version/Release (Experimentation)
• Dataset Run
• User
• Use case (name of trace/observation)

not exhaustive

Metrics

• Volume
• Cost
• Usage (eg tokens)
• Latency: overall, time-to-first-token
• Streaming speed: tokens/second
• Error rates
• Scores: user feedback, model based evals

not exhaustive

[ulan-yisaev]

I was about to request a similar feature. It would be fantastic to see this implemented!

[JanEisen]

Absolutely agree!

[ulan-yisaev]

Hi Marc,

I'd like to share some ideas regarding use cases for this 'Configurable dashboards' feature:

1. User Satisfaction Metrics: A widget to track the percentage of users satisfied or unsatisfied with the LLM app interactions. The ability to define and compare different time periods (e.g., day-to-day, month-to-month) would be crucial.
2. Peak Usage Times Analysis: A feature to identify the times when the app is most frequently used, displayed as hourly granulated bar charts. Including an option to group certain hours (like 21:00 to 06:00) would be beneficial for understanding user behavior patterns.
3. Query Volume Tracking: The ability to monitor the number of inquiries received within selectable time frames. This would help in analyzing trends and peak periods of app interactions.
4. Average Messages per Session: A widget to calculate and display the average number of messages exchanged in each session. This metric would provide insights into the depth of interactions and user engagement.
5. Topic Trend Visualization: Implementing a feature to identify and visually represent frequently discussed topics, potentially through a word cloud. Including a filter for selecting specific time periods would enable targeted analysis of user interests.

IMHO, these features would greatly enhance anyone's ability to analyze and improve their LLM app's performance and user experience. Looking forward to seeing how these ideas can be integrated into the Langfuse dashboards 😊

Thanks for considering these suggestions!

[norton120]

I am looking at trend reporting solutions for dataset scores - we run an experiment suite against a dataset for each release, then need to aggregate and present that data (current plan is s3+athena+metabase for a quick/dirty evaluation trends report). It would be very cool to have this baked into Langfuse already, and makes sense as the datasets and scores already reside there