Feature Request: Add Tagging Pipelines for Evaluation-Based Tagging

[wesleyearlstander]

Describe the feature or potential improvement

Summary

This proposal is to add tagging pipelines to Langfuse, enabling automated tagging of runs based on evaluation results. This functionality will streamline workflows by allowing users to add tags programmatically when evaluations meet specific criteria, enhancing the visibility and management of critical cases such as failed generations.

Use Case

The primary use case is automating the tagging of generations based on evaluation outcomes. For instance:

• Scenario: A generation is evaluated, but the quality score is absent or below a defined threshold because the output was incomplete or invalid.
• Desired Behavior: Automatically tag the run as failed.
• Follow-Up Action: These failed runs can then be filtered and pushed to an annotation queue for manual review or correction.

By incorporating tagging pipelines, we can:

• Quickly identify problematic generations: Tags like failed, low-quality, etc., provide instant filtering and visualization.
• Improve annotation workflows: Failed runs tagged programmatically can be automatically queued for annotation or flagged for further analysis.

Proposed Workflow

1. Users define a tagging pipeline in the Langfuse configuration or API.
2. The pipeline evaluates runs based on specified conditions (e.g., evaluation scores or other metrics).
3. When conditions are met, Langfuse automatically applies the defined tag(s).
4. (Optional) A follow-up action is triggered, such as moving the tagged runs to an annotation queue.

Expected Benefits

• Streamlined filtering and debugging: Easily filter for problematic runs using tags.
• Enhanced annotation workflows: Push tagged runs to annotation queues automatically, reducing manual overhead.
• Consistency in tagging: Automating this ensures uniform criteria for tagging runs.

Implementation Ideas

• Configurable Tagging Pipelines: Add a configuration layer where users can define tagging logic based on evaluations.
• Inline Tagging Rules: Allow users to attach tagging logic directly to evaluations when logged.
• Integration with Actions: Extend the functionality to trigger actions (e.g., pushing to queues) upon tagging.

Open Questions

• What conditions or operators should be supported (e.g., score thresholds, missing data)?
• Should tagging pipelines support multiple conditions (e.g., logical AND/OR for complex rules)?
• How should annotations integrate into tagging workflows (e.g., via direct APIs, custom actions, or UI elements)?

Additional information

No response