Proposal: Add Support for Uploading Dataset Items via UI

[vaddenz]

1. Introduction

Currently, users can only create dataset items through the API or SDK, requiring additional coding effort when dealing with large datasets. This proposal aims to introduce a user interface (UI) for uploading dataset items directly, enhancing accessibility and efficiency, particularly for non-developer users. This feature will:

• Enable non-developer users to manage large datasets independently.
• Facilitate the migration of datasets from Excel or other platforms to LangFuse, enhancing the platform’s usability.

2. Problem Statement

To enhance user experience and system performance, we need to address several challenges:

• Defining expected formats and field specifications for dataset files.
• Setting dataset size limits to avoid performance degradation.
• Deciding between synchronous and asynchronous import processes.
• Providing clear feedback during the import process, including error details.
• Handling duplicate data entries.
• Allowing field mapping from imported files to existing dataset fields.
• Ensuring the implementation is simple, reusable, and extensible.

3. Proposed Solution

3.1 Dataset File Format

Support will be added for Excel, CSV, and JSON formats. The proposed formats are:

• CSV/Excel: The first row should include field names, with each subsequent row representing one record.

  Example CSV:

  input,expected_output,metadata
  Tell me about LangFuse,LangFuse is an opensource LLM OPS platform.,{"lang":"en"}
  

• JSON: The file should be a list of records.

  Example JSON:

  [
    {
      "input": "Tell me about LangFuse",
      "expected_output": "LangFuse is an opensource LLM OPS platform.",
      "metadata": "{\"lang\":\"en\"}"
    }
  ]

3.2 Import Process and Feedback

• Asynchronous Import: Recommended for handling large datasets to avoid UI freezes and manage tasks effectively.

Synchronous import is easier to implement, but it may cause performance issue when the dataset is too large.
Asynchronous import is more suitable for large datasets, but managing the import tasks, process and error feedback
requires lots of work, and reduces the simpleness of LangFuse.

Here I recommend we implement client-side asynchronous import,
which means the file parsing and import task management is done on the browser side, and dataset items are created in
batches via a number of API calls. Besides, when importing on client side, users can easily check the import process.

• Error Handling: Allow users to download a CSV of failed items with error details for corrections and re-import.

Considering import errors/invalid data, we can let users download failed items as a CSV file to fix and re-import. For
example:

input,expected_output,metadata,error
,LangFuse is a opensource LLM OPS platform.,{"lang":"en"},Input cannot be empty

3.3 File Size Limitation

As described above, file parsing is done on client side. The file size limit should be set to a reasonable value,
e.g. 10MB, to prevent browser performance issues.

3.4 Handling Duplicates and Mapping

Initially, complex functionalities like handling duplicates and field mapping will not be implemented. However, the design will be extensible to accommodate these features in future versions.

4. Implementation Steps

1. Define supported file formats and provide downloadable templates.
2. Develop client-side parsers for Excel, CSV, and JSON.
3. Implement the asynchronous import mechanism on the client side.
4. Enable users to download error details for correction.

5. References

• Related discussion: Upload datasets via UI #1988

[vaddenz]

Hi there! Thought this would be a good point to get started to contribute to LangFuse. If you have any idea, comments are welcomed. 😄
@assafweinberg @Ami777 @marcklingen @maxdeichmann

[clemra]

Thank you for this excellent write up.