Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

evaluation: evaluate answer quality by scoring #376

Open
2 of 3 tasks
sykp241095 opened this issue Nov 13, 2024 · 1 comment
Open
2 of 3 tasks

evaluation: evaluate answer quality by scoring #376

sykp241095 opened this issue Nov 13, 2024 · 1 comment
Assignees

Comments

@sykp241095
Copy link
Member

sykp241095 commented Nov 13, 2024

讲问题分类标准如下:

  1. 📘 Level-1 基础知识: 查询 TiDB 的简单事实或常识,如配置参数、组件设计等。
  2. 🛠️ Level-2 操作指南:想执行某项操作但不知如何下手,寻求操作手册。
  3. ⚠️ Level-3 问题排查:遇到错误消息或意外结果,想找到原因并解决问题。
  4. 🗂️ Level-4 复杂任务计划: 设定一个需要多步骤的复杂目标,寻求完整的计划或指导。
  5. 💬 Level-5 其他话题:提出与 TiDB 或数据库无关的问题,或想讨论一般性话题。

We need to:

  • Prepare 50 questions for evaluation( 10 for each level) from asktug.com, which is a real questions source.
  • Create a tool or solution(or implement it in pingcap/autoflow
  • and then embedd it in /admin page) to run it manually.(attrs: name, csv(query+human answer), chat engine, run(size: int) )
@sykp241095 sykp241095 added this to the Release v0.3.1 milestone Dec 4, 2024
@hey-kong
Copy link

hey-kong commented Dec 5, 2024

TiDB_Questions.csv
Hi, I've prepared 50 questions from asktug.com.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants