Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Eval details #14

Open
ZhaoShuang1995 opened this issue Oct 19, 2023 · 1 comment
Open

Eval details #14

ZhaoShuang1995 opened this issue Oct 19, 2023 · 1 comment
Labels
disc-law-eval-benchmark Related to DISC-Law-Eval-Benchmark evaluation framework

Comments

@ZhaoShuang1995
Copy link

关于评测的话，可以提供一下具体的评测方式嘛？
例如objective中的单选题，few-shot setting是如何设置的呢？是找四条样例数据拼在当前问题之前嘛？
以及对于模型的回答，只需要包括选项的字母就算正确，还是说需要把选项中的文字也都包括了才算正确呀？
对于subjective的问题，可以提供一下gpt3.5的评测prompt模版嘛？
非常感谢~
@Charlie-XIAO
Copy link
Collaborator

We are working on making DISC-Law-Eval benchmark open source, including the evaluation dataset and the scripts, but they are not ready now. Sorry about that.

@Charlie-XIAO Charlie-XIAO added the disc-law-eval-benchmark Related to DISC-Law-Eval-Benchmark evaluation framework label Oct 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
disc-law-eval-benchmark Related to DISC-Law-Eval-Benchmark evaluation framework
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Charlie-XIAO @ZhaoShuang1995