Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

How to evaluate models on my own custom MCQA dataset? #24

Open
xinformatics opened this issue Sep 10, 2024 · 1 comment
Open

How to evaluate models on my own custom MCQA dataset? #24

xinformatics opened this issue Sep 10, 2024 · 1 comment

Comments

@xinformatics
Copy link

Thank you so much for your lightweight repo. Could you please guide me with evaluating models on a custom field specific dataset which I have created?

Data Format: There are around 1000 questions with 4 choices each and only one is correct.

Thank you
@Hothan01
Copy link
Collaborator

OK, thank you for your feedback. It seems that your data format is quite similar to MMLU. Therefore, you can refer to MMLU's data processing file make_dataset.py and its configuration settings.

If you have any further questions, I look forward to hearing from you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@xinformatics @Hothan01