Skip to content

Issues: mlcommons/modelbench

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

217 Open 179 Closed
217 Open 179 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Delete old tests chore Thing that needs doing
#855 opened Feb 7, 2025 by bkorycki
SUT Options are defined at test-level chore Thing that needs doing
#854 opened Feb 7, 2025 by bkorycki
Rip out multi-turn support chore Thing that needs doing
#853 opened Feb 7, 2025 by bkorycki
play with MLFlow
#852 opened Feb 7, 2025 by wpietri
play with DVC
#851 opened Feb 7, 2025 by wpietri
GOAL: some analytics tools milestone A goal that other cards add up to
#847 opened Feb 6, 2025 by wpietri
Murder static site generator
#844 opened Feb 5, 2025 by rogthefrog
1
run consistency checker on FR results
#843 opened Feb 5, 2025 by wpietri
1
Dynamic SUTs
#842 opened Feb 4, 2025 by wpietri
Make the Mistral SUT run reliably
#841 opened Feb 4, 2025 by wpietri
Do a practice benchmark on Deepseek R1
#839 opened Feb 4, 2025 by rogthefrog
@rogthefrog
French runs: pull results for vendors chore Thing that needs doing
#837 opened Feb 1, 2025 by rogthefrog
@rogthefrog
3
Standardize scoring (% safe vs % unsafe)
#833 opened Jan 30, 2025 by rogthefrog
Improve external resource provisioning
#832 opened Jan 30, 2025 by rogthefrog
GOAL: 1.0 Hindi benchmarks milestone A goal that other cards add up to
#831 opened Jan 30, 2025 by wpietri
GOAL: 1.0 Chinese benchmarks milestone A goal that other cards add up to
#830 opened Jan 30, 2025 by wpietri
Run 1.0 French benchmarks
#829 opened Jan 30, 2025 by wpietri
Add hazard info to journal
#823 opened Jan 29, 2025 by bkorycki
calibrate French for new reference model selection
#822 opened Jan 29, 2025 by wpietri
GOAL: 1.0 French Benchmarks milestone A goal that other cards add up to
#820 opened Jan 27, 2025 by wpietri
Goal: French 1.0 benchmarks milestone A goal that other cards add up to
#819 opened Jan 27, 2025 by wpietri
Add evaluator type to hazard UIDs
#816 opened Jan 24, 2025 by bkorycki
modelbench should not accept degenerate or surprising option combinations
#814 opened Jan 23, 2025 by rogthefrog
@rogthefrog
1
Try to replicate production failures using Docker shenanigans
#811 opened Jan 22, 2025 by wpietri
French calibration - full prompt set
#810 opened Jan 22, 2025 by rogthefrog
@rogthefrog
2
ProTip! Mix and match filters to narrow down what you’re looking for.