Overview issue: Leaderboard 2.0 release

[KennethEnevoldsen]

•  Leaderboard: Add zero-shot filter to leaderboard #1372
•  Leaderboard: Add runtime x performance plot #1396
• Ensure old results are present in the new leaderboard (Fetch models results from HF results#43)
• Review of a public demo of the leaderboard
• Add new results Addresults results#40
• Leaderboard 2.0: should we fetch data from model card? #1373
• Improve leaderboard 2.0 readability #1317 (Solved with hotfixes, waiting for Gradio)
• Leaderboard: Fix code benchmarks #1439

Once everything is ready I will push the preprint of the paper and we can make a coordinated public release.

[isaac-chung]

Added #1317 and #1373 as well. Feel free to remove / reclassify them if they are nice-to-haves instead of need-to-haves.

[x-tabdeveloping]

We now have a demo of the new leaderboard. Feel free to either open new issues as they come along or conduct discussions in this thread

[SaitejaUtpala]

new leaderboard seems to be down.

[x-tabdeveloping]

I believe it's due to an issue in Arctic models' metadata, I'll have a look

[x-tabdeveloping]

I have found the source of the problem, working on a fix rn.

[x-tabdeveloping]

Fixed

[gaohongkui]

Can leaderboard 2.0 add the ability to toggle main_score ? ~~

[KennethEnevoldsen]

@gaohongkui note sure what you mean?

[gaohongkui]

@gaohongkui note sure what you mean?

I mean, if the metric of task_type can be switched at a certain position on the page,
and then the leaderboard is sorted according to the new metric (for example, using recall@10 or ndcg@10 for Retrieval may result in different sorting results).

This is convenient for observing the model effect from more angles.

This might be considered a feature request. Should I discuss it here or open a separate issue?

[KennethEnevoldsen]

Ahh gotcha, feel free to open a seperate issue on this. Probably won't be in the first release of the leaderboard though

[KennethEnevoldsen]

Overview of what is missing

We opened the following issues during review:

•  Running models before leaderboard release #1411
• Leaderboard 2.0: Missing results #1571
•  leaderboard 2.0: Some task <> benchmark associations seem wrong #1514
•  leaderboard v2.0: disagreement in scores between v2.0 and v1.0 #1516
• MIRACLRetrieval results are missing for most models. #1550
• Track contamination in new leaderboard #1556
• leaderboard v2 - add autoupdate #1605

Nice to have, but not need to have:

•  Remove nan scores on leaderboard #1509
•  Confusion re: Retrieval w/Instructions #1013
•  Leaderboard: Add ONNX and GGUF to frameworks to allow us to filter based on these  #1622
•  leaderboard 2.0: Performance on subtasks (Bright retrieval)  #1517