Zero-shot default

[Muennighoff]

I worry that the Zero-shot default of Allow Unknown in the leaderboard will incentivize people not to share/talk about their training data if it includes any benchmark data. I think maybe just doing Allow all but making the cross a bit clearer? Can we somehow make it red like ❌

[x-tabdeveloping]

As it is now, we use a UTF-8 emoji, and there is no cross that renders red in Gradio :(

I have also tried a markdown field with emoji names between colons (:cross:) but it doesn't work in the table.

I agree with the incentivization part, but I would prefer not giving up on the default filter. I have heard some people have even stronger opinions about it. It's an easy fix though if we collectively decide that something else is a better move.

[KennethEnevoldsen]

We could do❓: Not known and ⚠️: Not zero.-shot

[x-tabdeveloping]

I'd prefer the cross, but we can change it if that's the vibe

[Muennighoff]

Noting that the Open LLM leaderboard also allows contaminated models by default (https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/); Maybe we do a community vote on this?

[Muennighoff]

The leaderboard also looks much more alive with Allow all as it is way more models 🤔