您好，请问Qwen2.5官方榜单中的评测数据集分别是使用多少shot评测的？

[13416157913]

Has this been raised before?

• I have checked the GitHub README.
• I have checked the Qwen documentation and cannot find an answer there.
• I have searched the issues and there is not a similar one.
• I confirm that this is not a bug report, a feature request, or a badcase.

Description

您好，请问Qwen2.5官方的评测榜单中所有的评测集分别是采用多少shot评测的？例如：MMLU、MMLU-Pro、GPQA、MATH、GSM8K、HumanEval、IFEval、MBPP、BBH等等

[hzhwcmhf]

For instruction-tuned models, we choose the best setting for each dataset.
Specifically,

• MMLU uses 0-shot or 5-shot;
• MMLU-Pro uses 5-shot;
• GPQA uses 0-shot;
• MATH uses 0-shot;
• GSM8k uses 0-shot;
• HumanEval uses 0-shot;
• IFEval uses 0-shot;
• MBPP uses 0-shot;

We do not report BBH score for instruction models. For base models, we use 3-shot.

[13416157913]

For base models, we use 3

Thanks a lot.

[13416157913]

• 0-shot or 5-shot;

https://arxiv.org/pdf/2407.10671
为什么Qwen2官方论文中Qwen2-72B-Instruct的math数据集使用的是4-shot，怎么Qwen2.5-72B-Instruct改成0-shot，论文中GPQA使用的是5-shot，而您提到的GPQA是0-shot，是基于什么样的原因改变的？

[xxxrzhang]

• 0-shot or 5-shot;

https://arxiv.org/pdf/2407.10671 为什么Qwen2官方论文中Qwen2-72B-Instruct的math数据集使用的是4-shot，怎么Qwen2.5-72B-Instruct改成0-shot，论文中GPQA使用的是5-shot，而您提到的GPQA是0-shot，是基于什么样的原因改变的？

hello,你的mbpp结果可以复现出来吗？我这边复现mbpp@1最高也就65，用的Qwen2.5-3B-Instruct