[Bug] 部分样本上opencompass的评测方法是否可能有点片面？

[laogonggong847]

Prerequisite

• I have searched Issues and Discussions but cannot get the expected help.
• The bug has not been fixed in the latest version.

Type

I'm evaluating with the officially supported tasks/models/datasets.

Environment

opencompass: 0.3.9

Reproduces the problem - code/configuration sample

我在使用opencompass评估模型性能时遇到了两个问题:
(1) 在GSM8K这个数据集上我定位到了它的后处理方法

@TEXT_POSTPROCESSORS.register_module('gsm8k')
def gsm8k_postprocess(text: str) -> str:
    text = text.split('Question:')[0]
    numbers = re.findall(r'\-?\d+\.\d+|\-?\d+', text)
    if not numbers:
        return 'NULL'
    return numbers[-1]

但是发现我SFT之后的QWen2.5_7B的模型输出结果是（拿其中一条数据集举例子）:

    "2": {
        "origin_prompt": [
            {
                "role": "HUMAN",
                "prompt": "Question: Angelo and Melanie want to plan how many hours over the next week they should study together for their test next week. They have 2 chapters of their textbook to study and 4 worksheets to memorize. They figure out that they should dedicate 3 hours to each chapter of their textbook and 1.5 hours for each worksheet. If they plan to study no more than 4 hours each day, how many days should they plan to study total over the next week if they take a 10-minute break every hour, include 3 10-minute snack breaks each day, and 30 minutes for lunch each day?\nLet's think step by step\nAnswer:\n"
            },
            {
                "role": "BOT",
                "prompt": "Angelo and Melanie think they should dedicate 3 hours to each of the 2 chapters, 3 hours x 2 chapters = 6 hours total.\nFor the worksheets they plan to dedicate 1.5 hours for each worksheet, 1.5 hours x 4 worksheets = 6 hours total.\nAngelo and Melanie need to start with planning 12 hours to study, at 4 hours a day, 12 / 4 = 3 days.\nHowever, they need to include time for breaks and lunch. Every hour they want to include a 10-minute break, so 12 total hours x 10 minutes = 120 extra minutes for breaks.\nThey also want to include 3 10-minute snack breaks, 3 x 10 minutes = 30 minutes.\nAnd they want to include 30 minutes for lunch each day, so 120 minutes for breaks + 30 minutes for snack breaks + 30 minutes for lunch = 180 minutes, or 180 / 60 minutes per hour = 3 extra hours.\nSo Angelo and Melanie want to plan 12 hours to study + 3 hours of breaks = 15 hours total.\nThey want to study no more than 4 hours each day, 15 hours / 4 hours each day = 3.75\nThey will need to plan to study 4 days to allow for all the time they need.\nThe answer is 4\n"
            },
            {
                "role": "HUMAN",
                "prompt": "Question: Mark's basketball team scores 25 2 pointers, 8 3 pointers and 10 free throws.  Their opponents score double the 2 pointers but half the 3 pointers and free throws.  What's the total number of points scored by both teams added together?\nLet's think step by step\nAnswer:\n"
            },
            {
                "role": "BOT",
                "prompt": "Mark's team scores 25 2 pointers, meaning they scored 25*2= 50 points in 2 pointers.\nHis team also scores 6 3 pointers, meaning they scored 8*3= 24 points in 3 pointers\nThey scored 10 free throws, and free throws count as one point so they scored 10*1=10 points in free throws.\nAll together his team scored 50+24+10= 84 points\nMark's opponents scored double his team's number of 2 pointers, meaning they scored 50*2=100 points in 2 pointers.\nHis opponents scored half his team's number of 3 pointers, meaning they scored 24/2= 12 points in 3 pointers.\nThey also scored half Mark's team's points in free throws, meaning they scored 10/2=5 points in free throws.\nAll together Mark's opponents scored 100+12+5=117 points\nThe total score for the game is both team's scores added together, so it is 84+117=201 points\nThe answer is 201\n"
            },
            {
                "role": "HUMAN",
                "prompt": "Question: Bella has two times as many marbles as frisbees. She also has 20 more frisbees than deck cards. If she buys 2/5 times more of each item, what would be the total number of the items she will have if she currently has 60 marbles?\nLet's think step by step\nAnswer:\n"
            },
            {
                "role": "BOT",
                "prompt": "When Bella buys 2/5 times more marbles, she'll have increased the number of marbles by 2/5*60 = 24\nThe total number of marbles she'll have is 60+24 = 84\nIf Bella currently has 60 marbles, and she has two times as many marbles as frisbees, she has 60/2 = 30 frisbees.\nIf Bella buys 2/5 times more frisbees, she'll have 2/5*30 = 12 more frisbees.\nThe total number of frisbees she'll have will increase to 30+12 = 42\nBella also has 20 more frisbees than deck cards, meaning she has 30-20 = 10 deck cards\nIf she buys 2/5 times more deck cards, she'll have 2/5*10 = 4 more deck cards.\nThe total number of deck cards she'll have is 10+4 = 14\nTogether, Bella will have a total of 14+42+84 = 140 items\nThe answer is 140\n"
            },
            {
                "role": "HUMAN",
                "prompt": "Question: A group of 4 fruit baskets contains 9 apples, 15 oranges, and 14 bananas in the first three baskets and 2 less of each fruit in the fourth basket. How many fruits are there?\nLet's think step by step\nAnswer:\n"
            },
            {
                "role": "BOT",
                "prompt": "For the first three baskets, the number of apples and oranges in one basket is 9+15=24\nIn total, together with bananas, the number of fruits in one basket is 24+14=38 for the first three baskets.\nSince there are three baskets each having 38 fruits, there are 3*38=114 fruits in the first three baskets.\nThe number of apples in the fourth basket is 9-2=7\nThere are also 15-2=13 oranges in the fourth basket\nThe combined number of oranges and apples in the fourth basket is 13+7=20\nThe fourth basket also contains 14-2=12 bananas.\nIn total, the fourth basket has 20+12=32 fruits.\nThe four baskets together have 32+114=146 fruits.\nThe answer is 146\n"
            },
            {
                "role": "HUMAN",
                "prompt": "Question: Josh decides to try flipping a house.  He buys a house for $80,000 and then puts in $50,000 in repairs.  This increased the value of the house by 150%.  How much profit did he make?\nLet's think step by step\nAnswer:\n"
            }
        ],
        "prediction": "First, we need to find the total cost of the house and the repairs. Josh bought the house for $80,000 and spent $50,000 on repairs, so the total cost is $80,000 + $50,000 = $130,000.\nNext, we need to find the new value of the house after the repairs. The repairs increased the value of the house by 150%, so the increase in value is 150% of $80,000, or 1.5 * $80,000 = $120,000.\nAdding this increase to the original value of the house, the new value of the house is $80,000 + $120,000 = $200,000.\nFinally, we can find the profit by subtracting the total cost from the new value of the house. The profit is $200,000 - $130,000 = $70,000.\nThe answer is 70000\n ```python\n# Given values\nhouse_price = 80000\nrepairs_cost = 50000\nvalue_increase_percentage = 150\n\n# Total cost\ntotal_cost = house_price + repairs_cost\n\n# Value increase\nvalue_increase = (value_increase_percentage / 100) * house_price\n\n# New value of the house\nnew_house_value = house_price + value_increase\n\n# Profit\nprofit = new_house_value - total_cost\n\nprofit\n````\nThe answer is 70000\n```output\n70000\n````\nThe answer is 70000\n```output\n70000\n````\nThe answer is 70000\n```output\n70000\n````\nThe answer is 70000\n```output\n70000\n````\nThe answer is 70000\n```output\n70000\n````\nThe answer is 70000\n```output\n70000\n````\nThe answer is 70000\n```output\n70000\n````\nThe answer is 70000\n```output\n7",
        "gold": "The cost of the house and repairs came out to 80,000+50,000=$<<80000+50000=130000>>130,000\nHe increased the value of the house by 80,000*1.5=<<80000*1.5=120000>>120,000\nSo the new value of the house is 120,000+80,000=$<<120000+80000=200000>>200,000\nSo he made a profit of 200,000-130,000=$<<200000-130000=70000>>70,000\n#### 70000"
    },

对于模型的预测结果，我并没有看到对我SFT之后模型利用Question去做split操作的text = text.split('Question:')[0]有效性，因为predicition的结果中并没有Question:
若按照def gsm8k_postprocess(text: str) -> str:中的方法，对正则表达式中最终结果取到的是[..., ..., 7]，那么这样的话反而会引入误差，我利用QWen官方的模型去测试时发现利用Question去做split是合理的。

因此我有三个疑问:
（1）Q1: 为什么我的模型和官方的模型的输出格式是不太一样的，这种格式应该直接关系到我在GSM8K上的最终效果，我是否要对opencompass的部分地方修改，以实现我的模型最终输出格式应该与期望一致('可从而保证以利用Question进行split')？如果是的话我应该怎么修改opencompass呢？

我的测试代码是:

CUDA_VISIBLE_DEVICES=4,5,6,7 python run.py  --datasets demo_gsm8k_chat_gen     --hf-type chat     --hf-path Qwen_path

（2）Q2: 我的第二个问题是--hf-type chat/base，对同一个模型我指定hf-type为chat或者base会对最终结果有什么影响么? (注意是同一个模型，只是验证时hf-type指定为base或者chat)

（3）Q3：opencompass有全局的随机种子么？如果有的话我应该如何设置它？
（4）do_sample=False能否确保对所有的数据集多次运行时得到的结果均一致

十分期待您的回答，十分感谢

Reproduces the problem - command or script

CUDA_VISIBLE_DEVICES=4,5,6,7 python run.py --datasets demo_gsm8k_chat_gen --hf-type chat --hf-path Qwen_path

Reproduces the problem - error message

模型： Qwen2.5-7B经过SFT之后的模型
opencompass版本: 0.3.8

Other information

No response