You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
1.There is a bug in lmms-eval/api/task.py line.736 and 737 if self.config.fewshot_config is not None: self.sampler = samplers.get_sampler(self.config.fewshot_config.get("sampler", "default") if self.config.fewshot_config else "default")(list(self.fewshot_docs()), self, rnd=random.Random(1234))
When evaluate some benches, like GSM8K, it will not set default sampler.
The solution maybe remove the 'if', I hope I'm right.
2.Could you add more text bench, such as 'cmmlu' or 'HumanEval' ?
The text was updated successfully, but these errors were encountered:
1.There is a bug in lmms-eval/api/task.py line.736 and 737
if self.config.fewshot_config is not None: self.sampler = samplers.get_sampler(self.config.fewshot_config.get("sampler", "default") if self.config.fewshot_config else "default")(list(self.fewshot_docs()), self, rnd=random.Random(1234))
When evaluate some benches, like GSM8K, it will not set default sampler.
The solution maybe remove the 'if', I hope I'm right.
2.Could you add more text bench, such as 'cmmlu' or 'HumanEval' ?
The text was updated successfully, but these errors were encountered: