Add mt-bench #75
Add mt-bench #75
Conversation
|
@NathanHB feel free to ping me once it's merged with main so we can integrate it :) |
|
Careful with the deletion of task_examples, quite sure some of these files are needed by the nanotron team. |
src/lighteval/models/base_model.py
Outdated
| @@ -345,6 +353,91 @@ def greedy_until_with_logits( | |||
| override_bs=override_bs, | |||
| ) | |||
|
|
|||
| def greedy_until_multi_turn(self, requests: list[GreedyUntilMultiTurnRequest], override_bs: Optional[int] = None) -> GenerateMultiTurnReturn: | |||
There was a problem hiding this comment.
You'll also need to update the other model launchers, or at least add a function in the abstract model which indicates that it needs to be implemented
There was a problem hiding this comment.
Quite cool overall!
Some code reorgs are need imo (management of multi turn context with all the other contexts, finding a better way to pass judgements for logging, implementing greedy_until_multi_turn in the other models, or at least the LightevalModel class), plus some doc.
Nice that the code is so neat, it looks like it will be easy to switch to "using a judge as metric" in the future!
What this PR does: - Uses custom metrics and tasks to add llm a as judge - adds multi turn generation - Adds mt-bench metric This implementation uses mt-bench prompts from [InflectionAI](https://github.com/InflectionAI/Inflection-Benchmarks). The code is inspired from the original implementation of mt-bench with notable differences. - mt-bench uses a custom-made chat templating system, we use the tokenizer - mt-bench uses an old version of the openai API, we use the newest one, with very simplified logic for chat prompt formating. We can easily add more models to act as judge. - We do not use varying temperature based on the sample we are evaluating. All samples are generated using `do_sample=False` and temperature set to `0.0`.
What this PR does:
This implementation uses mt-bench prompts from InflectionAI. The code is inspired from the original implementation of mt-bench with notable differences.
do_sample=Falseand temperature set to0.0.