[BUG] assertion error assert text[: len(left)] == left on MATH wen Qwen-Math-2.5

[d1shs0ap]

Describe the bug

Greedy generation: 100%|█████████████████████████████████████████████████████████████████████████████████| 1
Splits: 100%|███████████████████████████████████████████████████████████████████████████████████████████████Splits: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [1:01:38<00:00, 3698.19s/it]
WARNING:lighteval.logging.hierarchical_logger:  } [1:01:40.194374]
WARNING:lighteval.logging.hierarchical_logger:} [1:02:01.667415]
Traceback (most recent call last):
  File "/home//miniconda3/envs/test/bin/lighteval", line 8, in <module>
    sys.exit(cli_evaluate())
             ^^^^^^^^^^^^^^
  File "/home//miniconda3/envs/test/lib/python3.11/site-packages/lighteval/__main__.py", line 58, in cli_evaluate
    main_accelerate(args)
  File "/home//miniconda3/envs/test/lib/python3.11/site-packages/lighteval/logging/hierarchical_logger.py", line 175, in wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home//miniconda3/envs/test/lib/python3.11/site-packages/lighteval/main_accelerate.py", line 85, in main
    pipeline.evaluate()
  File "/home//miniconda3/envs/test/lib/python3.11/site-packages/lighteval/pipeline.py", line 236, in evaluate
    self._compute_metrics(sample_id_to_responses)
  File "/home//miniconda3/envs/test/lib/python3.11/site-packages/lighteval/pipeline.py", line 288, in _compute_metrics
    metrics = compute_metric(results=sample_responses, formatted_doc=doc, metrics=metric_category_metrics)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home//miniconda3/envs/test/lib/python3.11/site-packages/lighteval/metrics/__init__.py", line 126, in apply_generative_metric
    outputs.update(metric.compute(golds=golds, predictions=preds, formatted_doc=formatted_doc))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home//miniconda3/envs/test/lib/python3.11/site-packages/lighteval/metrics/utils.py", line 75, in compute
    return {self.metric_name: self.sample_level_fn(**kwargs)}  # result, formatted_doc,
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home//miniconda3/envs/test/lib/python3.11/site-packages/lighteval/metrics/metrics_sample.py", line 881, in compute
    all_answers.append(self.get_processed_pred(pred=pred))
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home//miniconda3/envs/test/lib/python3.11/site-packages/lighteval/metrics/metrics_sample.py", line 902, in get_processed_pred
    pred = self.normalize_pred(pred)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home//miniconda3/envs/test/lib/python3.11/site-packages/lighteval/metrics/normalizations.py", line 269, in math_normalizer
    text = _remove_boxed(_last_boxed_only_string(text))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home//miniconda3/envs/test/lib/python3.11/site-packages/lighteval/metrics/normalizations.py", line 108, in _remove_boxed
    assert text[: len(left)] == left
           ^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

To Reproduce

lighteval accelerate \
     --model_args "pretrained=Qwen/Qwen2.5-Math-7B" \
     --tasks "lighteval|math:algebra|0|0" \
     --override_batch_size 1 \
     --output_dir="./evals/"

Version info

lighteval==0.5.0