Validation during training for VLMs?

[benjamin-marie]

Is eval_dataset in the SFTTrainer supported by Unsloth for VLMs?

When I fine-tune Qwen2-VL and pass an evaluation dataset

trainer = SFTTrainer(
    model = model,
    tokenizer = tokenizer,
    data_collator = UnslothVisionDataCollator(model, tokenizer), # Must use!
    train_dataset = dataset_train,
    eval_dataset = dataset_validation,
    args = training_args
)

it triggers this error when it reaches the evaluation steps:

[/usr/local/lib/python3.10/dist-packages/transformers/trainer.py](https://localhost:8080/#) in compute_loss(self, model, inputs, return_outputs, num_items_in_batch)
   3652         else:
   3653             if isinstance(outputs, dict) and "loss" not in outputs:
-> 3654                 raise ValueError(
   3655                     "The model did not return a loss from the inputs, only the following keys: "
   3656                     f"{','.join(outputs.keys())}. For reference, the inputs it received are {','.join(inputs.keys())}."

ValueError: The model did not return a loss from the inputs, only the following keys: logits. For reference, the inputs it received are input_ids,attention_mask,pixel_values,image_grid_thw,labels.

I processed the evaluation dataset the same way as the training dataset (as in Unsloth's notebooks for VLMs).