When using LLaVA-Llama3 for batch inference with the generate function, the results are incorrect.

[smile-struggler]

When I use the generate function with batched input, the results are inconsistent compared to when the batch size is 1. There will be many empty strings sometimes.

# batch inference
batch_outputs = self.model.generate(
                inputs=input_ids_list[i:i+batch_size],
                images = images[i:i+batch_size],
                image_sizes = [(336,336)] * input_ids_list[i:i+batch_size].shape[0],
                attention_mask=attention_mask[i:i+batch_size],
                **generation_config)

# single inference
single_outputs = self.model.generate(
                inputs=input_ids_list[0:1],
                images = images[0:1],
                image_sizes = [(336,336)],
                attention_mask=attention_mask[0:1],
                **generation_config)

The batch inference results are correct when working with pure text. What could be the cause of this issue, and how can I resolve it? Thank you!