[Question] Question on image_newline for single image

[SCZwangxiao]

I think the image_newline here is the implementation of Row-ended tokens in the paper.

LLaVA/llava/model/llava_arch.py

Lines 82 to 86 in c121f04

	if 'unpad' in mm_patch_merge_type:
	embed_std = 1 / torch.sqrt(torch.tensor(self.config.hidden_size, dtype=self.dtype))
	self.image_newline = nn.Parameter(
	torch.randn(self.config.hidden_size, dtype=self.dtype) * embed_std
	)

However, for single image input, the tokens are not appended to each row as expected in the paper. Specifically, only one token is appended to the flatten patch tokens of the image.

LLaVA/llava/model/llava_arch.py

Lines 191 to 196 in c121f04

	image_feature = image_feature[0]
	if 'unpad' in mm_patch_merge_type:
	image_feature = torch.cat((
	image_feature,
	self.model.image_newline[None].to(image_feature.device)
	), dim=0)