Can I use glm 4v 9b to emdedding an image?

[neblen]

Feature request / 功能建议

use glm 4v 9b to emdedding an image,Because this model has powerful image understanding

Motivation / 动机

emdedding images

Your contribution / 您的贡献

Can I use glm 4v 9b to emdedding an image? I want to know if there is such a use

[sixsixcoder]

This model file can be used to emdedding images. https://huggingface.co/THUDM/glm-4v-9b/blob/main/visual.py

[neblen]

thanks~ I want to know if there are any specific examples of using image embedding. Since glm 4v 9b has good image understanding ability, I would like to try the effect of embedding @sixsixcoder

[neblen]

Is the vector from glm 9b 4v embedding so large?

[sixsixcoder]

There should be a tenosr transformation for this, you can refer to this code

def merge_glm_vision_embeddings(
    input_ids: torch.Tensor,
    inputs_embeds: torch.Tensor,
    vision_embeddings: torch.Tensor,
    boi_token_id: int,
    eoi_token_id: int,
) -> torch.Tensor:

    boi_positions = (input_ids == boi_token_id).nonzero(as_tuple=True)[0]
    eoi_positions = (input_ids == eoi_token_id).nonzero(as_tuple=True)[0]

    mask = torch.zeros_like(input_ids, dtype=torch.bool)

    for boi_pos, eoi_pos in zip(boi_positions, eoi_positions):
        assert boi_pos < eoi_pos
        mask[boi_pos:eoi_pos + 1] = True
    inputs_embeds[mask] = vision_embeddings.view(-1,
                                                 vision_embeddings.shape[-1])
    return inputs_embeds

[neblen]

thanks~ @sixsixcoder I use this EVA2CLIPModel function to get a matrix with very high dimensions.
An image will have 1602 patches, each patch is a 4096-dimensional vector.
Is this correct?

https://huggingface.co/THUDM/glm-4v-9b/blob/main/visual.py

class EVA2CLIPModel(nn.Module):
    def __init__(self, config):
        super().__init__()
        vision_config = Namespace(**config.vision_config)
        self.patch_embedding = PatchEmbedding(vision_config)
        self.transformer = Transformer(vision_config)
        self.linear_proj = GLU(config, in_features=config.hidden_size)
        self.conv = nn.Conv2d(in_channels=vision_config.hidden_size, out_channels=config.hidden_size, kernel_size=2,
                              stride=2)
        self.boi = nn.Parameter(torch.zeros(1, 1, config.hidden_size))
        self.eoi = nn.Parameter(torch.zeros(1, 1, config.hidden_size))
        self.scaling_factor = vision_config.scaling_factor

    def forward(self, images: "tensor(B, C, H, W)") -> "tensor(B, L, D)":
        x = self.patch_embedding(images)
        x = self.transformer(x)
        x = x[:, 1:]

        b, s, h = x.shape
        grid_size = int(s ** 0.5)
        x = x.view(b, grid_size, grid_size, h).permute(0, 3, 1, 2)
        x = self.conv(x)

        x = x.flatten(2).transpose(1, 2)
        x = self.linear_proj(x)

        # https://github.com/THUDM/GLM-4/issues/350
        boi = self.boi.expand(x.shape[0], -1, -1).to(x.device)
        eoi = self.eoi.expand(x.shape[0], -1, -1).to(x.device)
        x = torch.cat((boi, x, eoi), dim=1)
        x = x / self.scaling_factor
        return x

[sixsixcoder]

it is right

[neblen]

thanks a lot~ @sixsixcoder How much would the image understanding capability be reduced if the image patches were reduced to 1? I want to use this image understanding ability for image duplication detection. I don't know if it is feasible.

[sixsixcoder]

I have no prior experience with this. Utilizing just a single patch could potentially lead to the loss of image features, which would undoubtedly impact the model's comprehension of the image. Should your GPU memory be enough, it is advisable to conduct experiments using the image patches size as defined by the model.

[neblen]

thanks a lot~ @sixsixcoder By the way, is the embedding vector of the same image different each time? Is it possible to fix the seed and get the same vector?