Using A hugging face model/weights in Timm

[nighting0le01]

Hi i want to use the following model weights in timm but they are not available. how can I use these in timm? any ideas?

1. google/vit-large-patch16-224-in21k:https://huggingface.co/google/vit-large-patch16-224-in21k
2. google/vit-large-patch16-224 : https://huggingface.co/google/vit-large-patch16-224

A similar model in timm in pretrained on in21k ft and uses augreg:timm/vit_large_patch16_224.augreg_in21k . even though its a vit large it doesn't have the same layers. Please help me thanks

[rwightman]

augreg_in21k is better than the HF transformers weighs you posted above, they are from the 'How to Train Your ViT' paper.

timm did have the same weights as google/vit* transformers weights, they were originally adapted from the timm ones, but some of the originals didn't make it back after adding the augreg ones, can fix that if there's any interest... original weights have .orig_* tags, but only have a few of them there still....

[nighting0le01]

@rwightman also a dumb question the timm/vit_large_patch16_224.augreg_in21k are supposed to have very bad performance on in_1k right? if they are not ft on in_1k and we directly test it on 1_k adding the relevant head

[nighting0le01]

hello @rwightman actually there is difference in layers b/w the google/vit-large-patch16-224 and timm/vit-large-patch16-224. not only are the layer names different, the number of layers are also different.

keys names different in 2 models but they mean the same how to correct it in pytorch 
RuntimeError: Error(s) in loading state_dict for VisionTransformer:
        Missing key(s) in state_dict: "cls_token", "pos_embed", "patch_embed.proj.weight", "patch_embed.proj.bias", "blocks.0.norm1.weight", "blocks.0.norm1.bias", "blocks.0.attn.qkv.weight", "blocks.0.attn.qkv.bias", "blocks.0.attn.proj.weight", "blocks.0.attn.proj.bias", "blocks.0.norm2.weight", "blocks.0.norm2.bias", "blocks.0.mlp.fc1.weight", "blocks.0.mlp.fc1.bias", "blocks.0.mlp.fc2.weight", "blocks.0.mlp.fc2.bias", "blocks.1.norm1.weight", "blocks.1.norm1.bias", "blocks.1.attn.qkv.weight", "blocks.1.attn.qkv.bias", "blocks.1.attn.proj.weight", "blocks.1.attn.proj.bias", "blocks.1.norm2.weight", "blocks.1.norm2.bias", "blocks.1.mlp.fc1.weight", "blocks.1.mlp.fc1.bias", "blocks.1.mlp.fc2.weight", "blocks.1.mlp.fc2.bias", "blocks.2.norm1.weight", "blocks.2.norm1.bias", "blocks.2.attn.qkv.weight", "blocks.2.attn.qkv.bias", "blocks.2.attn.proj.weight", "blocks.2.attn.proj.bias", "blocks.2.norm2.weight", "blocks.2.norm2.bias", "blocks.2.mlp.fc1.weight", "blocks.2.mlp.fc1.bias", "blocks.2.mlp.fc2.weight", "blocks.2.mlp.fc2.bias", "blocks.3.norm1.weight", "blocks.3.norm1.bias", "blocks.3.attn.qkv.weight", "blocks.3.attn.qkv.bias", "blocks.3.attn.proj.weight", "blocks.3.attn.proj.bias", "blocks.3.norm2.weight", "blocks.3.norm2.bias", "blocks.3.mlp.fc1.weight", "blocks.3.mlp.fc1.bias", "blocks.3.mlp.fc2.weight", "blocks.3.mlp.fc2.bias", "blocks.4.norm1.weight", "blocks.4.norm1.bias", "blocks.4.attn.qkv.weight", "blocks.4.attn.qkv.bias", "blocks.4.attn.proj.weight", "blocks.4.attn.proj.bias", "blocks.4.norm2.weight", "blocks.4.norm2.bias", "blocks.4.mlp.fc1.weight", "blocks.4.mlp.fc1.bias", "blocks.4.mlp.fc2.weight", "blocks.4.mlp.fc2.bias", "blocks.5.norm1.weight", "blocks.5.norm1.bias", "blocks.5.attn.qkv.weight", "blocks.5.attn.qkv.bias", "blocks.5.attn.proj.weight", "blocks.5.attn.proj.bias", "blocks.5.norm2.weight", "blocks.5.norm2.bias", "blocks.5.mlp.fc1.weight", "blocks.5.mlp.fc1.bias", "blocks.5.mlp.fc2.weight", "blocks.5.mlp.fc2.bias", "blocks.6.norm1.weight", "blocks.6.norm1.bias", "blocks.6.attn.qkv.weight", "blocks.6.attn.qkv.bias", "blocks.6.attn.proj.weight", "blocks.6.attn.proj.bias", "blocks.6.norm2.weight", "blocks.6.norm2.bias", "blocks.6.mlp.fc1.weight", "blocks.6.mlp.fc1.bias", "blocks.6.mlp.fc2.weight", "blocks.6.mlp.fc2.bias", "blocks.7.norm1.weight", "blocks.7.norm1.bias", "blocks.7.attn.qkv.weight", "blocks.7.attn.qkv.bias", "blocks.7.attn.proj.weight", "blocks.7.attn.proj.bias", "blocks.7.norm2.weight", "blocks.7.norm2.bias", "blocks.7.mlp.fc1.weight", "blocks.7.mlp.fc1.bias", "blocks.7.mlp.fc2.weight", "blocks.7.mlp.fc2.bias", "blocks.8.norm1.weight", "blocks.8.norm1.bias", "blocks.8.attn.qkv.weight", "blocks.8.attn.qkv.bias", "blocks.8.attn.proj.weight", "blocks.8.attn.proj.bias", "blocks.8.norm2.weight", "blocks.8.norm2.bias", "blocks.8.mlp.fc1.weight", "blocks.8.mlp.fc1.bias", "blocks.8.mlp.fc2.weight", "blocks.8.mlp.fc2.bias", "blocks.9.norm1.weight", "blocks.9.norm1.bias", "blocks.9.attn.qkv.weight", "blocks.9.attn.qkv.bias", "blocks.9.attn.proj.weight", "blocks.9.attn.proj.bias", "blocks.9.norm2.weight", "blocks.9.norm2.bias", "blocks.9.mlp.fc1.weight", "blocks.9.mlp.fc1.bias", "blocks.9.mlp.fc2.weight", "blocks.9.mlp.fc2.bias", "blocks.10.norm1.weight", "blocks.10.norm1.bias", "blocks.10.attn.qkv.weight", "blocks.10.attn.qkv.bias", "blocks.10.attn.proj.weight", "blocks.10.attn.proj.bias", "blocks.10.norm2.weight", "blocks.10.norm2.bias", "blocks.10.mlp.fc1.weight", "blocks.10.mlp.fc1.bias", "blocks.10.mlp.fc2.weight", "blocks.10.mlp.fc2.bias", "blocks.11.norm1.weight", "blocks.11.norm1.bias", "blocks.11.attn.qkv.weight", "blocks.11.attn.qkv.bias", "blocks.11.attn.proj.weight", "blocks.11.attn.proj.bias", "blocks.11.norm2.weight", "blocks.11.norm2.bias", "blocks.11.mlp.fc1.weight", "blocks.11.mlp.fc1.bias", "blocks.11.mlp.fc2.weight", "blocks.11.mlp.fc2.bias", "blocks.12.norm1.weight", "blocks.12.norm1.bias", "blocks.12.attn.qkv.weight", "blocks.12.attn.qkv.bias", "blocks.12.attn.proj.weight", "blocks.12.attn.proj.bias", "blocks.12.norm2.weight", "blocks.12.norm2.bias", "blocks.12.mlp.fc1.weight", "blocks.12.mlp.fc1.bias", "blocks.12.mlp.fc2.weight", "blocks.12.mlp.fc2.bias", "blocks.13.norm1.weight", "blocks.13.norm1.bias", "blocks.13.attn.qkv.weight", "blocks.13.attn.qkv.bias", "blocks.13.attn.proj.weight", "blocks.13.attn.proj.bias", "blocks.13.norm2.weight", "blocks.13.norm2.bias", "blocks.13.mlp.fc1.weight", "blocks.13.mlp.fc1.bias", "blocks.13.mlp.fc2.weight", "blocks.13.mlp.fc2.bias", "blocks.14.norm1.weight", "blocks.14.norm1.bias", "blocks.14.attn.qkv.weight", "blocks.14.attn.qkv.bias", "blocks.14.attn.proj.weight", "blocks.14.attn.proj.bias", "blocks.14.norm2.weight", "blocks.14.norm2.bias", "blocks.14.mlp.fc1.weight", "blocks.14.mlp.fc1.bias", "blocks.14.mlp.fc2.weight", "blocks.14.mlp.fc2.bias", "blocks.15.norm1.weight", "blocks.15.norm1.bias", "blocks.15.attn.qkv.weight", "blocks.15.attn.qkv.bias", "blocks.15.attn.proj.weight", "blocks.15.attn.proj.bias", "blocks.15.norm2.weight", "blocks.15.norm2.bias", "blocks.15.mlp.fc1.weight", "blocks.15.mlp.fc1.bias", "blocks.15.mlp.fc2.weight", "blocks.15.mlp.fc2.bias", "blocks.16.norm1.weight", "blocks.16.norm1.bias", "blocks.16.attn.qkv.weight", "blocks.16.attn.qkv.bias", "blocks.16.attn.proj.weight", "blocks.16.attn.proj.bias", "blocks.16.norm2.weight", "blocks.16.norm2.bias", "blocks.16.mlp.fc1.weight", "blocks.16.mlp.fc1.bias", "blocks.16.mlp.fc2.weight", "blocks.16.mlp.fc2.bias", "blocks.17.norm1.weight", "blocks.17.norm1.bias", "blocks.17.attn.qkv.weight", "blocks.17.attn.qkv.bias", "blocks.17.attn.proj.weight", "blocks.17.attn.proj.bias", "blocks.17.norm2.weight", "blocks.17.norm2.bias", "blocks.17.mlp.fc1.weight", "blocks.17.mlp.fc1.bias", "blocks.17.mlp.fc2.weight", "blocks.17.mlp.fc2.bias", "blocks.18.norm1.weight", "blocks.18.norm1.bias", "blocks.18.attn.qkv.weight", "blocks.18.attn.qkv.bias", "blocks.18.attn.proj.weight", "blocks.18.attn.proj.bias", "blocks.18.norm2.weight", "blocks.18.norm2.bias", "blocks.18.mlp.fc1.weight", "blocks.18.mlp.fc1.bias", "blocks.18.mlp.fc2.weight", "blocks.18.mlp.fc2.bias", "blocks.19.norm1.weight", "blocks.19.norm1.bias", "blocks.19.attn.qkv.weight", "blocks.19.attn.qkv.bias", "blocks.19.attn.proj.weight", "blocks.19.attn.proj.bias", "blocks.19.norm2.weight", "blocks.19.norm2.bias", "blocks.19.mlp.fc1.weight", "blocks.19.mlp.fc1.bias", "blocks.19.mlp.fc2.weight", "blocks.19.mlp.fc2.bias", "blocks.20.norm1.weight", "blocks.20.norm1.bias", "blocks.20.attn.qkv.weight", "blocks.20.attn.qkv.bias", "blocks.20.attn.proj.weight", "blocks.20.attn.proj.bias", "blocks.20.norm2.weight", "blocks.20.norm2.bias", "blocks.20.mlp.fc1.weight", "blocks.20.mlp.fc1.bias", "blocks.20.mlp.fc2.weight", "blocks.20.mlp.fc2.bias", "blocks.21.norm1.weight", "blocks.21.norm1.bias", "blocks.21.attn.qkv.weight", "blocks.21.attn.qkv.bias", "blocks.21.attn.proj.weight", "blocks.21.attn.proj.bias", "blocks.21.norm2.weight", "blocks.21.norm2.bias", "blocks.21.mlp.fc1.weight", "blocks.21.mlp.fc1.bias", "blocks.21.mlp.fc2.weight", "blocks.21.mlp.fc2.bias", "blocks.22.norm1.weight", "blocks.22.norm1.bias", "blocks.22.attn.qkv.weight", "blocks.22.attn.qkv.bias", "blocks.22.attn.proj.weight", "blocks.22.attn.proj.bias", "blocks.22.norm2.weight", "blocks.22.norm2.bias", "blocks.22.mlp.fc1.weight", "blocks.22.mlp.fc1.bias", "blocks.22.mlp.fc2.weight", "blocks.22.mlp.fc2.bias", "blocks.23.norm1.weight", "blocks.23.norm1.bias", "blocks.23.attn.qkv.weight", "blocks.23.attn.qkv.bias", "blocks.23.attn.proj.weight", "blocks.23.attn.proj.bias", "blocks.23.norm2.weight", "blocks.23.norm2.bias", "blocks.23.mlp.fc1.weight", "blocks.23.mlp.fc1.bias", "blocks.23.mlp.fc2.weight", "blocks.23.mlp.fc2.bias", "norm.weight", "norm.bias", "head.weight", "head.bias". 
        Unexpected key(s) in state_dict: "vit.embeddings.cls_token", "vit.embeddings.position_embeddings", "vit.embeddings.patch_embeddings.projection.weight", "vit.embeddings.patch_embeddings.projection.bias", "vit.encoder.layer.0.attention.attention.query.weight", "vit.encoder.layer.0.attention.attention.query.bias", "vit.encoder.layer.0.attention.attention.key.weight", "vit.encoder.layer.0.attention.attention.key.bias", "vit.encoder.layer.0.attention.attention.value.weight", "vit.encoder.layer.0.attention.attention.value.bias", "vit.encoder.layer.0.attention.output.dense.weight", "vit.encoder.layer.0.attention.output.dense.bias", "vit.encoder.layer.0.intermediate.dense.weight", "vit.encoder.layer.0.intermediate.dense.bias", "vit.encoder.layer.0.output.dense.weight", "vit.encoder.layer.0.output.dense.bias", "vit.encoder.layer.0.layernorm_before.weight", "vit.encoder.layer.0.layernorm_before.bias", "vit.encoder.layer.0.layernorm_after.weight", "vit.encoder.layer.0.layernorm_after.bias", "vit.encoder.layer.1.attention.attention.query.weight", "vit.encoder.layer.1.attention.attention.query.bias", "vit.encoder.layer.1.attention.attention.key.weight", "vit.encoder.layer.1.attention.attention.key.bias", "vit.encoder.layer.1.attention.attention.value.weight", "vit.encoder.layer.1.attention.attention.value.bias", "vit.encoder.layer.1.attention.output.dense.weight", "vit.encoder.layer.1.attention.output.dense.bias", "vit.encoder.layer.1.intermediate.dense.weight", "vit.encoder.layer.1.intermediate.dense.bias", "vit.encoder.layer.1.output.dense.weight", "vit.encoder.layer.1.output.dense.bias", "vit.encoder.layer.1.layernorm_before.weight", "vit.encoder.layer.1.layernorm_before.bias", "vit.encoder.layer.1.layernorm_after.weight", "vit.encoder.layer.1.layernorm_after.bias", "vit.encoder.layer.2.attention.attention.query.weight", "vit.encoder.layer.2.attention.attention.query.bias", "vit.encoder.layer.2.attention.attention.key.weight", "vit.encoder.layer.2.attention.attention.key.bias", "vit.encoder.layer.2.attention.attention.value.weight", "vit.encoder.layer.2.attention.attention.value.bias", "vit.encoder.layer.2.attention.output.dense.weight", "vit.encoder.layer.2.attention.output.dense.bias", "vit.encoder.layer.2.intermediate.dense.weight", "vit.encoder.layer.2.intermediate.dense.bias", "vit.encoder.layer.2.output.dense.weight", "vit.encoder.layer.2.output.dense.bias", "vit.encoder.layer.2.layernorm_before.weight", "vit.encoder.layer.2.layernorm_before.bias", "vit.encoder.layer.2.layernorm_after.weight", "vit.encoder.layer.2.layernorm_after.bias", "vit.encoder.layer.3.attention.attention.query.weight", "vit.encoder.layer.3.attention.attention.query.bias", "vit.encoder.layer.3.attention.attention.key.weight", "vit.encoder.layer.3.attention.attention.key.bias", "vit.encoder.layer.3.attention.attention.value.weight", "vit.encoder.layer.3.attention.attention.value.bias", "vit.encoder.layer.3.attention.output.dense.weight", "vit.encoder.layer.3.attention.output.dense.bias", "vit.encoder.layer.3.intermediate.dense.weight", "vit.encoder.layer.3.intermediate.dense.bias", "vit.encoder.layer.3.output.dense.weight", "vit.encoder.layer.3.output.dense.bias", "vit.encoder.layer.3.layernorm_before.weight", "vit.encoder.layer.3.layernorm_before.bias", "vit.encoder.layer.3.layernorm_after.weight", "vit.encoder.layer.3.layernorm_after.bias", "vit.encoder.layer.4.attention.attention.query.weight", "vit.encoder.layer.4.attention.attention.query.bias", "vit.encoder.layer.4.attention.attention.key.weight", "vit.encoder.layer.4.attention.attention.key.bias", "vit.encoder.layer.4.attention.attention.value.weight", "vit.encoder.layer.4.attention.attention.value.bias", "vit.encoder.layer.4.attention.output.dense.weight", "vit.encoder.layer.4.attention.output.dense.bias", "vit.encoder.layer.4.intermediate.dense.weight", "vit.encoder.layer.4.intermediate.dense.bias", "vit.encoder.layer.4.output.dense.weight", "vit.encoder.layer.4.output.dense.bias", "vit.encoder.layer.4.layernorm_before.weight", "vit.encoder.layer.4.layernorm_before.bias", "vit.encoder.layer.4.layernorm_after.weight", "vit.encoder.layer.4.layernorm_after.bias", "vit.encoder.layer.5.attention.attention.query.weight", "vit.encoder.layer.5.attention.attention.query.bias", "vit.encoder.layer.5.attention.attention.key.weight", "vit.encoder.layer.5.attention.attention.key.bias", "vit.encoder.layer.5.attention.attention.value.weight", "vit.encoder.layer.5.attention.attention.value.bias", "vit.encoder.layer.5.attention.output.dense.weight", "vit.encoder.layer.5.attention.output.dense.bias", "vit.encoder.layer.5.intermediate.dense.weight", "vit.encoder.layer.5.intermediate.dense.bias", "vit.encoder.layer.5.output.dense.weight", "vit.encoder.layer.5.output.dense.bias", "vit.encoder.layer.5.layernorm_before.weight", "vit.encoder.layer.5.layernorm_before.bias", "vit.encoder.layer.5.layernorm_after.weight", "vit.encoder.layer.5.layernorm_after.bias", "vit.encoder.layer.6.attention.attention.query.weight", "vit.encoder.layer.6.attention.attention.query.bias", "vit.encoder.layer.6.attention.attention.key.weight", "vit.encoder.layer.6.attention.attention.key.bias", "vit.encoder.layer.6.attention.attention.value.weight", "vit.encoder.layer.6.attention.attention.value.bias", "vit.encoder.layer.6.attention.output.dense.weight", "vit.encoder.layer.6.attention.output.dense.bias", "vit.encoder.layer.6.intermediate.dense.weight", "vit.encoder.layer.6.intermediate.dense.bias", "vit.encoder.layer.6.output.dense.weight", "vit.encoder.layer.6.output.dense.bias", "vit.encoder.layer.6.layernorm_before.weight", "vit.encoder.layer.6.layernorm_before.bias", "vit.encoder.layer.6.layernorm_after.weight", "vit.encoder.layer.6.layernorm_after.bias", "vit.encoder.layer.7.attention.attention.query.weight", "vit.encoder.layer.7.attention.attention.query.bias", "vit.encoder.layer.7.attention.attention.key.weight", "vit.encoder.layer.7.attention.attention.key.bias", "vit.encoder.layer.7.attention.attention.value.weight", "vit.encoder.layer.7.attention.attention.value.bias", "vit.encoder.layer.7.attention.output.dense.weight", "vit.encoder.layer.7.attention.output.dense.bias", "vit.encoder.layer.7.intermediate.dense.weight", "vit.encoder.layer.7.intermediate.dense.bias", "vit.encoder.layer.7.output.dense.weight", "vit.encoder.layer.7.output.dense.bias", "vit.encoder.layer.7.layernorm_before.weight", "vit.encoder.layer.7.layernorm_before.bias", "vit.encoder.layer.7.layernorm_after.weight", "vit.encoder.layer.7.layernorm_after.bias", "vit.encoder.layer.8.attention.attention.query.weight", "vit.encoder.layer.8.attention.attention.query.bias", "vit.encoder.layer.8.attention.attention.key.weight", "vit.encoder.layer.8.attention.attention.key.bias", "vit.encoder.layer.8.attention.attention.value.weight", "vit.encoder.layer.8.attention.attention.value.bias", "vit.encoder.layer.8.attention.output.dense.weight", "vit.encoder.layer.8.attention.output.dense.bias", "vit.encoder.layer.8.intermediate.dense.weight", "vit.encoder.layer.8.intermediate.dense.bias", "vit.encoder.layer.8.output.dense.weight", "vit.encoder.layer.8.output.dense.bias", "vit.encoder.layer.8.layernorm_before.weight", "vit.encoder.layer.8.layernorm_before.bias", "vit.encoder.layer.8.layernorm_after.weight", "vit.encoder.layer.8.layernorm_after.bias", "vit.encoder.layer.9.attention.attention.query.weight", "vit.encoder.layer.9.attention.attention.query.bias", "vit.encoder.layer.9.attention.attention.key.weight", "vit.encoder.layer.9.attention.attention.key.bias", "vit.encoder.layer.9.attention.attention.value.weight", "vit.encoder.layer.9.attention.attention.value.bias", "vit.encoder.layer.9.attention.output.dense.weight", "vit.encoder.layer.9.attention.output.dense.bias", "vit.encoder.layer.9.intermediate.dense.weight", "vit.encoder.layer.9.intermediate.dense.bias", "vit.encoder.layer.9.output.dense.weight", "vit.encoder.layer.9.output.dense.bias", "vit.encoder.layer.9.layernorm_before.weight", "vit.encoder.layer.9.layernorm_before.bias", "vit.encoder.layer.9.layernorm_after.weight", "vit.encoder.layer.9.layernorm_after.bias", "vit.encoder.layer.10.attention.attention.query.weight", "vit.encoder.layer.10.attention.attention.query.bias", "vit.encoder.layer.10.attention.attention.key.weight", "vit.encoder.layer.10.attention.attention.key.bias", "vit.encoder.layer.10.attention.attention.value.weight", "vit.encoder.layer.10.attention.attention.value.bias", "vit.encoder.layer.10.attention.output.dense.weight", "vit.encoder.layer.10.attention.output.dense.bias", "vit.encoder.layer.10.intermediate.dense.weight", "vit.encoder.layer.10.intermediate.dense.bias", "vit.encoder.layer.10.output.dense.weight", "vit.encoder.layer.10.output.dense.bias", "vit.encoder.layer.10.layernorm_before.weight", "vit.encoder.layer.10.layernorm_before.bias", "vit.encoder.layer.10.layernorm_after.weight", "vit.encoder.layer.10.layernorm_after.bias", "vit.encoder.layer.11.attention.attention.query.weight", "vit.encoder.layer.11.attention.attention.query.bias", "vit.encoder.layer.11.attention.attention.key.weight", "vit.encoder.layer.11.attention.attention.key.bias", "vit.encoder.layer.11.attention.attention.value.weight", "vit.encoder.layer.11.attention.attention.value.bias", "vit.encoder.layer.11.attention.output.dense.weight", "vit.encoder.layer.11.attention.output.dense.bias", "vit.encoder.layer.11.intermediate.dense.weight", "vit.encoder.layer.11.intermediate.dense.bias", "vit.encoder.layer.11.output.dense.weight", "vit.encoder.layer.11.output.dense.bias", "vit.encoder.layer.11.layernorm_before.weight", "vit.encoder.layer.11.layernorm_before.bias", "vit.encoder.layer.11.layernorm_after.weight", "vit.encoder.layer.11.layernorm_after.bias", "vit.encoder.layer.12.attention.attention.query.weight", "vit.encoder.layer.12.attention.attention.query.bias", "vit.encoder.layer.12.attention.attention.key.weight", "vit.encoder.layer.12.attention.attention.key.bias", "vit.encoder.layer.12.attention.attention.value.weight", "vit.encoder.layer.12.attention.attention.value.bias", "vit.encoder.layer.12.attention.output.dense.weight", "vit.encoder.layer.12.attention.output.dense.bias", "vit.encoder.layer.12.intermediate.dense.weight", "vit.encoder.layer.12.intermediate.dense.bias", "vit.encoder.layer.12.output.dense.weight", "vit.encoder.layer.12.output.dense.bias", "vit.encoder.layer.12.layernorm_before.weight", "vit.encoder.layer.12.layernorm_before.bias", "vit.encoder.layer.12.layernorm_after.weight", "vit.encoder.layer.12.layernorm_after.bias", "vit.encoder.layer.13.attention.attention.query.weight", "vit.encoder.layer.13.attention.attention.query.bias", "vit.encoder.layer.13.attention.attention.key.weight", "vit.encoder.layer.13.attention.attention.key.bias", "vit.encoder.layer.13.attention.attention.value.weight", "vit.encoder.layer.13.attention.attention.value.bias", "vit.encoder.layer.13.attention.output.dense.weight", "vit.encoder.layer.13.attention.output.dense.bias", "vit.encoder.layer.13.intermediate.dense.weight", "vit.encoder.layer.13.intermediate.dense.bias", "vit.encoder.layer.13.output.dense.weight", "vit.encoder.layer.13.output.dense.bias", "vit.encoder.layer.13.layernorm_before.weight", "vit.encoder.layer.13.layernorm_before.bias", "vit.encoder.layer.13.layernorm_after.weight", "vit.encoder.layer.13.layernorm_after.bias", "vit.encoder.layer.14.attention.attention.query.weight", "vit.encoder.layer.14.attention.attention.query.bias", "vit.encoder.layer.14.attention.attention.key.weight", "vit.encoder.layer.14.attention.attention.key.bias", "vit.encoder.layer.14.attention.attention.value.weight", "vit.encoder.layer.14.attention.attention.value.bias", "vit.encoder.layer.14.attention.output.dense.weight", "vit.encoder.layer.14.attention.output.dense.bias", "vit.encoder.layer.14.intermediate.dense.weight", "vit.encoder.layer.14.intermediate.dense.bias", "vit.encoder.layer.14.output.dense.weight", "vit.encoder.layer.14.output.dense.bias", "vit.encoder.layer.14.layernorm_before.weight", "vit.encoder.layer.14.layernorm_before.bias", "vit.encoder.layer.14.layernorm_after.weight", "vit.encoder.layer.14.layernorm_after.bias", "vit.encoder.layer.15.attention.attention.query.weight", "vit.encoder.layer.15.attention.attention.query.bias", "vit.encoder.layer.15.attention.attention.key.weight", "vit.encoder.layer.15.attention.attention.key.bias", "vit.encoder.layer.15.attention.attention.value.weight", "vit.encoder.layer.15.attention.attention.value.bias", "vit.encoder.layer.15.attention.output.dense.weight", "vit.encoder.layer.15.attention.output.dense.bias", "vit.encoder.layer.15.intermediate.dense.weight", "vit.encoder.layer.15.intermediate.dense.bias", "vit.encoder.layer.15.output.dense.weight", "vit.encoder.layer.15.output.dense.bias", "vit.encoder.layer.15.layernorm_before.weight", "vit.encoder.layer.15.layernorm_before.bias", "vit.encoder.layer.15.layernorm_after.weight", "vit.encoder.layer.15.layernorm_after.bias", "vit.encoder.layer.16.attention.attention.query.weight", "vit.encoder.layer.16.attention.attention.query.bias", "vit.encoder.layer.16.attention.attention.key.weight", "vit.encoder.layer.16.attention.attention.key.bias", "vit.encoder.layer.16.attention.attention.value.weight", "vit.encoder.layer.16.attention.attention.value.bias", "vit.encoder.layer.16.attention.output.dense.weight", "vit.encoder.layer.16.attention.output.dense.bias", "vit.encoder.layer.16.intermediate.dense.weight", "vit.encoder.layer.16.intermediate.dense.bias", "vit.encoder.layer.16.output.dense.weight", "vit.encoder.layer.16.output.dense.bias", "vit.encoder.layer.16.layernorm_before.weight", "vit.encoder.layer.16.layernorm_before.bias", "vit.encoder.layer.16.layernorm_after.weight", "vit.encoder.layer.16.layernorm_after.bias", "vit.encoder.layer.17.attention.attention.query.weight", "vit.encoder.layer.17.attention.attention.query.bias", "vit.encoder.layer.17.attention.attention.key.weight", "vit.encoder.layer.17.attention.attention.key.bias", "vit.encoder.layer.17.attention.attention.value.weight", "vit.encoder.layer.17.attention.attention.value.bias", "vit.encoder.layer.17.attention.output.dense.weight", "vit.encoder.layer.17.attention.output.dense.bias", "vit.encoder.layer.17.intermediate.dense.weight", "vit.encoder.layer.17.intermediate.dense.bias", "vit.encoder.layer.17.output.dense.weight", "vit.encoder.layer.17.output.dense.bias", "vit.encoder.layer.17.layernorm_before.weight", "vit.encoder.layer.17.layernorm_before.bias", "vit.encoder.layer.17.layernorm_after.weight", "vit.encoder.layer.17.layernorm_after.bias", "vit.encoder.layer.18.attention.attention.query.weight", "vit.encoder.layer.18.attention.attention.query.bias", "vit.encoder.layer.18.attention.attention.key.weight", "vit.encoder.layer.18.attention.attention.key.bias", "vit.encoder.layer.18.attention.attention.value.weight", "vit.encoder.layer.18.attention.attention.value.bias", "vit.encoder.layer.18.attention.output.dense.weight", "vit.encoder.layer.18.attention.output.dense.bias", "vit.encoder.layer.18.intermediate.dense.weight", "vit.encoder.layer.18.intermediate.dense.bias", "vit.encoder.layer.18.output.dense.weight", "vit.encoder.layer.18.output.dense.bias", "vit.encoder.layer.18.layernorm_before.weight", "vit.encoder.layer.18.layernorm_before.bias", "vit.encoder.layer.18.layernorm_after.weight", "vit.encoder.layer.18.layernorm_after.bias", "vit.encoder.layer.19.attention.attention.query.weight", "vit.encoder.layer.19.attention.attention.query.bias", "vit.encoder.layer.19.attention.attention.key.weight", "vit.encoder.layer.19.attention.attention.key.bias", "vit.encoder.layer.19.attention.attention.value.weight", "vit.encoder.layer.19.attention.attention.value.bias", "vit.encoder.layer.19.attention.output.dense.weight", "vit.encoder.layer.19.attention.output.dense.bias", "vit.encoder.layer.19.intermediate.dense.weight", "vit.encoder.layer.19.intermediate.dense.bias", "vit.encoder.layer.19.output.dense.weight", "vit.encoder.layer.19.output.dense.bias", "vit.encoder.layer.19.layernorm_before.weight", "vit.encoder.layer.19.layernorm_before.bias", "vit.encoder.layer.19.layernorm_after.weight", "vit.encoder.layer.19.layernorm_after.bias", "vit.encoder.layer.20.attention.attention.query.weight", "vit.encoder.layer.20.attention.attention.query.bias", "vit.encoder.layer.20.attention.attention.key.weight", "vit.encoder.layer.20.attention.attention.key.bias", "vit.encoder.layer.20.attention.attention.value.weight", "vit.encoder.layer.20.attention.attention.value.bias", "vit.encoder.layer.20.attention.output.dense.weight", "vit.encoder.layer.20.attention.output.dense.bias", "vit.encoder.layer.20.intermediate.dense.weight", "vit.encoder.layer.20.intermediate.dense.bias", "vit.encoder.layer.20.output.dense.weight", "vit.encoder.layer.20.output.dense.bias", "vit.encoder.layer.20.layernorm_before.weight", "vit.encoder.layer.20.layernorm_before.bias", "vit.encoder.layer.20.layernorm_after.weight", "vit.encoder.layer.20.layernorm_after.bias", "vit.encoder.layer.21.attention.attention.query.weight", "vit.encoder.layer.21.attention.attention.query.bias", "vit.encoder.layer.21.attention.attention.key.weight", "vit.encoder.layer.21.attention.attention.key.bias", "vit.encoder.layer.21.attention.attention.value.weight", "vit.encoder.layer.21.attention.attention.value.bias", "vit.encoder.layer.21.attention.output.dense.weight", "vit.encoder.layer.21.attention.output.dense.bias", "vit.encoder.layer.21.intermediate.dense.weight", "vit.encoder.layer.21.intermediate.dense.bias", "vit.encoder.layer.21.output.dense.weight", "vit.encoder.layer.21.output.dense.bias", "vit.encoder.layer.21.layernorm_before.weight", "vit.encoder.layer.21.layernorm_before.bias", "vit.encoder.layer.21.layernorm_after.weight", "vit.encoder.layer.21.layernorm_after.bias", "vit.encoder.layer.22.attention.attention.query.weight", "vit.encoder.layer.22.attention.attention.query.bias", "vit.encoder.layer.22.attention.attention.key.weight", "vit.encoder.layer.22.attention.attention.key.bias", "vit.encoder.layer.22.attention.attention.value.weight", "vit.encoder.layer.22.attention.attention.value.bias", "vit.encoder.layer.22.attention.output.dense.weight", "vit.encoder.layer.22.attention.output.dense.bias", "vit.encoder.layer.22.intermediate.dense.weight", "vit.encoder.layer.22.intermediate.dense.bias", "vit.encoder.layer.22.output.dense.weight", "vit.encoder.layer.22.output.dense.bias", "vit.encoder.layer.22.layernorm_before.weight", "vit.encoder.layer.22.layernorm_before.bias", "vit.encoder.layer.22.layernorm_after.weight", "vit.encoder.layer.22.layernorm_after.bias", "vit.encoder.layer.23.attention.attention.query.weight", "vit.encoder.layer.23.attention.attention.query.bias", "vit.encoder.layer.23.attention.attention.key.weight", "vit.encoder.layer.23.attention.attention.key.bias", "vit.encoder.layer.23.attention.attention.value.weight", "vit.encoder.layer.23.attention.attention.value.bias", "vit.encoder.layer.23.attention.output.dense.weight", "vit.encoder.layer.23.attention.output.dense.bias", "vit.encoder.layer.23.intermediate.dense.weight", "vit.encoder.layer.23.intermediate.dense.bias", "vit.encoder.layer.23.output.dense.weight", "vit.encoder.layer.23.output.dense.bias", "vit.encoder.layer.23.layernorm_before.weight", "vit.encoder.layer.23.layernorm_before.bias", "vit.encoder.layer.23.layernorm_after.weight", "vit.encoder.layer.23.layernorm_after.bias", "vit.layernorm.weight", "vit.layernorm.bias", "classifier.weight", "classifier.bias". 

the google model has 392 layers whereas the timm model has 296 layers

[rwightman]

The weights are not compatible as the code is different, but it's possible to remap the state dict keys to load one in the other,

The google/vit-large-patch16-224 were originally adapted from the initial timm ports of the first set of Google vit weights, see https://github.com/huggingface/transformers/blob/main/src/transformers/models/vit/convert_vit_timm_to_pytorch.py ... you could reverse that mapping to go from transformers -> timm

And yeah, the 21k classifier would not work well for imagenet-1k (assuming you filtered out just the imagenet-1k classes)

EDIT: the # of layers differs because transformers did q, k, v as separate layers and timm merged them into one qkv