Bad performance on ObjectNet

[LixDemon]

I run your code and got a similar result with the paper except ObjectNet. The top-1 acc on ObjectNet is only 54.7%. I found the images in my ObjectNet is 50273 but not 50000. I downloaded the ObjectNet from the official url https://www.dropbox.com/s/raw/cxeztdtm16nzvuw/objectnet-1.0.zip and unzip it. Do you have any idea why this happened?

[SachinG007]

Hi,
Thanks for pointing this out. I just checked my dataset, and it has 23.7k images only, probably some issue would have happened when I downloaded. I will update the paper with the correct numbers for ObjectNet once I redownload and re-evaluate.
This issue would be their for all the baselines as well. So shouldn't affect any conclusions or ordering of various finetuning approaches.
Thanks again for pointing this out.

[LixDemon]

Really glad receive your reply! Another question I found is that when I evaluate on ObjectNet, not all images but only 18574 images of 113 classes are utilized. Is there something wrong with the code in ObjectNet?

[SachinG007]

The code uses ImageNet data loader from folders (https://pytorch.org/vision/stable/generated/torchvision.datasets.ImageFolder.html). Can you check how many class folders are there in the local data path?

[LixDemon]

There are 50273 images of 313 class folders in the local data path. Hope to see your revised results in your paper! Thanks!

[BierOne]

Hi, @SachinG007

I also have the same problem. If you already have any scores related to this, would you mind sharing them here? Thank you so much.

[SachinG007]

Hi @BierOne ,
I haven't yet got a chance to rerun the numbers. I can try though getting numbers by late next week.

[BierOne]

Glad to hear that! Thanks :)

[LixDemon]

Have you rerun the objectnet experiments yet? I got an acc of 55.4% for ZSL and 54.7% for FLYP. I can't match the performance of other methods (i.e. FT, LP, LP-FT), too.

[Simon4Yan]

@LixDemon May I know if you could share your trained weight on ImageNet? It seems this repository does not provide it. Thank you!

[SachinG007]

@LixDemon , I didn't yet get a chance to rerun the ImageNet experiments. I will get back on this. @Simon4Yan , I can upload the model once I rerun it.

Thanks

[SachinG007]

Hi @LixDemon and @Simon4Yan ,
I am uploading a FLYP finetuned (on ImageNet) checkpoint of CLIP (ViT-B-16) here.

The updated FLYP and Zeroshot numbers for ObjectNet:
Zeroshot:
Evaluating on ObjectNet
ObjectNet Top-1 accuracy: 0.5333

Finetuned:
Evaluating on ObjectNet
ObjectNet Top-1 accuracy: 0.5499

I will also update the paper once I get the numbers for the baselines as well.
Thanks.

[LixDemon]

Hi @SachinG007,
Really glad to hear that! Can't wait to see the revised paper.
Thanks for your detailed reply!

[Simon4Yan]

@SachinG007 Thank you for sharing the model weights. Also, I am interested in the revised version. Kind Regards!