Training details on data augmentation

[zaiweizhang]

Hi!

This work is very impressive. I have a question regarding the training pipeline for the student model with pseudo labels:

Are you following the exact same loss for the student training compared with V1?
That means: applying the data augmentation (color distortion, gaussian blurring, CutMix) during training.

[LiheYoung]

Yes. But in V2, we find that when training smaller models (e.g., ViT-S and ViT-B-based models) with the pseudo label from the largest ViT-G-based model, the augmentations are not necessary.