Skip to content

Latest commit

 

History

History
76 lines (56 loc) · 4.83 KB

README.md

File metadata and controls

76 lines (56 loc) · 4.83 KB

DropPos: Pre-Training Vision Transformers by Reconstructing Dropped Positions

Official Implementation of our paper "DropPos: Pre-Training Vision Transformers by Reconstructing Dropped Positions. In NeurIPS 2023.

by Haochen Wang, Junsong Fan, Yuxi Wang, Kaiyou Song, Tong Wang, and Zhaoxiang Zhang

[arXiv] [Paper]

🔔🔔🔔 We are happy to announce that DropPos has been accepted by NeurIPS 2023! 🔔🔔🔔

🔔🔔🔔 The pre-trained and fine-tuned models are available here with fetch code 4gik! 🔔🔔🔔

Notes

Motivation

TL;DR We present a novel self-supervised pre-text task to pre-train vision transformers, i.e., reconstructing dropped positions (DropPos), which achieved competitive results on various evaluation protocols, such as image classification, object detection, and semantic segmentation.

Abstract. As it is empirically observed that Vision Transformers (ViTs) are quite insensitive to the order of input tokens, the need for an appropriate self-supervised pretext task that enhances the location awareness of ViTs is becoming evident. To address this, we present DropPos, a novel pretext task designed to reconstruct Dropped Positions. The formulation of DropPos is simple: we first drop a large random subset of positional embeddings and then the model classifies the actual position for each non-overlapping patch among all possible positions solely based on their visual appearance. To avoid trivial solutions, we increase the difficulty of this task by keeping only a subset of patches visible. Additionally, considering there may be different patches with similar visual appearances, we propose position smoothing and attentive reconstruction strategies to relax this classification problem, since it is not necessary to reconstruct their exact positions in these cases. Empirical evaluations of DropPos show strong capabilities. DropPos outperforms supervised pre-training and achieves competitive results compared with state-of-the-art self-supervised alternatives on a wide range of downstream benchmarks. This suggests that explicitly encouraging spatial reasoning abilities, as DropPos does, indeed contributes to the improved location awareness of ViTs.

Method

Results

Method Model PT Epochs Top-1 Acc.
DropPos ViT-B/16 200 83.0
DropPos ViT-B/16 800 84.2
DropPos ViT-L/16 800 85.8

Acknowledgement

The pretraining and finetuning of our project are based on DeiT, MAE and HPM. Thanks for their wonderful work.

For object detection and semantic segmentation, please refer to Detectron2 and MMSegmentation, respectively. The configurations can be found in here and here for detection and segmentation, respectively.

License

This project is under the Apache License 2.0 license. See LICENSE for details.

Citation

@article{wang2023droppos,
  author  = {Wang, Haochen and Fan, Junsong and Wang, Yuxi and Song, Kaiyou and Wang, Tong and Zhang, Zhaoxiang},
  journal = {Advances in Neural Information Processing Systems (NeurIPS)},
  title   = {DropPos: Pre-Training Vision Transformers by Reconstructing Dropped Positions},
  year    = {2023},
}