👏👏👏👏👏👏👏👏👏👏👏 QueryPose is a sparse end-to-end multi-person pose regression framework:
QueryPose: Sparse Multi-Person Pose Regression via Spatial-Aware Part-Level Query,
Yabo Xiao, Kai Su, Xiaojuan Wang, Dongdong Yu, Lei Jin, Mingshu He, Zehuan Yuan;
Published on NeurIPS 2022
-
Simple: QueryPose is sparse end-to-end multi-person pose regression framework, which can directly predict multi-person keypoint sequences from the input image. Our method is built upon Sparse RCNN without transformer encoder.
-
Generalizability: QueryPose is able to achieve the competitive performance on diverise scenes.
-
Fast convergence: Achieving 60+ AP with only several training epochs (about 6 epochs). We will release the 1x model with competitive performance.
The pretrain model of HRNet-series can be downloaded from HRNet-pretrain. The swin-series can be downloaded from Swin Transformer
The results on MS COCO mini-val with 100 queries; We provide the light version models (without dynamic MLP in keypoint decoder), whose results are slightly lower than the model with dynamic MLP in keypoint decoder. One can reproduce the paper results when set LIGHT_VERSION to False.
| Backbone | keypoint AP | keypoint AP * | Times(ms) | download |
|---|---|---|---|---|
| HRNet32_100pro_3x | 69.8 | 72.3 | 97 | model |
| HRNet48_100pro_3x | 71.0 | 73.4 | 101 | model |
- The time is calculated on Tesla A100, Other GPU cards are not fully tested.
- For QueryPose, we leverage the heatmap-pretrain model for HRNet-series backbone instead of using online auxiliary heatmap learning in other regression-based methods. We observe that the online auxiliary heatmap learning is useless for our method.
The codebases are built on top of Detectron2 and Sparse RCNN. If you have any environment or compilation issues, please refer to them.
- Install and compile
git clone [email protected]:buptxyb666/QueryPose.git
cd QueryPose
python setup.py build develop
pip install timm
- Link coco dataset path to QueryPose/datasets/coco
mkdir -p datasets/coco
ln -s /path_to_coco_dataset/annotations datasets/coco/annotations
ln -s /path_to_coco_dataset/train2017 datasets/coco/train2017
ln -s /path_to_coco_dataset/val2017 datasets/coco/val2017
- Train QueryPose
python projects/querypose/train_net.py --num-gpus 8 \
--config-file projects/querypose/configs/querypose.hrnet32.100pro.3x.yaml
- Evaluate QueryPose
python projects/querypose/train_net.py --num-gpus 8 \
--config-file projects/querypose/configs/querypose.hrnet32.100pro.3x.yaml \
--eval-only MODEL.WEIGHTS path/to/model.pth
If you find this project useful for your research, please use the following BibTeX entry:
@inproceedings{xiaoquerypose,
title={QueryPose: Sparse Multi-Person Pose Regression via Spatial-Aware Part-Level Query},
author={Xiao, Yabo and Su, Kai and Wang, Xiaojuan and Yu, Dongdong and Jin, Lei and He, Mingshu and Yuan, Zehuan},
booktitle={Advances in Neural Information Processing Systems}
}