HaMuCo: Hand Pose Estimation via Multiview Collaborative Self-Supervised Learning

Xiaozheng Zheng^1,2†  Chao Wen^2†  Zhou Xue²  Pengfei Ren^1,2  Jingyu Wang^1*

¹Beijing University of Posts and Telecommunications   ²PICO IDL ByteDance  

^†Equal contribution   ^*Corresponding author

🤩 Accepted to ICCV 2023

---

HaMuCo is a multi-view self-supervised 3D hand pose estimation method that only requires 2D pseudo labels for training.

---

[Project Page] • [arXiv]

🔲 TODO

• FreiHAND evaluation code

• Multi-view inference code

📣 Updates

[07/2023] HaMuCo is accepted to ICCV 2023 🥳!

[01/2023] Training and evaluation codes on HanCo are released.

📁 Data Preparation

1. Download the HanCo dataset from the official website.

• https://lmb.informatik.uni-freiburg.de/data/HanCo/HanCo_rgb.zip
• https://lmb.informatik.uni-freiburg.de/data/HanCo/HanCo_xyz.zip
• https://lmb.informatik.uni-freiburg.de/data/HanCo/HanCo_shape.zip
• https://lmb.informatik.uni-freiburg.de/data/HanCo/HanCo_calib_meta.zip
• https://lmb.informatik.uni-freiburg.de/data/HanCo/HanCo_rgb_merged.zip

2. We provide the 2D pseudo labels generated from OpenPose in ./data/HanCo/HaMuCo_*.zip.

3. Unzip files and organize the data as follows:

${ROOT}  
|-- data  
|   |-- HanCo
|   |   |-- calib
|   |   |-- rgb 
|   |   |-- rgb_2d_keypoints
|   |   |-- rgb_merged
|   |   |-- xyz

🖥️ Installation

Requirements

• Python=3.7
• PyTorch=1.9.1+cu111
• torchgeometry (need some slight changes following here.)

Setup with Conda

conda create -n hamuco python=3.7
pip install torch==1.9.1+cu111 torchvision==0.10.1+cu111 torchaudio==0.9.1 -f https://download.pytorch.org/whl/torch_stable.html
cd HaMuCo
pip install -r ./requirements.txt

🏃‍♀️ Training

1. Run ./train.py to train and evaluate on the HanCo dataset.

🤟 Citation

If you find our work useful for your research, please consider citing the paper:

@inproceedings{
  zheng2023hamuco,
  title={HaMuCo: Hand Pose Estimation via Multiview Collaborative Self-Supervised Learning},
  author={Zheng, Xiaozheng and Wen, Chao and Xue, Zhou and Ren, Pengfei and Wang, Jingyu},
  booktitle={Proceedings of the IEEE/CVF international conference on computer vision},
  year={2023}
}

🗞️ License

Distributed under the MIT License. See LICENSE for more information.

🙌 Acknowledgements

The pytorch implementation of MANO is based on manopth. We thank the authors for their great job!