English | 简体中文

RT-DETR: DETRs Beat YOLOs on Real-time Object Detection

---

This is the official implementation of papers

• DETRs Beat YOLOs on Real-time Object Detection
• RT-DETRv2: Improved Baseline with Bag-of-Freebies for Real-Time Detection Transformer

Fig

🚀 Updates

• [2024.09.23] Added Backbone Support for Regnet and DLA34.
• [2024.08.27] Add hubconf.py file to support torch hub.
• [2024.08.22] Improve the performance of ✅ RT-DETRv2-S to 48.1 mAP (+1.6 compared to RT-DETR-R18).
• [2024.07.24] Release ✅ RT-DETRv2!
• [2024.02.27] Our work has been accepted to CVPR 2024!
• [2024.01.23] Fix difference on data augmentation with paper in rtdetr_pytorch #84.
• [2023.11.07] Add pytorch ✅ rtdetr_r34vd for requests #107, #114.
• [2023.11.05] Upgrade the logic of remap_mscoco_category to facilitate training of custom datasets, see detils in Train custom data part. #81.
• [2023.10.23] Add discussion for deployments, supported onnxruntime, TensorRT, openVINO.
• [2023.10.12] Add tuning code for pytorch version, now you can tuning rtdetr based on pretrained weights.
• [2023.09.19] Upload ✅ pytorch weights convert from paddle version.
• [2023.08.24] Release RT-DETR-R18 pretrained models on objects365. 49.2 mAP and 217 FPS.
• [2023.08.22] Upload ✅ rtdetr_pytorch source code. Please enjoy it!
• [2023.08.15] Release RT-DETR-R101 pretrained models on objects365. 56.2 mAP and 74 FPS.
• [2023.07.30] Release RT-DETR-R50 pretrained models on objects365. 55.3 mAP and 108 FPS.
• [2023.07.28] Fix some bugs, and add some comments. 1, 2.
• [2023.07.13] Upload ✅ training logs on coco.
• [2023.05.17] Release RT-DETR-R18, RT-DETR-R34, RT-DETR-R50-m（example for scaled).
• [2023.04.17] Release RT-DETR-R50, RT-DETR-R101, RT-DETR-L, RT-DETR-X.

📍 Implementations

• 🔥 RT-DETRv2
  • paddle: code&weight
  • pytorch: code&weight
• 🔥 RT-DETR
  • paddle: code&weight
  • pytorch: code&weight

Model	Input shape	Dataset	$AP^{val}$	$AP^{val}_{50}$	Params(M)	FLOPs(G)	T4 TensorRT FP16(FPS)
RT-DETR-R18	640	COCO	46.5	63.8	20	60	217
RT-DETR-R34	640	COCO	48.9	66.8	31	92	161
RT-DETR-R50-m	640	COCO	51.3	69.6	36	100	145
RT-DETR-R50	640	COCO	53.1	71.3	42	136	108
RT-DETR-R101	640	COCO	54.3	72.7	76	259	74
RT-DETR-HGNetv2-L	640	COCO	53.0	71.6	32	110	114
RT-DETR-HGNetv2-X	640	COCO	54.8	73.1	67	234	74
RT-DETR-R18	640	COCO + Objects365	49.2	66.6	20	60	217
RT-DETR-R50	640	COCO + Objects365	55.3	73.4	42	136	108
RT-DETR-R101	640	COCO + Objects365	56.2	74.6	76	259	74
RT-DETRv2-S	640	COCO	48.1 (+1.6)	65.1	20	60	217
RT-DETRv2-M*	640	COCO	49.9 (+1.0)	67.5	31	92	161
RT-DETRv2-M	640	COCO	51.9 (+0.6)	69.9	36	100	145
RT-DETRv2-L	640	COCO	53.4 (+0.3)	71.6	42	136	108
RT-DETRv2-X	640	COCO	54.3	72.8 (+0.1)	76	259	74

Notes:

• COCO + Objects365 in the table means finetuned model on COCO using pretrained weights trained on Objects365.

🦄 Performance

🏕️ Complex Scenarios

🌋 Difficult Conditions

Citation

If you use RT-DETR or RTDETRv2 in your work, please use the following BibTeX entries:

@misc{lv2023detrs,
      title={DETRs Beat YOLOs on Real-time Object Detection},
      author={Yian Zhao and Wenyu Lv and Shangliang Xu and Jinman Wei and Guanzhong Wang and Qingqing Dang and Yi Liu and Jie Chen},
      year={2023},
      eprint={2304.08069},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

@misc{lv2024rtdetrv2improvedbaselinebagoffreebies,
      title={RT-DETRv2: Improved Baseline with Bag-of-Freebies for Real-Time Detection Transformer}, 
      author={Wenyu Lv and Yian Zhao and Qinyao Chang and Kui Huang and Guanzhong Wang and Yi Liu},
      year={2024},
      eprint={2407.17140},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2407.17140}, 
}