Collection of papers and other resources for object detection and tracking using deep learning

Object Detection

• Mask R-CNN (pdf, arxiv, github) by Facebook AI Research!
  • Summary goes here...
• Tensorflow object detection API: https://github.com/tensorflow/models/tree/master/object_detection. Only the two SSD nets can run at 12.5 FPS on one GTX 1080 TI (less accurate than YOLO 604x604). Next two models at 4-5 FPS (4-5% mAP better than YOLO). Best model < 1 FPS. Currently code only allow inference of 1 image at a time. Speed might improve by 2.5 times when they allow multiple image inference.

Object Tracking

• Multi Object Tracking

  • Learning to Track: Online Multi-object Tracking by Decision Making (ICCV 2015) (Stanford) (pdf, github (Matlab), project page)

    • Summary

  • Tracking The Untrackable: Learning To Track Multiple Cues with Long-Term Dependencies (arxiv April 2017) (Stanford) (pdf, arxiv, project page)

    • Summary

  • Near-Online Multi-target Tracking with Aggregated Local Flow Descriptor (ICCV 2015) (NEC Labs) (pdf, author page)

    • Summary

  • A Multi-cut Formulation for Joint Segmentation and Tracking of Multiple Objects (highest MT on MOT2015) (University of Freiburg, Germany) (pdf, arxiv, author page)

    • Summary

• Single Object Tracking

  • Deep Reinforcement Learning for Visual Object Tracking in Videos (arxiv April 2017) (USC-Santa Barbara, Samsung Research) (pdf, arxiv, author page)

    • Summary

  • Visual Tracking by Reinforced Decision Making (arxiv February 2017) (Seoul National University, Chung-Ang University) (pdf, arxiv, author page)

    • Summary

  • Action-Decision Networks for Visual Tracking with Deep Reinforcement Learning (CVPR 2017) (Seoul National University) (pdf, project page)

    • Summary

  • End-to-end Active Object Tracking via Reinforcement Learning (arxiv 30 May 2017) (Peking University, Tencent AI Lab) (pdf, arxiv

Other potentially useful papers

• Deep Feature Flow for Video Recognition (pdf, arxiv, github) by Microsoft Research
  • Summary goes here...

Datasets

• IDOT dataset
• UA-DETRAC Benchmark Suite
• GRAM Road-Traffic Monitoring
• Stanford Drone Dataset
• Ko-PER Intersection Dataset
• TRANCOS Dataset
• Urban Tracker Dataset
• DARPA VIVID / PETS 2005 dataset (Non stationary camera)
• KIT-AKS Dataset (No ground truth)
• CBCL StreetScenes Challenge Framework (No top down viewpoint)
• MOT 2015 (mostly street level camera viewpoint)
• MOT 2016 (mostly street level camera viewpoint)
• MOT 2017 (mostly street level camera viewpoint)
• PETS 2009 (No vehicles)
• PETS 2017 (Low density; mostly pedestrians)
• KITTI Tracking Dataset (No top down viewpoint; non stationary camera)

Resources

• List of traffic surveillance datasets
• List of deep learning based tracking papers
• List of multi object tracking papers
• List of single object trackers with results on OTB
• List of Matlab frameworks, libraries and software

Tutorials

• Deep Reinforcement Learning: Pong from Pixels
• Demystifying Deep Reinforcement Learning

Code

• Multi Object Tracking
  • Combined Image- and World-Space Tracking in Traffic Scenes (ICRA 2017)[C++]
  • Learning to Track: Online Multi-Object Tracking by Decision Making (ICCV 2015)[MATLAB]
  • Multiple Hypothesis Tracking Revisited (ICCV 2015) (highest MT on MOT2015 among open source trackers)[MATLAB]
  • Joint Tracking and Segmentation of Multiple Targets (CVPR 2015)[MATLAB]
  • High-Speed Tracking-by-Detection Without Using Image Information (AVSS 2017)[Python]
  • Continuous Energy Minimization for Multitarget Tracking (TPAMI 2014 / CVPR 2011 / ICCV 2011)[MATLAB]
• Single Object Tracking
  • Detect to Track and Track to Detect (ICCV 2017)[MATLAB]
  • DeepTracking: Seeing Beyond Seeing Using Recurrent Neural Networks (AAAI 2016)[Torch 7]
  • Hierarchical Convolutional Features for Visual Tracking (ICCV 2015)[Matlab]
  • Learning Multi-Domain Convolutional Neural Networks for Visual Tracking (Winner of The VOT2015 Challenge)[Matlab/MatConvNet]
  • RATM: Recurrent Attentive Tracking Model[Python]
  • Visual Tracking with Fully Convolutional Networks (ICCV 2015)[Matlab]
  • Fully-Convolutional Siamese Networks for Object Tracking[Tensor flow]
  • ROLO : Spatially Supervised Recurrent Convolutional Neural Networks for Visual Object Tracking[Tensor flow]
  • Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking (ECCV 2016)[MATLAB]
  • ECO: Efficient Convolution Operators for Tracking (CVPR 2017)[MATLAB]
  • End-to-end representation learning for Correlation Filter based tracking (CVPR 21017)[MATLAB]
  • A collection of common tracking algorithms
• Static Object Detection and Matching
  • Matchnet
  • Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches
  • Asynchronous Methods for Deep Reinforcement Learning
  • YOLO9000: Better, Faster, Stronger - Real-Time Object Detection. 9000 classes!
  • Deformable Convolutional Networks
  • R-FCN: Object Detection via Region-based Fully Convolutional Networks
  • PVANet: Lightweight Deep Neural Networks for Real-time Object Detection
  • Mask RCNN in TensorFlow
• Video Object Detection
  • TPN: Tubelet Proposal Network (CVPR 2017)[Python]
  • T-CNN: Tubelets with Convolution Neural Networks (CVPR 2016)[Python]
  • Flow-Guided Feature Aggregation for Video Object Detection (NIPS 2016 / ICCV 2017)[Python/CUDA]
• Optical Flow
  • Fast Optical Flow using Dense Inverse Search (DIS)
  • FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks
• Misc
  • Darknet: Convolutional Neural Networks
  • RNNexp