Awesome Interaction-aware Behavior and Trajectory Prediction

This is a checklist of state-of-the-art research materials (datasets, blogs, papers and public codes) related to trajectory prediction. Wish it could be helpful for both academia and industry. (Still updating)

Maintainers: Jiachen Li, Hengbo Ma, Jinning Li (University of California, Berkeley)

Emails: {jiachen_li, hengbo_ma, jinning_li}@berkeley.edu

Please feel free to pull request to add new resources or send emails to us for questions, discussion and collaborations.

Also welcome to check the current research in our MSC Lab at UC Berkeley.

Research Intern: Please read this if you want to apply for research intern opportunities in our group.

Note: Here is also a collection of materials for reinforcement learning, decision making and motion planning.

Please cite our work if you found this useful:

@inproceedings{Jiachen_IROS19,
  title={Conditional Generative Neural System for Probabilistic Trajectory Prediction},
  author={Li, Jiachen and Ma, Hengbo and Tomizuka, Masayoshi},
  booktitle={in 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
  year={2019},
  organization={IEEE}
}

@inproceedings{Jiachen_ICRA19,
  title={Interaction-aware Multi-agent Tracking and Probabilistic Behavior Prediction via Adversarial Learning},
  author={Li, Jiachen* and Ma, Hengbo* and Tomizuka, Masayoshi},
  booktitle={in 2019 IEEE International Conference on Robotics and Automation (ICRA)},
  year={2019},
  organization={IEEE}
}

Table of Contents

• Datasets
  • Vehicles and Traffic
  • Pedestrians
  • Sport Players
• Literature and Codes
  • Survey Papers
  • Physics Systems with Interaction
  • Intelligent Vehicles & Traffic
  • Mobile Robots
  • Pedestrians
  • Vehicle-Pedestrians Interaction
  • Sport Players
  • Benchmark and Evaluation Metrics
  • Others

Datasets

Vehicles and Traffic

Dataset	Agents	Scenarios	Sensors
INTERACTION	Vehicles / cyclists/ people	Roundabout / intersection	Camera
KITTI	Vehicles / cyclists/ people	Highway / rural areas	Camera / LiDAR
HighD	Vehicles	Highway	Camera
NGSIM	Vehicles	Highway	Camera
Cyclists	Cyclists	Urban	Camera
nuScenes	Vehicles	Urban	Camera / LiDAR / RADAR
BDD100k	Vehicles / cyclists / people	Highway / urban	Camera
Apolloscapes	Vehicles / cyclists / people	Urban	Camera
Udacity	Vehicles	Urban	Camera
Cityscapes	Vehicles/ people	Urban	Camera
Stanford Drone	Vehicles / cyclists/ people	Urban	Camera
Argoverse	Vehicles / people	Urban	Camera / LiDAR
TRAF	Vehicles/buses/cyclists/bikes / people/animals	Urban	Camera
Lyft Level 5	Vehicles/cyclists/people	Urban	Camera/ LiDAR

Pedestrians

Dataset	Agents	Scenarios	Sensors
UCY	People	Zara / students	Camera
ETH	People	Urban	Camera
VIRAT	People / vehicles	Urban	Camera
KITTI	Vehicles / cyclists/ people	Highway / rural areas	Camera / LiDAR
ATC	People	Shopping center	Range sensor
Daimler	People	From moving vehicle	Camera
Central Station	People	Inside station	Camera
Town Center	People	Urban street	Camera
Edinburgh	People	Urban	Camera
Cityscapes	Vehicles/ people	Urban	Camera
Argoverse	Vehicles / people	Urban	Camera / LiDAR
Stanford Drone	Vehicles / cyclists/ people	Urban	Camera
TrajNet	People	Urban	Camera
PIE	People	Urban	Camera
ForkingPaths	People	Urban / Simulation	Camera

Sport Players

Dataset	Agents	Scenarios	Sensors
Football	People	Football field	Camera
NBA SportVU	People	Basketball Hall	Camera
NFL	People	American Football	Camera

Literature and Codes

Survey Papers

• Human Motion Trajectory Prediction: A Survey, 2019 [paper]
• A literature review on the prediction of pedestrian behavior in urban scenarios, ITSC 2018. [paper]
• Survey on Vision-Based Path Prediction. [paper]
• Autonomous vehicles that interact with pedestrians: A survey of theory and practice. [paper]
• Trajectory data mining: an overview. [paper]
• A survey on motion prediction and risk assessment for intelligent vehicles. [paper]

Physics Systems with Interaction

• The Trajectron: Probabilistic Multi-Agent Trajectory Modeling With Dynamic Spatiotemporal Graphs, ICCV 2019. [paper] [code]
• Interaction Templates for Multi-Robot Systems, IROS 2019. [paper]
• Factorised Neural Relational Inference for Multi-Interaction Systems, ICML workshop 2019. [paper] [code]
• Physics-as-Inverse-Graphics: Joint Unsupervised Learning of Objects and Physics from Video, 2019. [paper]
• Neural Relational Inference for Interacting Systems, ICML 2018. [paper] [code]
• Unsupervised Learning of Latent Physical Properties Using Perception-Prediction Networks, UAI 2018. [paper]
• Relational inductive biases, deep learning, and graph networks, 2018. [paper]
• Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions, ICLR 2018. [paper]
• Graph networks as learnable physics engines for inference and control, ICML 2018. [paper]
• Flexible Neural Representation for Physics Prediction, 2018. [paper]
• A simple neural network module for relational reasoning, 2017. [paper]
• VAIN: Attentional Multi-agent Predictive Modeling, NIPS 2017. [paper]
• Visual Interaction Networks, 2017. [paper]
• A Compositional Object-Based Approach to Learning Physical Dynamics, ICLR 2017. [paper]
• Interaction Networks for Learning about Objects, Relations and Physics, 2016. [paper][code]

Intelligent Vehicles & Traffic

• Kernel Trajectory Maps for Multi-Modal Probabilistic Motion Prediction, CoRL 2019. [paper] [code]
• EvolveGraph: Heterogeneous Multi-Agent Multi-Modal Trajectory Prediction with Evolving Interaction Graphs, 2019. [paper]
• Social-WaGDAT: Interaction-aware Trajectory Prediction via Wasserstein Graph Double-Attention Network, 2019. [paper]
• Forecasting Trajectory and Behavior of Road-Agents Using Spectral Clustering in Graph-LSTMs, 2019. [paper] [code]
• Joint Prediction for Kinematic Trajectories in Vehicle-Pedestrian-Mixed Scenes, ICCV 2019. [paper]
• Analyzing the Variety Loss in the Context of Probabilistic Trajectory Prediction, ICCV 2019. [paper]
• Looking to Relations for Future Trajectory Forecast, ICCV 2019. [paper]
• Jointly Learnable Behavior and Trajectory Planning for Self-Driving Vehicles, IROS 2019. [paper]
• Sharing Is Caring: Socially-Compliant Autonomous Intersection Negotiation, IROS 2019. [paper]
• INFER: INtermediate Representations for FuturE PRediction, IROS 2019. [paper] [code]
• Deep Predictive Autonomous Driving Using Multi-Agent Joint Trajectory Prediction and Traffic Rules, IROS 2019. [paper]
• NeuroTrajectory: A Neuroevolutionary Approach to Local State Trajectory Learning for Autonomous Vehicles, IROS 2019. [paper]
• Urban Street Trajectory Prediction with Multi-Class LSTM Networks, IROS 2019. [N/A]
• Spatiotemporal Learning of Directional Uncertainty in Urban Environments with Kernel Recurrent Mixture Density Networks, IROS 2019. [paper]
• Conditional generative neural system for probabilistic trajectory prediction, IROS 2019. [paper]
• Interaction-aware multi-agent tracking and probabilistic behavior prediction via adversarial learning, ICRA 2019. [paper]
• Generic Tracking and Probabilistic Prediction Framework and Its Application in Autonomous Driving, IEEE Trans. Intell. Transport. Systems, 2019. [paper]
• Coordination and trajectory prediction for vehicle interactions via bayesian generative modeling, IV 2019. [paper]
• Wasserstein generative learning with kinematic constraints for probabilistic interactive driving behavior prediction, IV 2019. [paper]
• GRIP: Graph-based Interaction-aware Trajectory Prediction, ITSC 2019. [paper]
• AGen: Adaptable Generative Prediction Networks for Autonomous Driving, IV 2019. [paper]
• TraPHic: Trajectory Prediction in Dense and Heterogeneous Traffic Using Weighted Interactions, CVPR 2019. [paper], [code]
• Multi-Step Prediction of Occupancy Grid Maps with Recurrent Neural Networks, CVPR 2019. [paper]
• Argoverse: 3D Tracking and Forecasting With Rich Maps, CVPR 2019 [paper]
• Robust Aleatoric Modeling for Future Vehicle Localization, CVPR 2019. [paper]
• Pedestrian occupancy prediction for autonomous vehicles, IRC 2019. [paper]
• Context-based path prediction for targets with switching dynamics, 2019.[paper]
• Deep Imitative Models for Flexible Inference, Planning, and Control, 2019. [paper]
• Infer: Intermediate representations for future prediction, 2019. [paper][code]
• Multi-agent tensor fusion for contextual trajectory prediction, 2019. [paper]
• Context-Aware Pedestrian Motion Prediction In Urban Intersections, 2018. [paper]
• Generic probabilistic interactive situation recognition and prediction: From virtual to real, ITSC 2018. [paper]
• Generic vehicle tracking framework capable of handling occlusions based on modified mixture particle filter, IV 2018. [paper]
• Multi-Modal Trajectory Prediction of Surrounding Vehicles with Maneuver based LSTMs, 2018. [paper]
• Sequence-to-sequence prediction of vehicle trajectory via lstm encoder-decoder architecture, 2018. [paper]
• R2P2: A ReparameteRized Pushforward Policy for diverse, precise generative path forecasting, ECCV 2018. [paper]
• Predicting trajectories of vehicles using large-scale motion priors, IV 2018. [paper]
• Vehicle trajectory prediction by integrating physics-and maneuver based approaches using interactive multiple models, 2018. [paper]
• Motion Prediction of Traffic Actors for Autonomous Driving using Deep Convolutional Networks, 2018. [paper]
• Generative multi-agent behavioral cloning, 2018. [paper]
• Deep Sequence Learning with Auxiliary Information for Traffic Prediction, KDD 2018. [paper], [code]
• Multipolicy decision-making for autonomous driving via changepoint-based behavior prediction, 2017. [paper]
• Probabilistic long-term prediction for autonomous vehicles, IV 2017. [paper]
• Probabilistic vehicle trajectory prediction over occupancy grid map via recurrent neural network, ITSC 2017. [paper]
• Desire: Distant future prediction in dynamic scenes with interacting agents, CVPR 2017. [paper][code]
• Imitating driver behavior with generative adversarial networks, 2017. [paper][code]
• Infogail: Interpretable imitation learning from visual demonstrations, 2017. [paper][code]
• Long-term planning by short-term prediction, 2017. [paper]
• Long-term path prediction in urban scenarios using circular distributions, 2017. [paper]
• Deep learning driven visual path prediction from a single image, 2016. [paper]
• Understanding interactions between traffic participants based on learned behaviors, 2016. [paper]
• Visual path prediction in complex scenes with crowded moving objects, CVPR 2016. [paper]
• A game-theoretic approach to replanning-aware interactive scene prediction and planning, 2016. [paper]
• Intention-aware online pomdp planning for autonomous driving in a crowd, ICRA 2015. [paper]
• Online maneuver recognition and multimodal trajectory prediction for intersection assistance using non-parametric regression, 2014. [paper]
• Patch to the future: Unsupervised visual prediction, CVPR 2014. [paper]
• Mobile agent trajectory prediction using bayesian nonparametric reachability trees, 2011. [paper]

Mobile Robots

• Multimodal probabilistic model-based planning for human-robot interaction, ICRA 2018. [paper][code]
• Decentralized Non-communicating Multiagent Collision Avoidance with Deep Reinforcement Learning, ICRA 2017. [paper]
• Augmented dictionary learning for motion prediction, ICRA 2016. [paper]
• Predicting future agent motions for dynamic environments, ICMLA 2016. [paper]
• Bayesian intention inference for trajectory prediction with an unknown goal destination, IROS 2015. [paper]
• Learning to predict trajectories of cooperatively navigating agents, ICRA 2014. [paper]

Pedestrians

• Disentangling Human Dynamics for Pedestrian Locomotion Forecasting with Noisy Supervision, WACV 2020. [paper]
• EvolveGraph: Heterogeneous Multi-Agent Multi-Modal Trajectory Prediction with Evolving Interaction Graphs, 2020. [paper]
• Social-WaGDAT: Interaction-aware Trajectory Prediction via Wasserstein Graph Double-Attention Network, 2020. [paper]
• Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction, CVPR 2020. [Paper], [Code]
• The Garden of Forking Paths: Towards Multi-Future Trajectory Prediction, CVPR 2020. [paper], [code/dataset]
• Disentangling Human Dynamics for Pedestrian Locomotion Forecasting with Noisy Supervision, WACV 2020. [paper]
• STGAT: Modeling Spatial-Temporal Interactions for Human Trajectory Prediction, ICCV 2019. [paper] [code]
• Instance-Level Future Motion Estimation in a Single Image Based on Ordinal Regression, ICCV 2019. [paper]
• Social and Scene-Aware Trajectory Prediction in Crowded Spaces, ICCV workshop 2019. [paper] [code]
• Stochastic Sampling Simulation for Pedestrian Trajectory Prediction, IROS 2019. [paper]
• Long-Term Prediction of Motion Trajectories Using Path Homology Clusters, IROS 2019. [paper]
• StarNet: Pedestrian Trajectory Prediction Using Deep Neural Network in Star Topology, IROS 2019. [paper]
• Learning Generative Socially-Aware Models of Pedestrian Motion, IROS 2019. [paper]
• Situation-Aware Pedestrian Trajectory Prediction with Spatio-Temporal Attention Model, CVWW 2019. [paper]
• Path predictions using object attributes and semantic environment, VISIGRAPP 2019. [paper]
• Probabilistic Path Planning using Obstacle Trajectory Prediction, CoDS-COMAD 2019. [paper]
• Human Trajectory Prediction using Adversarial Loss, 2019. [paper]
• Social Ways: Learning Multi-Modal Distributions of Pedestrian Trajectories with GANs, CVPR 2019. [Precognition Workshop], [paper], [code]
• Peeking into the Future: Predicting Future Person Activities and Locations in Videos, CVPR 2019. [paper], [code]
• Learning to Infer Relations for Future Trajectory Forecast, CVPR 2019. [paper]
• TraPHic: Trajectory Prediction in Dense and Heterogeneous Traffic Using Weighted Interactions, CVPR 2019. [paper]
• Which Way Are You Going? Imitative Decision Learning for Path Forecasting in Dynamic Scenes, CVPR 2019. [paper]
• Overcoming Limitations of Mixture Density Networks: A Sampling and Fitting Framework for Multimodal Future Prediction, CVPR 2019. [paper][code]
• Sophie: An attentive gan for predicting paths compliant to social and physical constraints, CVPR 2019. [paper][code]
• Pedestrian path, pose, and intention prediction through gaussian process dynamical models and pedestrian activity recognition, 2019. [paper]
• Multimodal Interaction-aware Motion Prediction for Autonomous Street Crossing, 2019. [paper]
• The simpler the better: Constant velocity for pedestrian motion prediction, 2019. [paper]
• Pedestrian trajectory prediction in extremely crowded scenarios, 2019. [paper]
• Srlstm: State refinement for lstm towards pedestrian trajectory prediction, 2019. [paper]
• Location-velocity attention for pedestrian trajectory prediction, WACV 2019. [paper]
• Pedestrian Trajectory Prediction in Extremely Crowded Scenarios, Sensors, 2019. [paper]
• A data-driven model for interaction-aware pedestrian motion prediction in object cluttered environments, ICRA 2018. [paper]
• Move, Attend and Predict: An attention-based neural model for people’s movement prediction, Pattern Recognition Letters 2018. [paper]
• GD-GAN: Generative Adversarial Networks for Trajectory Prediction and Group Detection in Crowds, ACCV 2018, [paper], [demo]
• Ss-lstm: a hierarchical lstm model for pedestrian trajectory prediction, WACV 2018. [paper]
• Social Attention: Modeling Attention in Human Crowds, ICRA 2018. [paper][code]
• Pedestrian prediction by planning using deep neural networks, ICRA 2018. [paper]
• Joint long-term prediction of human motion using a planning-based social force approach, ICRA 2018. [paper]
• Human motion prediction under social grouping constraints, IROS 2018. [paper]
• Future Person Localization in First-Person Videos, CVPR 2018. [paper]
• Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks, CVPR 2018. [paper][code]
• Group LSTM: Group Trajectory Prediction in Crowded Scenarios, ECCV 2018. [paper]
• Mx-lstm: mixing tracklets and vislets to jointly forecast trajectories and head poses, CVPR 2018. [paper]
• Intent prediction of pedestrians via motion trajectories using stacked recurrent neural networks, 2018. [paper]
• Transferable pedestrian motion prediction models at intersections, 2018. [paper]
• Probabilistic map-based pedestrian motion prediction taking traffic participants into consideration, 2018. [paper]
• A Computationally Efficient Model for Pedestrian Motion Prediction, ECC 2018. [paper]
• Context-aware trajectory prediction, ICPR 2018. [paper]
• Set-based prediction of pedestrians in urban environments considering formalized traffic rules, ITSC 2018. [paper]
• Building prior knowledge: A markov based pedestrian prediction model using urban environmental data, ICARCV 2018. [paper]
• Depth Information Guided Crowd Counting for Complex Crowd Scenes, 2018. [paper]
• Tracking by Prediction: A Deep Generative Model for Mutli-Person Localisation and Tracking, WACV 2018. [paper]
• “Seeing is Believing”: Pedestrian Trajectory Forecasting Using Visual Frustum of Attention, WACV 2018. [paper]
• Long-Term On-Board Prediction of People in Traffic Scenes under Uncertainty, CVPR 2018. [paper], [code+data]
• Encoding Crowd Interaction with Deep Neural Network for Pedestrian Trajectory Prediction, CVPR 2018. [paper], [code]
• Walking Ahead: The Headed Social Force Model, 2017. [paper]
• Real-time certified probabilistic pedestrian forecasting, 2017. [paper]
• A multiple-predictor approach to human motion prediction, ICRA 2017. [paper]
• Forecasting interactive dynamics of pedestrians with fictitious play, CVPR 2017. [paper]
• Forecast the plausible paths in crowd scenes, IJCAI 2017. [paper]
• Bi-prediction: pedestrian trajectory prediction based on bidirectional lstm classification, DICTA 2017. [paper]
• Aggressive, Tense or Shy? Identifying Personality Traits from Crowd Videos, IJCAI 2017. [paper]
• Natural vision based method for predicting pedestrian behaviour in urban environments, ITSC 2017. [paper]
• Human Trajectory Prediction using Spatially aware Deep Attention Models, 2017. [paper]
• Soft + Hardwired Attention: An LSTM Framework for Human Trajectory Prediction and Abnormal Event Detection, 2017. [paper]
• Forecasting Interactive Dynamics of Pedestrians with Fictitious Play, CVPR 2017. [paper]
• Social LSTM: Human trajectory prediction in crowded spaces, CVPR 2016. [paper][code]
• Comparison and evaluation of pedestrian motion models for vehicle safety systems, ITSC 2016. [paper]
• Age and Group-driven Pedestrian Behaviour: from Observations to Simulations, 2016. [paper]
• Structural-RNN: Deep learning on spatio-temporal graphs, CVPR 2016. [paper][code]
• Intent-aware long-term prediction of pedestrian motion, ICRA 2016. [paper]
• Context-based detection of pedestrian crossing intention for autonomous driving in urban environments, IROS 2016. [paper]
• Novel planning-based algorithms for human motion prediction, ICRA 2016. [paper]
• Learning social etiquette: Human trajectory understanding in crowded scenes, ECCV 2016. [paper][code]
• GLMP-realtime pedestrian path prediction using global and local movement patterns, ICRA 2016. [paper]
• Knowledge transfer for scene-specific motion prediction, ECCV 2016. [paper]
• STF-RNN: Space Time Features-based Recurrent Neural Network for predicting People Next Location, SSCI 2016. [code]
• Goal-directed pedestrian prediction, ICCV 2015. [paper]
• Trajectory analysis and prediction for improved pedestrian safety: Integrated framework and evaluations, 2015. [paper]
• Predicting and recognizing human interactions in public spaces, 2015. [paper]
• Learning collective crowd behaviors with dynamic pedestrian-agents, 2015. [paper]
• Modeling spatial-temporal dynamics of human movements for predicting future trajectories, AAAI 2015. [paper]
• Unsupervised robot learning to predict person motion, ICRA 2015. [paper]
• A controlled interactive multiple model filter for combined pedestrian intention recognition and path prediction, ITSC 2015. [paper]
• Real-Time Predictive Modeling and Robust Avoidance of Pedestrians with Uncertain, Changing Intentions, 2014. [paper]
• Behavior estimation for a complete framework for human motion prediction in crowded environments, ICRA 2014. [paper]
• Pedestrian’s trajectory forecast in public traffic with artificial neural network, ICPR 2014. [paper]
• Will the pedestrian cross? A study on pedestrian path prediction, 2014. [paper]
• BRVO: Predicting pedestrian trajectories using velocity-space reasoning, 2014. [paper]
• Context-based pedestrian path prediction, ECCV 2014. [paper]
• Pedestrian path prediction using body language traits, 2014. [paper]
• Online maneuver recognition and multimodal trajectory prediction for intersection assistance using non-parametric regression, 2014. [paper]
• Learning intentions for improved human motion prediction, 2013. [paper]

Vehicle-Pedestrians Interaction

Sport Players

• EvolveGraph: Heterogeneous Multi-Agent Multi-Modal Trajectory Prediction with Evolving Interaction Graphs, 2019. [paper]
• Diverse Generation for Multi-Agent Sports Games, CVPR 2019. [paper]
• Stochastic Prediction of Multi-Agent Interactions from Partial Observations, ICLR 2019. [paper]
• Generating Multi-Agent Trajectories using Programmatic Weak Supervision, ICLR 2019. [paper]
• Generative Multi-Agent Behavioral Cloning, ICML 2018. [paper]
• Where Will They Go? Predicting Fine-Grained Adversarial Multi-Agent Motion using Conditional Variational Autoencoders, ECCV 2018. [paper]
• Coordinated Multi-Agent Imitation Learning, ICML 2017. [paper]
• Generating long-term trajectories using deep hierarchical networks, 2017. [paper]
• Learning Fine-Grained Spatial Models for Dynamic Sports Play Prediction, ICDM 2014. [paper]
• Generative Modeling of Multimodal Multi-Human Behavior, 2018. [paper]
• What will Happen Next? Forecasting Player Moves in Sports Videos, ICCV 2017, [paper]

Benchmark and Evaluation Metrics

• PIE: A Large-Scale Dataset and Models for Pedestrian Intention Estimation and Trajectory Prediction, ICCV 2019. [paper]
• Towards a fatality-aware benchmark of probabilistic reaction prediction in highly interactive driving scenarios, ITSC 2018. [paper]
• How good is my prediction? Finding a similarity measure for trajectory prediction evaluation, ITSC 2017. [paper]
• Trajnet: Towards a benchmark for human trajectory prediction. [website]

Others

• Cyclist trajectory prediction using bidirectional recurrent neural networks, AI 2018. [paper]
• Road infrastructure indicators for trajectory prediction, 2018. [paper]
• Using road topology to improve cyclist path prediction, 2017. [paper]
• Trajectory prediction of cyclists using a physical model and an artificial neural network, 2016. [paper]