1-Drone Flights
+
+ 
+ 
+ 
+ 
+
+
+
+
+
+
+ We recommend using Chrome for the best experience. Videos might not work on Safari.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ + 💎 GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Control +
+ + +
+
+
+
+
+
+
+
+
+
+ arXiv
+
+
+
+
+
+
+
+
+
+ Code
+
+
+
+
+
+
+Unconditional Generations
++ We show examples of unconditional generations from the model in diverse scenes with different driving dynamics. +
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Ego Control
+
+ We show examples of ego-motion controllability. All videos are generated by GEM using the same starting frame but with different trajectory control input.
+ We observe that the model follows the control signals and generates realistic scenes.
+
+
+
+
+
+
+
+
+
+
+ Object Manipulation
+
+ GEM can move objects in the scene using DINO features.
+ In the following examples, we show an unconditional generation by GEM and the same generation with motion control.
+ The green box indidcates the source DINO features and the blue ones indicate the target position tokens used.
+ We observe that the object moves from the green box to the blue box.
+
+
+
+
+ Unconditional Generation
+ +
+
+
+
+
+
+
+
+
+ Object Motion Control
+ +
+
+Object Insertion
+ + +
+
+
+
+
+
+ Unconditional Generation
+ +
+
+
+ Insertion Control
+ ++ In the following example, we insert a car on the left and control the motion of another car on the right. +
+ +
+
+
+
+
+
+Human Pose Manipulation
+
+ GEM can use human poses to control the motion of pedestrians in the scene.
+ In this examples, the pedesterians are crossing the street or stopping based on the human poses controls.
+
+
+
+
+
+
+
+
+
+
+
+Long Generation
+
+ We compare our long generation with the only world model trained on OpenDV capable of generating long sequences.
+ We observe that our generations have higher ego motion temporal consistency and more realistic dynamics.
+
+
+
+
+
+
+
+
+
+ GEM
+ +
+
+ Vista
+ +
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Interesting Observations
+ +
+ We show interesting behaviors observed in the generated videos.
These behaviors do not necessarily exist in the ground truth videos, but emerge from the model's learned dynamics.
+
+
+
+
+
+
+ Break Lights go off before moving
+ +
+
+ Smooth takeover dynamics on a long generation
+ +
+
+
+MultiModal
++ GEM generates two modalities simultaneously: RGB and Depth. We show examples of multimodal generations. +
+ +
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+MultiDomain
++ GEM is finetuned on two other ego centric domains and we observe it quickly adapts to these new domains. +
+ + +
+
+
+
+
+
+ 2-Human EgoCentric
+
+ 
+ 
+ 
+ 
+
+
+
+
+
+
+
+ Pseudo-Labelling
++ Some visualisations of the outputs of our pseudo-labeling pipeline. +
+
+
+
+
+