Project Page | Paper | Video
This is an official code for the paper "Dynamic Point Fields" (ICCV 2023 Oral).
In a nutshell: we propose to model dynamic surfaces with a point-based model, where the motion of a point over time is represented by an implicit deformation field. Working directly with points (rather than SDFs) allows us to easily incorporate various well-known deformation constraints, e.g. as-isometric-as-possible (AIAP). We showcase the usefulness of this approach for creating animatable avatars in complex clothing:
dpf_avatars.mp4
Please see Section 4.3 of the paper for more details.
We release our code as a bundle of Colab notebooks covering the main experiments and applications discussed in the paper.
| Description | Link |
|---|---|
| Static Surface Reconstruction with Optimised Clouds |  |
| Learning Surface Deformations | |
| Single Scan Avatar Animation | |
| Loading and Evaluating Original Paper Results (Sections 4.1, Table 1) | TBD |
| Loading and Evaluating Original Paper Results (Sections 4.2, Table 2) | TBD |
Point-based surface representations 🔗 code
We begin this work by revisiting point clouds as surface modeling primitives, and demonstrate how surfaces of arbitrary complexity can be efficiently modeled as sets of 3D points with associated features, such as normal directions:
Compared to the state-of-the-art implicit models for 3D surface representations (NGLOD, NGP), optimised point cloud model offers better reconstruction quality on all metrics, while taking zero inference time thanks to its explicit nature:
Learning surface deformations 🔗 code
The explicit 3D surface formulation allows us to use some classic constraints for learning deformations in 3D space. In this work, we use the as-isometric-as-possible regularisation to guide our learning, which enforces the preservation of distances between points in the canonical and deformed spaces:
Please pay attention to the preservation of fingers in the deformed cloud when optimising with the isometric loss:
isometric.mp4
Guided deformation learning & single scan animation 🔗 code
In the case of dynamic humans, we can directly supervise our deformation network with the information on the dynamics of the vertices of the underlying minimally clothed human model :
We showcase the advantages of our approach for animating 3D humans in challenging clothing. Using the introduced isometric and guidance losses to drive the source 3D shape allows to perform a single scan animation. Using this technique, we can repose the canonical scan to highly challenging target poses:
single_scan_animation.mp4
If you find our work useful in your research, please consider citing:
@inproceedings{prokudin2023dynamic,
title = {Dynamic Point Fields},
author = {Prokudin, Sergey and Ma, Qianli and Raafat, Maxime and Valentin, Julien and Tang, Siyu},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
pages = {7964--7976},
month = oct,
year = {2023}
}
See the LICENSE file.