Projecting Input on as Texture

[camjac251]

I really like where this project is going. The video looked really cool and made PRNet artist friendly in Nuke.

Would it be possible to see a video demonstration of the results of facefit with projection mapping? I'm attempting to use PRNet in python but notice the model is creating some odd geo and texture artifacts (which I haven't been able to fix with my limited knowledge in skimage)
. This tool could replace FaceBuilder and FaceTracker by KeenTools, which allows you to build a face and track it but it requires manual keying here and there.

[mishurov]

I will try as I get a time. I'm not sure about the resulting quality to be frank.

As you've seen the original PRNet's code, it initially uses convolutional network that guesses 3d locations of the point grid 256*256, then it filters out points which don't belong to the facial geometry. They have also a predefined set of triangle indices and based on that they build an output mesh.

They also mention, that the resulting geometry isn't very detailed.

I've made an additional branch, https://github.com/mishurov/facefit/tree/tflite it uses Google's neural network for TensorFlow Lite, https://ai.googleblog.com/2019/03/real-time-ar-self-expression-with.html I've extracted the model file from ARCore's APK file.

Unfortunately, as I'm aware, TFLite isn't supposed to work on desktop machines at all yet I've managed to compile it for Linux even with sse4 support and blas because they support x86 mobile emulators.

The geometry is fine, but the tracking is still unstable and jumpy in both branches. Google mentions in that article about "a windowed smoothing" but I couldn't figure out what they use, I tried moving average, Kalman filter and weighting proportionally to max delta distance, based on current and prev frame, yet I couldn't achieve that stability as on their gifs, my results are either with lags or wobbling.

P.S. If you were confused with skimage, I can explain what they do. Facial detector detects a rectangle containing a face, then they obtain with skimage a 2d similarity matrix which maps points on the original big frame to 256*256 square image containing the face to feed PRNet with that square image. After PRNet returns 3d points, they use inverse matrix to map x, y coordinates of 3d points back to the whole image.