StyleLipSync — Unofficial PyTorch Implementation

Unofficial Implementation of StyleLipSync: Style-based Personalized Lip-sync Video Generation. Additional explanation.

TO-DO

The recent item is sorted upper side.

On-going StyleLipSync Implementation

I implement same architecture of StyleLipSync model follows original paper

1. Generator with SaMF
2. Encoders(face, ref, aud)

On-going Unseen Face Adaptation

I implement same architecture of StyleLipSync model follows original paper

On-going Test for HDTF dataset

I test the result using HDTF(Cross-ID) for 300 videos

2023/10/31 StyleGAN2 Setting

I use pre-trained StyleGAN2 model of rosinality/stylegan2-pytorch which is 550k trained using FFHQ dataset.

2023/10/30 Pose-aware Masking

I use same 3D face reconstruction model but implement different way.
My pipeline consists of the following steps: Estimate 3D parameters and vertices -> Extract Open/Close mesh by adjusting expression parameters -> Normalize to a neutral pose -> Remove points where (y >= 0) -> Revert to the original estimated pose and project to 2D.