SVFR: A Unified Framework for Generalized Video Face Restoration

🔥 Overview

SVFR is a unified framework for face video restoration that supports tasks such as BFR, Colorization, Inpainting, and their combinations within one cohesive system.

🎬 Demo

BFR

Case1	Case2
case1_bfr.mp4	case4_bfr.mp4

BFR+Colorization

Case3	Case4
case10_bfr_colorization.mp4	case12_bfr_colorization.mp4

BFR+Colorization+Inpainting

Case5	Case6
case14_bfr+colorization+inpainting.mp4	case15_bfr+colorization+inpainting.mp4

🎙️ News

• [2025.01.02]: We released the initial version of the inference code and models. Stay tuned for continuous updates!
• [2024.12.17]: This repo is created!

🚀 Getting Started

Note: It is recommended to use a GPU with 16GB or more VRAM.

Setup

Use the following command to install a conda environment for SVFR from scratch:

conda create -n svfr python=3.9 -y
conda activate svfr

Install PyTorch: make sure to select the appropriate CUDA version based on your hardware, for example,

pip install torch==2.2.2 torchvision==0.17.2 torchaudio==2.2.2

Install Dependencies:

pip install -r requirements.txt

Download checkpoints

Download the Stable Video Diffusion

conda install git-lfs
git lfs install
git clone https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt models/stable-video-diffusion-img2vid-xt

Download SVFR

You can download checkpoints manually through link on Google Drive.

Put checkpoints as follows:

└── models
    ├── face_align
    │   ├── yoloface_v5m.pt
    ├── face_restoration
    │   ├── unet.pth
    │   ├── id_linear.pth
    │   ├── insightface_glint360k.pth
    └── stable-video-diffusion-img2vid-xt
        ├── vae
        ├── scheduler
        └── ...

Inference

Inference single or multi task

python3 infer.py \
 --config config/infer.yaml \
 --task_ids 0 \
 --input_path ./assert/lq/lq1.mp4 \
 --output_dir ./results/ \
 --crop_face_region

task_id:

0 -- bfr
1 -- colorization
2 -- inpainting
0,1 -- bfr and colorization
0,1,2 -- bfr and colorization and inpainting
...

crop_face_region:

Add the --crop_face_region flag at the end of the command to preprocess the input video by cropping the face region. This helps focus on the facial area and enhances processing results.

Inference with additional inpainting mask

# For Inference with Inpainting
# Add '--mask_path' if you need to specify the mask file.

python3 infer.py \
 --config config/infer.yaml \
 --task_ids 0,1,2 \
 --input_path ./assert/lq/lq3.mp4 \
 --output_dir ./results/ \
 --mask_path ./assert/mask/lq3.png \
 --crop_face_region

License

The code of SVFR is released under the MIT License. There is no limitation for both academic and commercial usage.

The pretrained models we provided with this library are available for non-commercial research purposes only, including both auto-downloading models and manual-downloading models.

Acknowledgments

This work is built on the architecture of Sonic.

BibTex

@misc{wang2025svfrunifiedframeworkgeneralized,
      title={SVFR: A Unified Framework for Generalized Video Face Restoration}, 
      author={Zhiyao Wang and Xu Chen and Chengming Xu and Junwei Zhu and Xiaobin Hu and Jiangning Zhang and Chengjie Wang and Yuqi Liu and Yiyi Zhou and Rongrong Ji},
      year={2025},
      eprint={2501.01235},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2501.01235}, 
}