2024-08-02.md

2024/08/02

Animals Singing Dance Monkey 🎤

animal_dance_monkey_480p.mp4

🎉 We are excited to announce the release of a new version featuring animals mode, along with several other updates. Special thanks to the dedicated efforts of the LivePortrait team. 💪 We also provided an one-click installer for Windows users, checkout the details here.

Updates on Animals mode

We are pleased to announce the release of the animals mode, which is fine-tuned on approximately 230K frames of various animals (mostly cats and dogs). The trained weights have been updated in the liveportrait_animals subdirectory, available on HuggingFace or Google Drive. You should download the weights before running. There are two ways to run this mode.

Please note that we have not trained the stitching and retargeting modules for the animals model due to several technical issues. This may be addressed in future updates. Therefore, we recommend disabling stitching by setting the --no_flag_stitching option when running the model. Additionally, paste-back is also not recommended.

Install X-Pose

We have chosen X-Pose as the keypoints detector for animals. This relies on transformers==4.22.0 and pillow>=10.2.0 (which are already updated in requirements.txt) and requires building an OP named MultiScaleDeformableAttention.

Refer to the PyTorch installation for Linux and Windows users.

Next, build the OP MultiScaleDeformableAttention by running:

cd src/utils/dependencies/XPose/models/UniPose/ops
python setup.py build install
cd - # this returns to the previous directory

To run the model, use the inference_animals.py script:

python inference_animals.py -s assets/examples/source/s39.jpg -d assets/examples/driving/wink.pkl --no_flag_stitching --driving_multiplier 1.75

Alternatively, you can use Gradio for a more user-friendly interface. Launch it with:

python app_animals.py # --server_port 8889 --server_name "0.0.0.0" --share

Warning

X-Pose is only for Non-commercial Scientific Research Purposes, you should remove and replace it with other detectors if you use it for commercial purposes.

Updates on Humans mode

• Driving Options: We have introduced an expression-friendly driving option to reduce head wobbling, now set as the default. While it may be less effective with large head poses, you can also select the pose-friendly option, which is the same as the previous version. This can be set using --driving_option or selected in the Gradio interface. Additionally, we added a --driving_multiplier option to adjust driving intensity, with a default value of 1, which can also be set in the Gradio interface.

• Retargeting Video in Gradio: We have implemented a video retargeting feature. You can specify a target lip-open ratio to adjust the mouth movement in the source video. For instance, setting it to 0 will close the mouth in the source video 🤐.

Others

• Poe supports LivePortrait. Check out the news on X.
• ComfyUI-LivePortraitKJ (1.1K 🌟) now includes MediaPipe as an alternative to InsightFace, ensuring the license remains under MIT and Apache 2.0.
• ComfyUI-AdvancedLivePortrait features real-time portrait pose/expression editing and animation, and is registered with ComfyUI-Manager.

Below are some screenshots of the new features and improvements:

The Gradio Interface of Animals Mode

Driving Options and Multiplier

The Feature of Retargeting Video