[new] Stable Video Diffusion Image-to-Video

[cibernicola]

Stable Video Diffusion (SVD) Image-to-Video is a diffusion model that takes in a still image as a conditioning frame, and generates a video from it.

Model Details
Model Description
(SVD) Image-to-Video is a latent diffusion model trained to generate short video clips from an image conditioning. This model was trained to generate 25 frames at resolution 576x1024 given a context frame of the same size, finetuned from SVD Image-to-Video [14 frames]. We also finetune the widely used f8-decoder for temporal consistency. For convenience, we additionally provide the model with the standard frame-wise decoder here.

https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt

[ghost]

I read it needs ~40GB of VRAM for inference. There might be LCM versions of it soon, or other optimizations that enables it to be run on a 3090 or 4090

edit: Apparently it is possible already: https://twitter.com/timudk/status/1727064128223855087?t=lLeTOO8JYxuEcEiQm7WCWA&s=34

[cibernicola]

https://twitter.com/EMostaque/status/1727068577893843416

[ilikenwf]

https://education.civitai.com/quickstart-guide-to-stable-video-diffusion/

How can I use SVD?

ComfyUI is leading the pack when it comes to SVD image generation, with official SVD support! 25 frames of 1024×576 video uses < 10 GB VRAM to generate.

It’s entirely possible to run on a GTX 1080 with 8GB of VRAM!

There's already a decent model for it on civitai beyond the stock one:
https://civitai.com/models/211940/stable-video-diffusion-img2vid