Artifacts in Long Video Reconstruction with VAE (e.g., 225 Frames, Sample Rate = 2)

[WilliamLLee]

When using VAE to reconstruct long videos (e.g., 225 frames) with a sample_rate of 2 (or more), artifacts tend to appear towards the end of the video. Has anyone encountered this issue, and if so, what might be causing it? How can these artifacts be mitigated?

[WilliamLLee]

This issue seems to arise from the tiling problem. For the same video clip, when tiling is enabled, artifacts appear; when tiling is disabled, the reconstruction works as expected. I am using the WFVAE inference code version included in Open-Sora-Plan. You can find it here: WFVAE Inference Code.

Could you please let me know if there is any difference between the WFVAE inference code in this repository and the one in Open-Sora-Plan? If there is no difference, I suspect there might be a bug in the tiling implementation within the WFVAE inference code. I am currently debugging it, and I wanted to check if you have encountered and solved this issue before?

• without tiling:
  https://github.com/user-attachments/assets/81224993-8192-423f-bbb5-41a5a952ffa5

• with tiling
  https://github.com/user-attachments/assets/3166edac-fcc5-4457-beff-1af0e399bbda

[qqingzheng]

Hi, it is caused by the setting of tiling size. You should choose a suitable tiling size for a specific frame number.

Open-Sora-Plan: https://github.com/PKU-YuanGroup/Open-Sora-Plan/blob/main/opensora/models/causalvideovae/model/vae/modeling_wfvae.py#L348

WF-VAE: https://github.com/PKU-YuanGroup/WF-VAE/blob/main/causalvideovae/model/vae/modeling_wfvae.py#L331

Fo high sample rate scenarios, I suggest using lpips weight = 0.1 and enabling dynamic sample (which adopted by CogVideoX) fine-tuning WF-VAE to achieve better results.