Skip to content

Commit

Permalink
fix(diffusers/pipelines): fix some bugs to make mindone.diffusers.pip…
Browse files Browse the repository at this point in the history 
…elines compatible with MindSpore 2.3.0 (mindspore-lab#640)
  • Loading branch information
@townwish4git
townwish4git committed Aug 27, 2024
1 parent b3d2a7c commit 3b8094c
Show file tree
Hide file tree
Showing 4 changed files with 74 additions and 6 deletions.
71 changes: 71 additions & 0 deletions docs/diffusers/limitations.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -53,3 +53,74 @@ The table below represents the current support in mindone/diffusers for each of
| AttnSkipDownBlock2D | ❌ | ✅ | ✅ | ✅ | contains FirDownsample2D |
| SkipDownBlock2D | ❌ | ✅ | ✅ | ✅ | contains FirDownsample2D |
| ResnetBlock2D (kernel='fir') | ❌ | ✅ | ✅ | ✅ | ops.Conv2D has poor precision in fp16 and PyNative mode |

## Pipelines
The table below represents the current support in mindone/diffusers for each of those pipelines in **MindSpore 2.3.0**,
whether they have support in Pynative fp16 mode, Graph fp16 mode, Pynative fp32 mode or Graph fp32 mode.

> Hint: Due to the precision issue with GroupNorm affecting almost all pipelines under FP16, leading to inference
> precision issues of pipelines, the experiments in the table below default to upcasting GroupNorm to FP32 to avoid
> this issue.
| **Pipelines** | **Pynative FP16** | **Pynative FP32** | **Graph FP16** | **Graph FP32** | **Description** |
|:------------------------------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|
| AnimateDiffPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| AnimateDiffVideoToVideoPipeline | :white_check_mark: | :x: | :white_check_mark: | :white_check_mark: | In FP32 and Pynative mode, this pipeline will run out of memory |
| BlipDiffusionPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| ConsistencyModelPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| DDIMPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| DDPMPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| DiTPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| I2VGenXLPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | ops.bmm and ops.softmax have precision issues under FP16, so we need to upcast them to FP32 to get a good result |
| IFImg2ImgPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| IFImg2ImgSuperResolutionPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| IFInpaintingPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| IFInpaintingSuperResolutionPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| IFPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| IFSuperResolutionPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| Kandinsky3Img2ImgPipeline | :x: | :x: | :x: | :x: | Kandinsky3 only provides FP16 weights; additionally, T5 has precision issues, so to achieve the desired results, you need to directly input prompt_embeds and attention_mask. |
| Kandinsky3Pipeline | :x: | :x: | :x: | :x: | Kandinsky3 only provides FP16 weights; additionally, T5 has precision issues, so to achieve the desired results, you need to directly input prompt_embeds and attention_mask. |
| KandinskyImg2ImgPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| KandinskyInpaintPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| KandinskyPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| KandinskyV22ControlnetImg2ImgPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| KandinskyV22ControlnetPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| KandinskyV22Img2ImgPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| KandinskyV22InpaintPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| KandinskyV22Pipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| LatentConsistencyModelImg2ImgPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| LatentConsistencyModelPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| LDMSuperResolutionPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| LDMTextToImagePipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| PixArtAlphaPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| ShapEImg2ImgPipeline | :white_check_mark: | :white_check_mark: | :x: | :x: | The syntax in Render only supports Pynative mode |
| ShapEPipeline | :white_check_mark: | :white_check_mark: | :x: | :x: | The syntax in Render only supports Pynative mode |
| StableCascadePipeline | :x: | :white_check_mark: | :x: | :white_check_mark: | This pipeline does not support FP16 due to precision issues |
| StableDiffusion3Pipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| StableDiffusionAdapterPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| StableDiffusionControlNetImg2ImgPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| StableDiffusionControlNetInpaintPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| StableDiffusionControlNetPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| StableDiffusionDepth2ImgPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| StableDiffusionDiffEditPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| StableDiffusionGLIGENPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| StableDiffusionGLIGENTextImagePipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| StableDiffusionImageVariationPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| StableDiffusionImg2ImgPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| StableDiffusionInpaintPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| StableDiffusionInstructPix2PixPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| StableDiffusionLatentUpscalePipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| StableDiffusionPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| StableDiffusionUpscalePipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| StableDiffusionXLAdapterPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| StableDiffusionXLControlNetImg2ImgPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| StableDiffusionXLControlNetInpaintPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| StableDiffusionXLControlNetPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| StableDiffusionXLImg2ImgPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| StableDiffusionXLInpaintPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| StableDiffusionXLInstructPix2PixPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| StableDiffusionXLPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| StableVideoDiffusionPipeline | :white_check_mark: | :x: | :white_check_mark: | :x: | This pipeline will run out of memory under FP32; ops.bmm and ops.softmax have precision issues under FP16, so we need to upcast them to FP32 to get a good result |
| UnCLIPImageVariationPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| UnCLIPPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| WuerstchenPipeline | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | GlobalResponseNorm has precision issue under FP16, so we need to upcast it to FP32 to get a good result |
Loading

0 comments on commit 3b8094c

Please sign in to comment.