📒A small curated list of Awesome SD/DiT/Diffusion Inference with Distributed/Caching/Sampling. For Awesome LLM Inference, please check 📖Awesome-LLM-Inference
- 📙Awesome SD Inference with Sampling
- 📙Awesome SD Inference with Caching
- 📙Awesome SD Inference with Multi-GPUs
- 📙Other Awesome SD Inference Paper with codes
@misc{Awesome-SD-Inference@2024,
title={Awesome-SD-Inference: A small curated list of Awesome SD/DiT/ViT/Diffusion Inference with Distributed/Caching/Sampling.},
url={https://github.com/DefTruth/Awesome-SD-Inference},
note={Open-source software available at https://github.com/DefTruth/Awesome-SD-Inference},
author={DefTruth},
year={2024}
}
Date | Title | Paper | Code | Recom |
---|---|---|---|---|
2020.06 | 🔥[DDPM] Denoising Diffusion Probabilistic Models(@UC Berkeley) | [pdf] | [diffusion] |
⭐️⭐️ |
2020.10 | 🔥[DDIM] DENOISING DIFFUSION IMPLICIT MODELS(@cs.stanford.edu) | [pdf] | ⭐️⭐️ | |
2022.02 | 🔥[PNDM] PSEUDO NUMERICAL METHODS FOR DIFFUSION MODELS ON MANIFOLDS(@) | [pdf] | [PNDM] |
⭐️⭐️ |
2022.02 | 🔥[DPM-Solver] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps(@Cheng Lu) | [pdf] | [dpm-solver] |
⭐️⭐️ |
2022.11 | 🔥[DPM-Solver++] DPM-SOLVER++: FAST SOLVER FOR GUIDED SAMPLING OF DIFFUSION PROBABILISTIC MODELS(@Cheng Lu) | [pdf] | [dpm-solver] |
⭐️⭐️ |
2023.10 | 🔥[DPM-Solver-v3] DPM-Solver-v3: Improved Diffusion ODE Solver with Empirical Model Statistics(@Kaiwen Zheng) | [pdf] | [DPM-Solver-v3] |
⭐️⭐️ |
2023.11 | 🔥[Parallel Sampling] Parallel Sampling of Diffusion Models(@Stanford University) | [pdf] | [paradigms] |
⭐️⭐️ |
2023.11 | 🔥[SAMPLER SCHEDULER] SAMPLER SCHEDULER FOR DIFFUSION MODELS(@sysu) | [pdf] | ⭐️⭐️ | |
2024.02 | 🔥[Parallel Sampling] Accelerating Parallel Sampling of Diffusion Models(@Zhiwei Tang) | [pdf] | [ParaTAA-Diffusion] |
⭐️⭐️ |
2024.01 | 🔥[YONOS] You Only Need One Step: Fast Super-Resolution with Stable Diffusion via Scale Distillation(@Samsung AI) | [pdf] | ⭐️⭐️ | |
2024.01 | 🔥[S^2-DM] S^2-DMs: Skip-Step Diffusion Models(@Yixuan Wang) | [pdf] | ⭐️⭐️ | |
2024.08 | 🔥[StepSaver] StepSaver: Predicting Minimum Denoising Steps for Diffusion Model Image Generation(@intel) | [pdf] | ⭐️⭐️ | |
2024.09 | 🔥[DC-Solver] DC-Solver: Improving Predictor-Corrector Diffusion Sampler via Dynamic Compensation(@Tsinghua University) | [pdf] | [DC-Solver] |
⭐️⭐️ |
- UNet Based (DeepCache)
![image](https://private-user-images.githubusercontent.com/31974251/352797438-a7257462-80d3-40af-a4ce-3550508fabe7.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjYxNDkzNjAsIm5iZiI6MTcyNjE0OTA2MCwicGF0aCI6Ii8zMTk3NDI1MS8zNTI3OTc0MzgtYTcyNTc0NjItODBkMy00MGFmLWE0Y2UtMzU1MDUwOGZhYmU3LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MTIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTEyVDEzNTEwMFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTc2NmNkNGMxODA5NDlkM2ZiNDNhODAwNjMwMDI1OTE1NDIxMGZmMjg2ZDMxNTU5ZTg3ZjAwMmY0YWQ3MDVjYTcmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.Nm_rQr5D0IBhIP6F94_kL-ExKEcLspvPfSQ0F7DuR6Q)
- DiT Based (Fast-Forward Caching)
![image](https://private-user-images.githubusercontent.com/31974251/352797183-fad8f187-d4ac-4290-9943-7b34116fed05.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjYxNDkzNjAsIm5iZiI6MTcyNjE0OTA2MCwicGF0aCI6Ii8zMTk3NDI1MS8zNTI3OTcxODMtZmFkOGYxODctZDRhYy00MjkwLTk5NDMtN2IzNDExNmZlZDA1LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MTIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTEyVDEzNTEwMFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWQyMjBkOWE0ZGZkMDMwZmMxNTU1NWQ4YWNjNTQzMzlhNTliMmIzODM2YWNmMDIwYmUwMGFiYWJlZmNhMGY2MjEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.o2Tk2xgZVQ75qoaV1xnzbSKiWaZtGJApbPOoojQn7dk)
Date | Title | Paper | Code | Recom |
---|---|---|---|---|
2023.05 | 🔥🔥[Cache-Enabled Sparse Diffusion] Accelerating Text-to-Image Editing via Cache-Enabled Sparse Diffusion Inference(@pku.edu.cn etc) | [pdf] | ⭐️⭐️ | |
2023.12 | 🔥🔥[DeepCache] DeepCache: Accelerating Diffusion Models for Free(@nus.edu) | [pdf] | [DeepCache] |
⭐️⭐️ |
2023.12 | 🔥🔥[Block Caching] Cache Me if You Can: Accelerating Diffusion Models through Block Caching(@Meta GenAI etc) | [pdf] | ⭐️⭐️ | |
2023.12 | 🔥🔥[Approximate Caching] Approximate Caching for Efficiently Serving Diffusion Models(@Adobe) | [pdf] | ⭐️⭐️ | |
2024.06 | 🔥🔥[Layer Caching] Learning-to-Cache: Accelerating Diffusion Transformer via Layer Caching(@nus.edu) | [pdf] | [learning-to-cache] |
⭐️⭐️ |
2024.07 | 🔥[ElasticCache-LVLM] Efficient Inference of Vision Instruction-Following Models with Elastic Cache(@Tsinghua University etc) | [pdf] | [ElasticCache] |
⭐️ |
2024.07 | 🔥🔥[Fast-Forward Caching(DiT)] FORA: Fast-Forward Caching in Diffusion Transformer Acceleration(@microsoft.com etc) | [pdf] | [FORA] |
⭐️⭐️ |
2024.07 | 🔥🔥[Faster I2V Generation] Faster Image2Video Generation: A Closer Look at CLIP Image Embedding’s Impact on Spatio-Temporal Cross-Attentions(@Ashkan Taghipour etc) | [pdf] | ⭐️⭐️ | |
2024.04 | 🔥🔥[T-GATE V1] Cross-Attention Makes Inference Cumbersome in Text-to-Image Diffusion Models(@Wentian Zhang etc) | [pdf] | [T-GATE] |
⭐️⭐️ |
2024.04 | 🔥🔥[T-GATE V2] Faster Diffusion via Temporal Attention Decomposition(@Haozhe Liu etc) | [pdf] | [T-GATE] |
⭐️⭐️ |
- UNet Based: Displaced Patch parallelism (DistriFusion)
![image](https://private-user-images.githubusercontent.com/31974251/352798949-aefb2ae7-73eb-4e9c-bf1a-ec540f4dfa7d.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjYxNDkzNjAsIm5iZiI6MTcyNjE0OTA2MCwicGF0aCI6Ii8zMTk3NDI1MS8zNTI3OTg5NDktYWVmYjJhZTctNzNlYi00ZTljLWJmMWEtZWM1NDBmNGRmYTdkLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MTIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTEyVDEzNTEwMFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWI3Nzg5NTEwOTUxOTgzZDA5NTE3ODU1ODE0MjFjZTAzYjcyZGNiYWU2NjVkY2JkNjI1M2VlNzg0NjAzMWJmZTgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.CzSe7CYn27Jm01FwiTCxcWwWzm_Jc2XD8WIpNSklfOo)
- DiT Based: Displaced Patch parallelism (PipeFusion)
![image](https://private-user-images.githubusercontent.com/31974251/352799269-692c5d54-19b3-4ce7-9613-9eb8bb035c7d.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjYxNDkzNjAsIm5iZiI6MTcyNjE0OTA2MCwicGF0aCI6Ii8zMTk3NDI1MS8zNTI3OTkyNjktNjkyYzVkNTQtMTliMy00Y2U3LTk2MTMtOWViOGJiMDM1YzdkLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MTIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTEyVDEzNTEwMFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTMxNDM0Y2U4OGJjZmEyNjE5YmVmZDg5N2JhODE3M2QzNGM4NTY4Y2VjODFkMTExZTE3NDM0Mzg1MzE1ZmQ0ZTgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.n9owmFriyn5XvGMBMapmI8uWHIRq9ING8Vu3iKLexLg)
Date | Title | Paper | Code | Recom |
---|---|---|---|---|
2024.02 | 🔥🔥[DistriFusion] DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models(@MIT etc) | [pdf] | [distrifuser] |
⭐️⭐️ |
2024.05 | 🔥🔥[PipeFusion] PipeFusion: Displaced Patch Pipeline Parallelism for Inference of Diffusion Transformer Models(@Tencent etc) | [pdf] | [xDiT] |
⭐️⭐️ |
2024.06 | 🔥🔥[AsyncDiff] AsyncDiff: Parallelizing Diffusion Models by Asynchronous Denoising(@nus.edu) | [pdf] | [AsyncDiff] |
⭐️⭐️ |
2024.05 | 🔥🔥[TensorRT-LLM SDXL] SDXL Distributed Inference with TensorRT-LLM and synchronous comm(@Zars19) | [pdf] | [SDXL-TensorRT-LLM] |
⭐️⭐️ |
2024.06 | 🔥🔥[Clip Parallelism] Video-Infinity: Distributed Long Video Generation(@nus.edu) | [pdf] | [Video-Infinity] |
⭐️⭐️ |
2024.05 | 🔥🔥[FIFO-Diffusion] FIFO-Diffusion: Generating Infinite Videos from Text without Training(@Seoul National University) | [pdf] | [FIFO-Diffusion] |
⭐️⭐️ |
Date | Title | Paper | Code | Recom |
---|---|---|---|---|
2024.06 | 🔥[DiTFastAttn] DiTFastAttn: Attention Compression for Diffusion Transformer Models(@Zhihang Yuan etc) | [pdf] | [DiTFastAttn] |
⭐️⭐️ |
2024.08 | 🔥[Transfusion] Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model(@meta) | [pdf] | [transfusion-pytorch] |
⭐️⭐️ |
2024.08 | 🔥[VQ4DiT] VQ4DiT: Efficient Post-Training Vector Quantization for Diffusion Transformers(@ZJU) | [pdf] | ⭐️⭐️ | |
2024.08 | 🔥[LBQ] Low-Bitwidth Floating Point Quantization for Efficient High-Quality Diffusion Models(@toronto.edu) | [pdf] | ⭐️⭐️ | |
2024.08 | 🔥[EE-Diffusion] A Simple Early Exiting Framework for Accelerated Sampling in Diffusion Models(@KAIST AI) | [pdf] | [ee-diffusion] |
⭐️⭐️ |
2024.08 | 🔥[TFM-PTQ] Temporal Feature Matters: A Framework for Diffusion Model Quantization(@SenseTime) | [pdf] | ⭐️⭐️ | |
2024.08 | 🔥[Diffusion-RWKV] Diffusion-RWKV: Scaling RWKV-Like Architectures for Diffusion Models(@Zhengcong Fei) | [pdf] | [Diffusion-RWKV] |
⭐️⭐️ |
2024.09 | 🔥[LinFusion] LINFUSION: 1 GPU, 1 MINUTE, 16K IMAGE(@NUS) | [pdf] | [LinFusion] |
⭐️⭐️ |
GNU General Public License v3.0
Welcome to star & submit a PR to this repo!