Skip to content

Latest commit

 

History

History
394 lines (186 loc) · 27.5 KB

README_ord.md

File metadata and controls

394 lines (186 loc) · 27.5 KB
Raw

Awesome Image Editing

This is the github repository of our work "A Survey of Multimodal-Guided Image Editing with Text-to-Image Diffusion Models".

Editing Tasks Discussed in Our Survey

image

Unified Framework

image

Table of contents

Content-Aware Editing


Content-Free Editing


Experiment and Data


Object Manipulation and Attribute Manipulation:

1. Training-Free Approaches

📄 UniTune: Text-Driven Image Editing by Fine Tuning a Diffusion Model on a Single Image | 📖 TOG 2023 | 🔀 $F_{inv}^T+F_{edit}^{Norm}$ | 🌐 Code

📄 Highly Personalized Text Embedding for Image Manipulation by Stable Diffusion | 📖 Arxiv 2023 | 🔀 $F_{inv}^T+F_{edit}^{Norm}$| 🌐 Code

📄 Imagic: Text-Based Real Image Editing with Diffusion Models | 📖 CVPR 2023 | 🔀 $F_{inv}^T+F_{edit}^{Blend}$ | [🌐 Code]

📄 Forgedit: Text Guided Image Editing via Learning and Forgetting | 📖 Arxiv 2023 | 🔀 $F_{inv}^T+F_{edit}^{Blend}$ | 🌐 Code

📄 Doubly Abductive Counterfactual Inference for Text-based Image Editing | 📖 CVPR 2024 | 🔀 $F_{inv}^T+F_{edit}^{Blend}$ | 🌐 Code

📄 SINE: Sinle Image Editing with Text-to-Image Diffusion Models | 📖 CVPR 2023 | 🔀 $F_{inv}^T+F_{edit}^{Score}$ | 🌐 Code

📄 EDICT: Exact Diffusion Inversion via Coupled Transformations | 📖 CVPR 2023 | 🔀 $F_{inv}^F+F_{edit}^{Norm}$ | 🌐 Code

📄 Exact Diffusion Inversion via Bi-directional Integration Approximation | 📖 Arxiv 2023 | 🔀 $F_{inv}^F+F_{edit}^{Norm}$ | 🌐 Code

📄 Effective Real Image Editing with Accelerated Iterative Diffusion Inversion | 📖 ICCV 2023 | 🔀 $F_{inv}^F+F_{edit}^{Norm}$ | 🌐 Code

📄 Null-text Inversion for Editing Real Images using Guided Diffusion Models | 📖 CVPR 2023 | 🔀 $F_{inv}^F+F_{edit}^{Attn}$ | 🌐 Code

📄 Negative-prompt Inversion: Fast Image Inversion for Editing with Text-guided Diffusion Models | 📖 Arxiv 2023 | 🔀 $F_{inv}^F+F_{edit}^{Attn}$ | 🌐 Code

📄ProxEdit: Improving Tuning-Free Real Image Editing with Proximal Guidance | 📖 WACV 2024 | 🔀 $F_{inv}^F+F_{edit}^{Attn}$ | 🌐 Code

📄 Fixed-point Inversion for Text-to-image diffusion models | 📖 Arxiv 2023 | 🔀 $F_{inv}^F+F_{edit}^{Attn}$ | 🌐 Code

📄PnP Inversion: Boosting Diffusion-based Editing with 3 Lines of Code | 📖 ICLR 2024 | 🔀 $F_{inv}^F+F_{edit}^{Attn}$| 🌐 Code

📄 Dynamic Prompt Learning: Addressing Cross-Attention Leakage for Text-Based Image Editing | 📖 NeurIPS 2023 | 🔀 $F_{inv}^F+F_{edit}^{Attn}$ | 🌐 Code

📄 An Edit Friendly DDPM Noise Space: Inversion and Manipulations | 📖 CVPR 2024 | 🔀 $F_{inv}^F+F_{edit}^{Attn}$ | 🌐 Code

📄 Prompt-to-Prompt Image Editing with Cross-Attention Control | 📖 ICLR 2023 | 🔀 $F_{inv}^F+F_{edit}^{Attn}$ | 🌐 Code

📄 Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation | 📖 CVPR 2023 | 🔀 $F_{inv}^F+F_{edit}^{Attn}$ | 🌐 Code

📄 Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing | 📖 CVPR 2024 | 🔀 $F_{inv}^F+F_{edit}^{Attn}$ | 🌐 Code

📄 StyleDiffusion: Prompt-Embedding Inversion for Text-Based Editing | 📖 Arxiv 2023 | 🔀 $F_{inv}^F+F_{edit}^{Attn}$ | 🌐 Code

📄 Prompt Tuning Inversion for Text-Driven Image Editing Using Diffusion Models | 📖 ICCV 2023 | 🔀 $F_{inv}^F+F_{edit}^{Blend}$ | 🌐 Code

📄 Object-aware Inversion and Reassembly for Image Editing | 📖 ICLR 2024 | 🔀 $F_{inv}^F+F_{edit}^{Blend}$ | 🌐 Code

📄 DiffEdit: Diffusion-based semantic image editing with mask guidance | 📖 ICLR 2023 | 🔀 $F_{inv}^F+F_{edit}^{Blend}$ | 🌐 Code

📄 PFB-Diff: Progressive Feature Blending Diffusion for Text-driven Image Editing | 📖 Arxiv 2023 | 🔀 $F_{inv}^F+F_{edit}^{Blend}$ | 🌐 Code

📄 Uncovering the Disentanglement Capability in Text-to-Image Diffusion Models | 📖 CVPR 2023 | 🔀 $F_{inv}^F+F_{edit}^{Blend}$| 🌐 Code

📄 Noise Map Guidance: Inversion with Spatial Context for Real Image Editing | 📖 ICLR 2024 | 🔀 $F_{inv}^F+F_{edit}^{Score}$ | 🌐 Code

📄 pix2pix-zero | 📖 SIGGRAPH 2023 | 🔀 $F_{inv}^F+F_{edit}^{Score}$ | 🌐 Code

📄 SEGA: Instructing Diffusion using Semantic Dimensions | 📖 NeurIPS 2023 | 🔀 $F_{inv}^F+F_{edit}^{Score}$ | 🌐 Code

📄 The Stable Artist: Steering Semantics in Diffusion Latent Space | 📖 Arxiv 2022 | 🔀 $F_{inv}^F+F_{edit}^{Score}$ | 🌐 Code

📄 LEDITS: Real Image Editing with DDPM Inversion and Semantic Guidance | 📖 Arxiv 2023 | 🔀 $F_{inv}^F+F_{edit}^{Score}$ | 🌐 Code

📄 LEDITS++: Limitless Image Editing using Text-to-Image Models | 📖 CVPR 2024 | 🔀 $F_{inv}^F+F_{edit}^{Score}$ | 🌐 Code

📄 Magicremover: Tuning-free Text-guided Image inpainting with Diffusion Models | 📖 ICLR 2024 | 🔀 $F_{inv}^F+F_{edit}^{Score}$ | 🌐 Code

📄 Region-Aware Diffusion for Zero-shot Text-driven Image Editing | 📖 Arxiv 2023 | 🔀 $F_{inv}^F+F_{edit}^{Optim}$ | 🌐 Code

📄 Delta Denoising Score | 📖 ICCV 2023 | 🔀 $F_{inv}^F+F_{edit}^{Optim}$ | 🌐 Code

📄 Contrastive Denoising Score for Text-guided Latent Diffusion Image Editing | 📖 CVPR 2024 | 🔀 $F_{inv}^F+F_{edit}^{Optim}$ | 🌐 Code

📄 Ground-A-Score: Scaling Up the Score Distillation for Multi-Attribute Editing | 📖 Arxiv 2024 | 🔀 $F_{inv}^F+F_{edit}^{Optim}$ | 🌐 Code

📄 Custom-Edit: Text-Guided Image Editing with Customized Diffusion Models | 📖 CVPR 2023 | 🔀 $F_{inv}^T+F_{inv}^F+F_{edit}^{Attn}$ | 🌐 Code

📄 Photoswap: Personalized Subject Swapping in Images | 📖 NeurIPS 2023 | 🔀 $F_{inv}^T+F_{inv}^F+F_{edit}^{Attn}$ | 🌐 Code

📄 DreamEdit: Subject-driven Image Editing | 📖 TMLR 2023 |🔀 $F_{inv}^T+F_{inv}^F+F_{edit}^{Blend}$ | 🌐 Code

2. Training-Based Approaches

📄 InstructPix2Pix: Learning to Follow Image Editing Instructions | 📖 CVPR 2023 | 🌐 Code

📄 MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing | 📖 NeurIPS 2023 | 🌐 Code

📄 HIVE: Harnessing Human Feedback for Instructional Visual Editing | 📖 Arxiv 2023 | 🌐 Code

📄 Emu Edit: Precise Image Editing via Recognition and Generation Tasks | 📖 Arxiv 2023 | 🌐 Code

📄 GUIDING INSTRUCTION-BASED IMAGE EDITING VIA MULTIMODAL LARGE LANGUAGE MODELS | 📖 ICLR 2024 | 🌐 Code

📄 SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Models | 📖CVPR 2024 | 🌐 Code

📄 Referring Image Editing: Object-level Image Editing via Referring Expressions | 📖CVPR 2024 | 🌐 Code


Attribute Manipulation:

1. Training-Free Approaches

📄 KV Inversion: KV Embeddings Learning for Text-Conditioned Real Image Action Editing | 📖 PRCV 2023 | 🔀 $F_{inv}^F+F_{edit}^{Attn}$ | 🌐 Code

📄 Localizing Object-level Shape Variations with Text-to-Image Diffusion Models | 📖 ICCV 2023 | 🔀 $F_{inv}^F+F_{edit}^{Attn}$ | 🌐 Code

📄 MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing | 📖 ICCV 2023 | 🔀 $F_{inv}^F+F_{edit}^{Attn}$ | 🌐 Code

📄 Tuning-Free Inversion-Enhanced Control for Consistent Image Editing | 📖 AAAI 2023 | 🔀 $F_{inv}^F+F_{edit}^{Attn}$ | 🌐 Code

📄 Cross-Image Attention for Zero-Shot Appearance Transfer | 📖 SIGGRAPH 2024 | 🔀 $F_{inv}^F+F_{edit}^{Attn}$ | 🌐 Code

2. Training-Based Approaches


Spatial Transformation:

1. Training-Free Approaches

📄 DesignEdit: Multi-Layered Latent Decomposition and Fusion for Unified & Accurate Image Editing | 📖 Arxiv 2024 | 🔀 $F_{inv}^F+F_{edit}^{Attn}$ | 🌐 Code

📄 Diffusion Self-Guidance for Controllable Image Generation | 📖 NeurIPS 2023 | 🔀 $F_{inv}^F+F_{edit}^{Score}$ | 🌐 Code

📄 DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Models | 📖 ICLR 2024 | 🔀 $F_{inv}^F+F_{edit}^{Score}$ | 🌐 Code

📄 DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editing | 📖 ICLR 2024 | 🔀 $F_{inv}^T+F_{inv}^F+F_{edit}^{Optim}$ | 🌐 Code

📄 DiffEditor: Boosting Accuracy and Flexibility on Diffusion-based Image Editing | 📖 ICLR 2024 | 🔀 $F_{inv}^T+F_{inv}^F+F_{edit}^{Score}$ | 🌐 Code

2. Training-Based Approaches


Inpainting:

1. Training-Free Approaches

📄 HD-Painter: High-Resolution and Prompt-Faithful Text-Guided Image Inpainting with Diffusion Models | 📖 Arxiv 2023 | 🔀 $F_{inv}^F+F_{edit}^{Attn}$ | 🌐 Code

📄 TF-ICON: Diffusion-Based Training-Free Cross-Domain Image Composition | 📖 ICCV 2023 | 🔀 $F_{inv}^F+F_{edit}^{Blend}$ | 🌐 Code

📄 Blended Latent Diffusion | 📖 TOG 2023 | 🔀 $F_{inv}^F+F_{edit}^{Blend}$ | 🌐 Code

📄 High-Resolution Image Editing via Multi-Stage Blended Diffusion | 📖 Arxiv 2022 | 🔀 $F_{inv}^F+F_{edit}^{Blend}$ | 🌐 Code

📄 Differential Diffusion: Giving Each Pixel Its Strength | 📖 Arxiv 2023 | 🔀 $F_{inv}^F+F_{edit}^{Blend}$ | 🌐 Code

📄 Tuning-Free Image Customization with Image and Text Guidance | 📖 CVPR 2024 | 🔀 $F_{inv}^F+F_{edit}^{Blend}$ | 🌐 Code

2. Training-Based Approaches

📄 Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpainting | 📖 CVPR 2024| 🌐 Code

📄 SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Model | 📖 CVPR 2023 | 🌐 Code

📄 A Task is Worth One Word: Learning with Task Prompts for High-Quality Versatile Image Inpainting | 📖 Arxiv 2023 | 🌐 Code

📄 Paint by Example: Exemplar-based Image Editing with Diffusion Models | 📖 CVPR 2023 | 🌐 Code

📄 ObjectStitch: Object Compositing with Diffusion Model | 📖 CVPR 2023 | 🌐 Code

📄 Reference-based Image Composition with Sketch via Structure-aware Diffusion Model | 📖 CVPR 2023 | 🌐 Code

📄 Paste, Inpaint and Harmonize via Denoising: Subject-Driven Image Editing with Pre-Trained Diffusion Model | 📖 ICASSP 2024 | 🌐 Code

📄 AnyDoor: Zero-shot Object-level Image Customization | 📖 CVPR 2024 | 🌐 Code


Style Change:

1. Training-Free Approaches

📄 Inversion-Based Style Transfer with Diffusion Models | 📖 CVPR 2023 | 🔀 $F_{inv}^T+F_{inv}^F+F_{edit}^{Attn}$ | 🌐 Code

📄 Style Injection in Diffusion: A Training-free Approach for Adapting Large-scale Diffusion Models for Style Transfer | 📖 CVPR 2024 | 🔀 $F_{inv}^F+F_{edit}^{Attn}$ | 🌐 Code

📄 Z∗: Zero-shot Style Transfer via Attention Rearrangement | 📖 Arxiv 2023 | 🔀 $F_{inv}^F+F_{edit}^{Attn}$ | 🌐 Code

2. Training-Based Approaches


Image Translation:

1. Training-Free Approaches

📄 FreeControl: Training-Free Spatial Control of Any Text-to-Image Diffusion Model with Any Condition | 📖 CVPR 2024 | 🌐 Code

2. Training-Based Approaches

📄 Adding Conditional Control to Text-to-Image Diffusion Models | 📖 ICCV 2023 | 🌐 Code

📄 T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models | 📖 AAAI 2024 | 🌐 Code

📄 SCEdit: Efficient and Controllable Image Diffusion Generation via Skip Connection Editing | 📖 CVPR 2024 | 🌐 Code

📄 Cocktail: Mixing Multi-Modality Controls for Text-Conditional Image Generation | 📖 NeurIPS 2023 | 🌐 Code

📄 Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Model | 📖 NeurIPS 2023 | 🌐 Code

📄 CycleNet: Rethinking Cycle Consistency in Text-Guided Diffusion for Image Manipulation | 📖 NeurIPS 2023 | 🌐 Code

📄 One-Step Image Translation with Text-to-Image Models | 📖 Arxiv 2024] | 🌐 Code


Subject-Driven Customization:

1. Training-Free Approaches

📄 An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion | 📖 ICLR 2023 | 🔀 $F_{inv}^T+F_{edit}^{Norm}$ | 🌐 Code

📄 DreamArtist: Towards Controllable One-Shot Text-to-Image Generation via Positive-Negative Prompt-Tuning | 📖 Arxiv 2022 | 🔀 $F_{inv}^T+F_{edit}^{Norm}$ | 🌐 Code

📄 P+: Extended Textual Conditioning in Text-to-Image Generation | 📖 Arxiv 2023 | 🔀 $F_{inv}^T+F_{edit}^{Norm}$ | 🌐 Code

📄 A Neural Space-Time Representation for Text-to-Image Personalization | 📖 TOG 2023 | 🔀 $F_{inv}^T+F_{edit}^{Norm}$ | 🌐 Code

📄 DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation | 📖 CVPR 2023 | 🔀 $F_{inv}^T+F_{edit}^{Norm}$ | 🌐 Code

📄A Data Perspective on Enhanced Identity Preservation for Diffusion Personalization | 📖 ICLR 2024 | 🔀 $F_{inv}^T+F_{edit}^{Norm}$ | 🌐 Code

📄 FaceChain-SuDe: Building Derived Class to Inherit Category Attributes for One-shot Subject-Driven Generation | 📖 CVPR 2024 | 🔀 $F_{inv}^T+F_{edit}^{Norm}$ | 🌐 Code

📄 Multi-Concept Customization of Text-to-Image Diffusion | 📖 CVPR 2023 | 🔀 $F_{inv}^T+F_{edit}^{Norm}$ | 🌐 Code

📄 Cones: Concept Neurons in Diffusion Models for Customized Generation | 📖 ICML 2023 | 🔀 $F_{inv}^T+F_{edit}^{Norm}$ | 🌐 Code

📄 SVDiff: Compact Parameter Space for Diffusion Fine-Tuning | 📖 ICCV 2023 | 🔀 $F_{inv}^T+F_{edit}^{Norm}$ | 🌐 Code

📄 Low-Rank Adaptation for Fast Text-to-Image Diffusion Fine-Tuning | 📖 | 🔀 $F_{inv}^T+F_{edit}^{Norm}$| 🌐 Code

📄 A Closer Look at Parameter-Efficient Tuning in Diffusion Models | 📖 Arxiv 2023 | 🔀 $F_{inv}^T+F_{edit}^{Norm}$ | 🌐 Code

📄 Break-a-scene: Extracting multiple concepts from a single image | 📖 SIGGRAPH 2023 | 🔀 $F_{inv}^T+F_{edit}^{Norm}$ | 🌐 Code

📄 Clic: Concept Learning in Context | 📖 Arxiv 2023 | 🔀 $F_{inv}^T+F_{edit}^{Norm}$ | 🌐 Code

📄 Disenbooth: Disentangled parameter-efficient tuning for subject-driven text-to-image generation | 📖 Arxiv 2023 | 🔀 $F_{inv}^T+F_{edit}^{Norm}$ | 🌐 Code

📄 Decoupled Textual Embeddings for Customized Image Generation | 📖 AAAI 2024 | 🔀 $F_{inv}^T+F_{edit}^{Norm}$ | 🌐 Code

📄 ViCo: Detail-Preserving Visual Condition for Personalized Text-to-Image Generation | 📖 Arxiv 2023 | 🔀 $F_{inv}^T+F_{edit}^{Attn}$ | 🌐 Code

📄 DreamMatcher: Appearance Matching Self-Attention for Semantically-Consistent Text-to-Image Personalization | 📖 CVPR 2024 | 🔀 $F_{inv}^T+F_{edit}^{Attn}$ | 🌐 Code

📄 Pick-and-Draw: Training-free Semantic Guidance for Text-to-Image Personalization | 📖 Arxiv 2024 | 🔀 $F_{inv}^F+F_{edit}^{Optim}$ | 🌐 Code

2. Training-Based Approaches

📄 Taming Encoder for Zero Fine-tuning Image Customization with Text-to-Image Diffusion Models | 📖 ICLR 2024 | 🌐 Code

📄 InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning | 📖 CVPR 2024] | 🌐 Code

📄 Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Models | 📖 Arxiv 2023 | 🌐 Code

📄 Enhancing Detail Preservation for Customized Text-to-Image Generation: A Regularization-Free Approach | 📖 ICLR 2024 | 🌐 Code

📄 FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attention | 📖 Arxiv 2023 | 🌐 Code

📄 PhotoMaker: Customizing Realistic Human Photos via Stacked {ID} Embedding | 📖 Arxiv 2023 | 🌐 Code

📄 PhotoVerse: Tuning-Free Image Customization with Text-to-Image Diffusion Models | 📖 Arxiv 2023 | 🌐 Code

📄 InstantID: Zero-shot Identity-Preserving Generation in Seconds | 📖 Arxiv 2024 | 🌐 Code

📄 ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generation | 📖 ICCV 2023 | 🌐 Code

📄 BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing | 📖 NeurIPS 2023 | 🌐 Code

📄 Domain-Agnostic Tuning-Encoder for Fast Personalization of Text-To-Image Models | 📖 SIGGRAPH 2023 | 🌐 Code

📄 Unified Multi-Modal Latent Diffusion for Joint Subject and Text Conditional Image Generation | 📖 Arxiv 2023 | 🌐 Code

📄 Subject-Diffusion: Open Domain Personalized Text-to-Image Generation without Test-time Fine-tuning | 📖 Arxiv 2023 | 🌐 Code

📄 Instruct-Imagen: Image Generation with Multi-modal Instruction | 📖 Arxiv 2024 | 🌐 Code


Attribute-Driven Customization:

1. Training-Free Approaches

📄 ProSpect: Prompt Spectrum for Attribute-Aware Personalization of Diffusion Models | 📖 Arxiv 2023 | 🔀 $F_{inv}^T+F_{edit}^{Norm}$ | 🌐 Code

📄 An Image is Worth Multiple Words: Multi-attribute Inversion for Constrained Text-to-Image Synthesis | 📖 Arxiv 2023 | 🔀 $F_{inv}^T+F_{edit}^{Norm}$ | 🌐 Code

📄 Concept Decomposition for Visual Exploration and Inspiration | 📖 TOG 2023 | 🔀 $F_{inv}^T+F_{edit}^{Norm}$ | 🌐 Code

📄 ReVersion: Diffusion-Based Relation Inversion from Images | 📖 Arxiv 2023 | 🔀 $F_{inv}^T+F_{edit}^{Norm}$ | 🌐 Code

📄 Learning Disentangled Identifiers for Action-Customized Text-to-Image Generation | 📖 Arxiv 2023 | 🔀 $F_{inv}^T+F_{edit}^{Norm}$ | 🌐 Code

📄 Lego: Learning to Disentangle and Invert Concepts Beyond Object Appearance in Text-to-Image Diffusion Models | 📖 Arxiv 2023 | 🔀 $F_{inv}^T+F_{edit}^{Norm}$ | 🌐 Code

📄 StyleDrop: Text-to-Image Generation in Any Style | 📖 NeurIPS 2023 | 🔀 $F_{inv}^T+F_{edit}^{Norm}$ | 🌐 Code

2. Training-Based Approaches

📄 ArtAdapter: Text-to-Image Style Transfer using Multi-Level Style Encoder and Explicit Adaptation | 📖 Arxiv 2023 | 🌐 Code

📄 DreamCreature: Crafting Photorealistic Virtual Creatures from Imagination | 📖 Arxiv 2023 | 🌐 Code

📄 Language-Informed Visual Concept Learning | 📖 ICLR 2024| 🌐 Code

📄 pOps: Photo-Inspired Diffusion Operators | 📖 Arxiv 2024 | 🌐 Code