FreeCompose:
Generic Zero-Shot Image Composition with Diffusion Prior

Zhekai Chen* · Wen Wang* · Zhen Yang · Zeqing Yuan · Hao Chen · Chunhua Shen
Zhejiang University

We offer a novel approach to image composition, which integrates multiple input images into a single, coherent image. Rather than concentrating on specific use cases such as appearance editing (image harmonization) or semantic editing (semantic image composition), we showcase the potential of utilizing the powerful generative prior inherent in large-scale pre-trained diffusion models to accomplish generic image composition applicable to both scenarios.

We observe that the pre-trained diffusion models automatically identify simple copy-paste boundary areas as low-density regions during denoising. Building on this insight, we propose to optimize the composed image towards high-density regions guided by the diffusion prior. In addition, we introduce a novel mask-guided loss to further enable flexible semantic image composition.

Extensive experiments validate the superiority of our approach in achieving generic zero-shot image composition. Additionally, our approach shows promising potential in various tasks, such as object removal and multi-concept customization.

Method

FreeCompose overview. Our FreeCompose pipeline consists of three phases: object removal, image harmonization, and semantic image composition. In each phase, the pipeline takes an input image and two text prompts to calculate the loss. In the object removal phase, an additional mask is required to select K, V values. In the semantic image composition phase, text prompts can be replaced by other formats, and an additional K, V replacement is implemented for identity consistency.

Demos

Object Removal

Imgae Harmonization

Semantic Image Composition

SDXL Plug-and-Play

Code Structure

• assets/: contains the data for cases.
• eval/: contains the evaluation code, including fid, lpips and so on.
• utils/: contains the utility functions for the project.
• main.ipynb: the main notebook for the project.

Environment Setup

conda create -n freecompose python=3.9.11
conda activate freecompose
pip install -r requirements.txt

Note: Because of proxy reason, we use hf-mirror as the mirror for huggingface. However, maybe you need to use the original mirror. Then, comment or delete the following line in some files:

os.environ['HF_ENDPOINT'] = "https://hf-mirror.com"

Usage

We provide a notebook for the project. You can run the notebook to see the results of the project.

Acknowledgement

We would like to thank the authors of the following repositories for their great work:

• Prompt-2-prompt
• Controlcom
• Paint-by-Example

License

For non-commercial academic use, this project is licensed under the 2-clause BSD License. For commercial use, please contact Chunhua Shen.

Citing

If you find our work useful, please consider citing:

@article{FreeCompose,
  title={FreeCompose: Generic Zero-Shot Image Composition with Diffusion Prior},
  author={Chen, Zhekai and Wang, Wen and Yang, Zhen and Yuan, Zeqing and Chen, Hao and Shen, Chunhua},
  journal={arXiv preprint arXiv:2407.04947},
  year={2024}
}