Generative-Active-Learning-for-Image-Synthesis-Personalization

Description:
This is the official implementation of Generative Active Learning for Image Synthesis Personalization. Our code is built on diffusers.

Generative-Active-Learning-for-Image-Synthesis-Personalization
Xulu Zhang^1,2, Wengyu Zhang¹, Xiaoyong Wei¹, Jinlin Wu^2,4, Zhaoxiang Zhang^2,3,4, Zhen Lei^2,3,4, Qing Li^1
1Department of Computing, the Hong Kong Polytechnic University,
²Center for Artificial Intelligence and Robotics, HKISI, CAS,
³School of Artificial Intelligence, UCAS,
⁴State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA

Abstract:
This paper presents a pilot study that explores the application of active learning, traditionally studied in the context of discriminative models, to generative models. We specifically focus on image synthesis personalization tasks. The primary challenge in conducting active learning on generative models lies in the open-ended nature of querying, which differs from the closed form of querying in discriminative models that typically target a single concept. We introduce the concept of anchor directions to transform the querying process into a semi-open problem. We propose a direction-based uncertainty sampling strategy to enable generative active learning and tackle the exploitation-exploration dilemma. Extensive experiments are conducted to validate the effectiveness of our approach, demonstrating that an open-source model can achieve superior performance compared to closed-source models developed by large companies, such as Google's StyleDrop.

Description

This repo contains the official implementation of GAL. Our code is built on diffusers.

Setup

To set up the environment, please run:

git clone https://github.com/zhangxulu1996/GAL4Personalization.git
cd GAL4Personalization
conda create -n gal python=3.9
conda activate gal
pip install -r requirements.txt

Data Preparation

We conduct experiments on the concepts used in previous studies. You can find the code and resources for the "Custom Diffusion" concept here and for the "Textual Inversion" concept here.

Dreambooth and Custom Diffusion use a small set of real images to prevent overfitting. You can refer this guidance to prepare the regularization dataset.

The data directory structure should look as follows:

├── reg_data
│   └── [concept name]
│   │   └── images
│   │       └── [regularization images]
├── reference_images
│   └── [object/style]
│   │   └── [concept name]
│   │   │   └── [reference images]
│   │   │   └── [init.json]

Regularization Dataset

You can download the regularization dataset we used from Google Drive. Note that these images are from LAION-400M. After downloading the regularization dataset, you can put them under the folder "reg_data". There is another available way that uses synthetic images generated by pre-trained diffusion models.

Usage

Training with GAL

To train object-driven DreamBooth on Stable Diffusion with GAL, run:

sh scripts/run_obj_gal.sh

To train style-driven DreamBooth on Stable Diffusion with GAL, run:

sh scripts/run_style_gal.sh

Reproduce Results

To reproduce the results in the paper, please refer to the reproduce_object and reproduce_style notebook. It contains the necessary code and instructions.

Results

The sample results obtained from our proposed method:

Citation

If you make use of our work, please cite our paper:

@inproceedings{zhang2024generative,
  title={Generative active learning for image synthesis personalization},
  author={Zhang, Xulu and Zhang, Wengyu and Wei, Xiaoyong and Wu, Jinlin and Zhang, Zhaoxiang and Lei, Zhen and Li, Qing},
  booktitle={ACM Multimedia 2024},
  year={2024}
}