PromptIQA: Boosting the Performance and Generalization for No-Reference Image Quality Assessment via Prompts

• The 18th European Conference on Computer Vision ECCV 2024

---

🚀 🚀 🚀 News:

• To be updated...
• ✅ February, 2025: We pubulish the training code.
• ✅ September, 2024: We pubulish the checkpoints and testing code.
• ✅ September, 2024: We pubulish the online demo.
• ✅ March, 2024: We created this repository.

Checklist

• Code for training
• Code for PromptIQA
• Code for testing
• Checkpoint
• Online Demo on huggingface

Catalogue

1. Introduction
2. Try Our Demo
3. Usage For Training
4. Usage For Testing
5. Results
6. Citation
7. Acknowledgement

Introduction

This is an official implementation of PromptIQA: Boosting the Performance and Generalization for No-Reference Image Quality Assessment via Prompts by Pytorch.

---

Due to the diversity of assessment requirements in various application scenarios for the IQA task, existing IQA methods struggle to directly adapt to these varied requirements after training. Thus, when facing new requirements, a typical approach is fine-tuning these models on datasets specifically created for those requirements. However, it is time-consuming to establish IQA datasets. In this work, we propose a Prompt-based IQA (PromptIQA) that can directly adapt to new requirements without fine-tuning after training. On one hand, it utilizes a short sequence of Image-Score Pairs (ISP) as prompts for targeted predictions, which significantly reduces the dependency on the data requirements. On the other hand, PromptIQA is trained on a mixed dataset with two proposed data augmentation strategies to learn diverse requirements, thus enabling it to effectively adapt to new requirements. Experiments indicate that the PromptIQA outperforms SOTA methods with higher performance and better generalization.

Figure1: The framework of the proposed PromptIQA.

Try Our Demo 🕹️

Click 👇 to try our demo online.

Huggingface

Training and Testing

Preparation

Run the following script to create the environment:

conda create -n promptiqa python=3.8 -y

The dependencies for this work as follows:

einops==0.7.0
numpy==1.24.4
opencv_python==4.8.0.76
openpyxl==3.1.2
Pillow==10.0.0
scipy
timm==0.5.4
torch==2.0.1+cu118
torchvision==0.15.2+cu118
tqdm==4.66.1
gradio

You can also run the following command to install the environment directly:

pip install -r requirements.txt

Usage For Training

Getting the datasets

You can download the total 12 datasets from the following download link. (TBU)

Dataset	Task	Ref. Image	Dist. Types	Image Number	Label Type	Range	Download
LIVE	Synthetic Distortion Nature IQA	29	5	779	DMOS	[1, 100]	Link
CSIQ		30	6	866	DMOS	[0, 1]	Link
TID2013		25	24	3,000	MOS	[0, 9]	Link
Kadid-10k		81	25	10,125	MOS	[1, 5]	Link
BID	Authentic Distortion Nature IQA	-	-	586	MOS	[0, 5]	Link
SPAQ		-	-	11,125	MOS	[0, 100]	Link
LIVEC		-	-	1,162	MOS	[1, 100]	Link
KonIQ-10K		-	-	10,073	MOS	[0, 100]	Link
GFIQA20k	Face IQA	-	-	19,988	MOS	[0, 1]	Link
AGIQA3k	AI generated IQA	-	-	2,982	MOS	[0, 1]	Link
AIGCIQA2023		-	-	2,400	MOS	[0, 1]	Link
UWIQA	Underwater IQA	-	-	890	MOS	[0, 1]	Link

Training process

1. You should replace the dataset path in dataset_info.json to your own dataset path.

2. Run the following command to train the PromptIQA (Please review the train.py for more options).

   P.S.⚠️ Supported datasets can be found in dataset_info.json

CUDA_VISIBLE_DEVICES=4,5,6,7 \
python train.py --dist-url 'tcp://localhost:12754' \
    --dataset spaq csiq livec koniq10k kadid live \ 
    --multiprocessing-distributed --world-size 1 --rank 0 \
    --batch_size 44 --epochs 50 --seed 2024 \
    --random_flipping_rate 0.1 --random_scale_rate 0.5 \
    --model promptiqa \
    --save_path ./Exp/PromptIQA_2024

You can directly run the following script:

sh train.sh

The number of prompts is batch_size / GPU_Num - 1

Usage For Testing

Pre-training Weight

You can get our pretraining weight from Huggingface.

Then put the checkpoints in ./PromptIQA/checkpoints

Running On The Demo

You can use the following command to run the test demo:

python3 app.py

Running Testing Code

You can use the following command to run the testing code:

python3 test.py

Results

We achieved state-of-the-art performance on most IQA datasets simultaniously within one single model.

More detailed results can be found in the paper.

Individual Dataset Comparison.

Citation

If our work is useful to your research, we will be grateful for you to cite our paper:

@inproceedings{chen2024promptiqa,
  title={Promptiqa: Boosting the performance and generalization for no-reference image quality assessment via prompts},
  author={Chen, Zewen and Qin, Haina and Wang, Juan and Yuan, Chunfeng and Li, Bing and Hu, Weiming and Wang, Liang},
  booktitle={European Conference on Computer Vision},
  pages={247--264},
  year={2024},
  organization={Springer}
}

Stars ⭐️

Acknowledgement

We sincerely thank the great work HyperIQA, MANIQA and MoCo. The code structure is partly based on their open repositories.