DDR: Exploiting Deep Degradation Response as Flexible Image Descriptor

The Thirty-Eighth Annual Conference on Neural Information Processing Systems (NeurIPS), 2024

Juncheng Wu¹, Zhangkai Ni², Hanli Wang², Wenhan Yang³, Yuyin Zhou¹, Shiqi Wang⁴

¹UC Santa Cruz, ²Tongji University, ³Pengcheng Laboratory, ⁴City University of Hong Kong

This repository provides the official PyTorch implementation for the paper “DDR: Exploiting Deep Degradation Response as Flexible Image Descriptor”, NeurIPS-2024. Paper

About DDR

Image deep features extracted by pre-trained networks are known to contain rich and informative representations. In this paper, we present Deep Degradation Response (DDR), a method to quantify changes in image deep features under varying degradation conditions. Specifically, our approach facilitates flexible and adaptive degradation, enabling the controlled synthesis of image degradation through text-driven prompts. Extensive evaluations demonstrate the versatility of DDR as an image descriptor, with strong correlations observed with key image attributes such as complexity, colorfulness, sharpness, and overall quality. Moreover, we demonstrate the efficacy of DDR across a spectrum of applications. It excels as a blind image quality assessment metric, outperforming existing methodologies across multiple datasets. Additionally, DDR serves as an effective unsupervised learning objective in image restoration tasks, yielding notable advancements in image deblurring and single-image super-resolution.

TL;DR: We propose a novel image descriptor based on deep degradation response. We use a pre-trained CLIP model to fuse text-guided degradation into image deep features. Our method demonstrates promising results on low-level tasks on opinion-unaware BIQA, image deblurring, and real-world image super-resolution.

Experimental Results

1. Opinion-unaware BIQA

Performance comparisons of different OU-BIQA models on eight public datasets using SRCC. The top performer on each dataset is marked in bold.

2. Image Deblurring

Quantitative result of image motion deblurring. Combining proposed DDR with reconstruction loss leads to result with less degradation and higher fidelity.

3. Real-world Image Super-resolution

Quantitative result on real-world SISR dataset

Quick Start

• Python>=3.6
• Pytorch>=1.8
• PIL

image = Image.open("test.png")
x = torch.stack([ToTensor()(image)]).to(device)	# to batch
criterion = DDR()
res = criterion(x)

Citation

If you find our work useful, please cite it as

@article{wu2024ddr,
  title={DDR: Exploiting Deep Degradation Response as Flexible Image Descriptor},
  author={Wu, Juncheng and Ni, Zhangkai and Wang, Hanli and Yang, Wenhan and Zhou, Yuyin and Wang, Shiqi},
  booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
  year={2024}
}

Contact

Thanks for your attention! If you have any suggestions or questions, feel free to leave a message here or contact Dr. Zhangkai Ni ([email protected]) or Juncheng Wu ([email protected])

License

MIT License