CLIP-IQA, Optimised GPU Usage and Some Fixes

PyTorch Image Quality (PIQ) v0.8.0 Release Notes

• CLIP-IQA (#331)
• Fixes (#359, #364, #368)

New Features

CLIP-IQA

With this release, we introduce implementation of CLIP-IQA. In contrast to other available implementations, our apprach is standalone (say no to any additional dependencies) and delivers the estimations, which match the official implementation.

The image quality is measured based on a general notion of text-to-image similarity learned by the CLIP model during its large-scale pre-training on a large dataset with paired texts and images.
Such approach follows the idea that two antonyms (“Good photo” and “Bad photo”) can be used as anchors in the text embedding space representing good and bad images in terms of their image quality.
After the anchors are defined, one can use them to determine the quality of a given image in the following way:

1. Compute the image embedding of the image of interest using the pre-trained CLIP model;
2. Compute the text embeddings of the selected anchor antonyms;
3. Compute the angle (cosine similarity) between the image embedding (1) and both text embeddings (2);
4. Compute the Softmax of cosine similarities (3) -> CLIP-IQA score.

Fixes

• Added proper meshgrid indexing (#359);
• Fixed usage of metrics on GPU (#364);
• Added documentation for new measures (#368);

Full Changelog: v0.7.1...v0.8.0

Contributors: @rbischof, @zakajd, @snk4tr, @denproc.

Optimisation of CPU usage for FSIM and VSI, Enhanced Benchmarking and Installation, Updated Integration, Deployment and Documentation

PyTorch Image Quality (PIQ) v0.7.1 Release Notes: Optimisations

• Enhancements (#317, #330, #325, #326, #334, #339, #340, #337, #338, #341, #342, #356)

Enhancements

• Added SR-SIM and SR-SIMc to benchmark script (#317);
• Updated Github CI/CD (#330, #339, #340);
• Added batch-wise computation of features for feature-based metrics (#325);
• Updated Readme (26d044e, #326);
• Optmised CPU usage for FSIM and VSI (#334, #342);
• Optimised setup (#337);
• Added Conda deployment for generic environment (#338);
• Added Manifest (#356);

Contributors: @pooya-mohammadi , @zakajd, @snk4tr, @denproc.

Full Changelog: v0.7.0...v0.7.1

Information Content Weighted SSIM (IW-SSIM), Pre-Commit Hooks and Updated Benchmarking

PyTorch Image Quality (PIQ) v0.7.0 Release Notes

• Information Content Weighted Structural Similarity (IW-SSIM) index (#301, #311);
• Pre-commit Hooks (#293);
• Enhancements (#273, #276, #286, #285, #290, #296, #292, #302, #305, #313, #270);

New Features

Information Content Weighted Structural Similarity (IW-SSIM) Index (#301, #311)

The new release of PIQ introduces metric and loss function interfaces for Information Content Weighted Structural Similarity (IW-SSIM) index. PIQ implementation of IW-SSIM is standalone. It doesn't require any additional packages for computing the index and includes optimised use of Laplacian pyramids. The performance of the proposed IW-SSIM index matches SRCC and KRCC estimations on TID2013, and KADID10k datasets.

IW-SSIM was proposed by Zhou Wang and Qiang Li. IW-SSIM takes into account local information content for image quality assessment. The content is estimated using advanced statistical models of natural images. Experiments showed consistent improvement in the performance for IQA, while IW-SSIM led to the best overall performance.

Pre-Commit Hooks (#293)

With this release, pre-commit hooks were added to the project. Pre-commit hooks enable automatic validation of code before committing it into the project. Such approach allows to automate code style checks locally and reduces number of commits to fix code style issues. To enable the feature on your machine, please follow the contribution guide.

Enhancements

• Updated library version parsers to meet semver and PEP404 formats (#273, #290);
• Updates for github/actions (#296);
• Upgrade for variable creation depending on input type (#292);
• Upgrade Benchmarking for metrics delivered with PIQ package (#270);
• Update README file (#270);

Bug Fix

• Fix for documentation description of feature-base metrics (#276);
• Fix for MDSI loss computation (#286);
• Fix for Precision and Recall metrics (#285);
• Fix for PSNR description (#302);
• Fix exceptions for some metric (#305);
• Fix BRISQUE interpolation to match MATLAB resize (#313);

Contributors: @merunes-goldman, @zakajd, @snk4tr, @denproc.

SR-SIM, DSS, PIPAL Benchmark and Enhancements

PyTorch Image Quality (PIQ) v0.6.0 Release Notes

• Spectral Residual based Similarity (SR-SIM, SR-SIMc) Metric (#202)
• DCT Subbands Similarity (DSS) Metric (#225, #268)
• Benchmark on PIPAL dataset (#269)

New Features

Spectral Residual based Similarity (#202)

With current release, we added Spectral Residual based Similarity (SR-SIM) measure. The metric was introduced based on a specific visual saliency model, spectral residual visual saliency.
In addition, we also implemented SR-SIMc, which is a chromatic version of the SR-SIM.

DCT Subbands Similarity (DSS) (#225, #268)

DCT Subbands Similarity (DSS) was presented visual quality metric that correlates well with human visual perception. The measure uses properties of human visual perception, evaluating changes in structural information in sub-bands in DCT domain. DSS showed great results according to public image datasets benchmarks, while being computationally efficient.

PIPAL Benchmark (#269)

In this release we added another public image dataset benchmark. PIPAL is the largest human-rated set of images to date and the only one containing rich number of realistic distortions from GAN models. Benchmarking metrics performance on this set can give a good estimate of their usefulness in GAN modes evaluation.
Benchmark results are available at README.rst and documentatioin.

Bug Fix

• Fixed readme formatting on pypi (#263);
• Added type check for ContentLoss before copying weights tensor (#264);
• Fixed bug with layers/weighs length in the ContentLoss (#259);

Contributors: @zakajd, @snk4tr, @denproc, @leihuayi.

Precision-Recall (P&R), Documentation and Enhancements

PyTorch Image Quality (PIQ) v0.5.5 Release Notes

• Precision-Recall (P&R) Metric (#247)
• Documentation (#217)
• Enhancements (#211, #219, #220, #229, #230, #233, #234)
• Bug Fix (#213, #237, #244, #238, #246, #250, #243)

New Features

Precision-Recall (#247)

In this release, we added new Precision-Recall metric. The metric was introduced for assessing generative models, i.e. estimate the quality and coverage of the generated samples. The metric can separately and reliably measure both of these aspects in image generation tasks by forming explicit, non-parametric representations of the manifolds of real and generated data.

Documentation (#217)

With this release we introduce piq.readthedocs.io documentation to provide assistance using our library. Installation and usage guides help to start using PIQ framework in your projects. The documentation includes relevant information about metric interfaces and metric specific differences. In order to keep the documentation up-to-date, we use automatic pipeline for generation and deployment to piq.readthedocs.io. In addition, we updated in #233 pipeline allows to update README.rst and documentation simultaneously.

Enhancements

• Updated README with benchmarking results and references (#211);
• Updated PR template with check list for new metrics (#219);
• Added BibTex Citation (#220);
• Added guide for documentation upgrades (#229);
• Unified tensor names for all metrics, measures and losses (#230);
• Upgraded documentation pipeline to keep both documentation and README.rst up-to-date (#233);
• Upgraded input validation to simplify the interface and allow advanced users to turn off validation (#234);

Bug Fix

• Fixed missing average pooling in SSIM, implementation enhancements (#213);
• Removed rarely used library dependencies from requirements.txt (#237);
• Added exceptional import to MSID due to updated requirements.txt (#244);
• Added downsampling to DISTS boosting the performance (#238);
• Fixed use of FFT due to interface changes introduced in torch==1.8.1 (#246);
• Fixed the condition for FFT due to interface changes introduced in torch==1.8.1(#250);
• Unified documentation style across all docstrings (#243);

Contributors: @zakajd, @snk4tr, @denproc, @hecoding.

PieAPP, Usability Enhancements and Bug Fixes

PyTorch Image Quality (PIQ) v0.5.4 Release Notes

New metric, readme update, usability enhancements and bug fixes.

New Features

New metric - PieAPP (#184)

Documentation

• README update and small readme-related interface changed (#204, #206)

Enhancements

• Unified tensor descriptions (#172) bring more consistency between different metrics and measures
• More flexibility with respect to input data with a new allow_negative flag for some metrics, which support negative inputs (#169)
• MyPy tests bringing more reliability are a part of our CI pipeline (#180)
• More efficient computations of FID (no additional CPU load while computed on GPU) (#186)
• New tests for correct data_range for virtually all present metrics (#195)
• Corrections in the fid_inception in case of normalize_mode=False (#191)
• Bug fixes in FSIMc (#200) and VIFp (#210)

Contributors: @zakajd, @snk4tr, @akamaus, @bes-dev, @denproc.

Haar Wavelet-Based Perceptual Similarity Index, Documentation and other Enhancements

PyTorch Image Quality (PIQ) v0.5.2 Release Notes

• New Feature: Haar Wavelet-Based Perceptual Similarity Index (#164)
• Documentation (#167, #174 )
• Enhancements (#156, #158, #162, #175, #171)

New Features

Haar Wavelet-Based Perceptual Similarity Index (HaarPSI) (#164)

Documentation

• Fixed the description of the arguments of LPIPS and DISTS (#167)
• Updated README (#174)

Enhancements

• Got rid of redundant warning checking the type of the input tensor (#156)
• Made testing 10 times faster after update of scipy version to use (#158)
• Added the missing parameter to the GMSD, MS-GMSD and MS-GMSDc metrics (#162)
• Added no_grad context for computing features, where is no need in gradients (#175)
• Fixed logic of the GMSD test for different pixel value ranges (#171)

Contributors: @zakajd, @PavelOstyakov, @snk4tr, @denproc, .

MDSI, GMSD interfaces, Anaconda Cloud Deployment, CI and Bugfix

PyTorch Image Quality (PIQ) v0.5.1 Release Notes

• New Feature: Mean Deviation Similarity Index (MDSI) (#148)
• New Features: Functional interface for Gradient Magnitude Similarity Deviation (GMSD) and Multi-Scale Gradient Magnitude Similarity Deviation (MS-GMSD) (#151)
• Project Infrastructure: Package Deployment to Anaconda Cloud (#138)
• Project Infrastructure: Optimisation of CI workflows (#153)
• Bugfix (#140, #144, #146, #148, #154)

New Features

Mean Deviation Similarity Index (MDSI) (#148)

With this release we introduce Mean Deviation Similarity Index (MDSI). The proposed implementation is similar to original MATLAB implementation and supports the same functionality. Check README.md for usage examples.

Functional interface for Gradient Magnitude Similarity Deviation (GMSD) and Multi-Scale Gradient Magnitude Similarity Deviation (MS-GMSD) (#151)

Now, the functional interfaces for GMSD and Multi-Scale GMSD are available for the users. The precision was improved for each metric making GMSD prediction fully aligned with the original MATLAB version. Visit README.md for new usage examples.

Project Infrastructure

Package Deployment at Anaconda Cloud (#138)

The PIQ framework was added to Anaconda Cloud at photosynthesis-team channel. The proposed CD pipeline allows deploying the latest release to the Anaconda Cloud automatically. The library is compatible with win-64, linux-64 and osx-64 and python>=3.6. For installation tips, visit the README.md.

Optimisation of CI workflows (#153)

The testing CI was extended with validation using python 3.8 resulting in the library tested for python 3.6, 3.7, 3.8. The flake8 CI validation using python 3.7 was deprecated because it duplicates the functionality of the same validation using python 3.6.

Bugfix

Fixed Import of Feature Encoders (#140)

Added the description of the feature extractors into piq.feature_extractors.__init__.py for more convenient user experience.

Description of the LPIPS, DISTS, Content loss and Style loss (#144)

Added the description of the LPIPS, DISTS, ContentLoss and StyleLoss and their usage to the README.md.

Add chromatic parameter to FSIM loss (#146)

Added the chromatic parameter of the FSIMLoss to take into account chromatic components computing the FSIM.

Minor enhancements (#148)

• Added the validation of the kernel_size to be odd in brisque function;
• Added pow_for_complex to calculate tensors of any values (real and complex) in the power of any real number;
• Changed the padding for MS-SSIM to be similar to TensorFlow version;
• Updated docstring to the same format (tabulation).

Less strict requirements (#154)

The requirements were made less strict to avoid force update of the environment with torch for incompatible CUDA support during piq installation. The brisque measure was updated with a warning that back propagation is not available for torch==1.5.0 due to bug in argmin and argmax. The same warning was added to README.md.

Contributors: @denproc, @zakajd, @snk4tr.

FSIM(c), VSI, Perceptual Losses and piq.Functional

PyTorch Image Quality (PIQ) v0.5.0 Release Notes

• New feature: Perceptual losses (StyleLoss, ContentLoss , LPIPS, DISTS)
• New feature: Visual Saliency-induced Index (VSI)
• New feature: Feature Similarity (FSIM/FSIMc)
• New feature: piq.functional
• Code quality assessment
• BugFix

New Features

Perceptual losses (StyleLoss, ContentLoss , LPIPS and DISTS) (#96, #131)

Content loss, Style loss, Learned Perceptual Image Patch Similarity (LPIPS) and Deep Image Structure and Texture Similarity (DISTS) metrics were added to the library. One could choose the extractor VGG16 or VGG19. Any number of layers is supported. Any weights are supported. One can use per-level (Style transfer case) or per-feature (as in LPIPS) weights. Check out Examples at README.md for more details.

Visual Saliency-induced Index (VSI) (#119)

The Visual Saliency-induced Index (VSI) was introduced into the library in this release. The VSI is compatible with greyscale and RGB images. In case of greyscale, the only channel is copied to imitate RGB colour space. The VSI was compared with the original MATLAB implementation. Tests show that our implementation produces the same result. Check out Examples at README.md for more details.

Feature Similarity (FSIM/FSIMc) (#104)

The Feature Similarity (FSIM/FSIMc) was also included in this release. The FSIM measure is compatible with greyscale and RGB images, but only the luminance component is used. The FSIMc extension takes chrominance components into account. The FSIM and FSIMc were compared with the original MATLAB implementation. Tests show that our implementation produces the same result. Check out Examples at README.md for more details.

piq.functional (#104, #119, #135)

With this release, we introduce piq.functional, which contains the most common functions such as colour space conversions, filters, mesh grid, similarity and gradient maps. The decision was made in order to reduce copy-paste code and improve readability providing the user with the ability to reuse implemented functionality for their own needs.

Code quality assessment (#99)

CodeCov was introduced to the project to improve the quality of the contributed code to the library. The Codecov bot posts reports directly to pull requests highlighting the test coverage of the code.

BugFixes

GS (#112)

The issue (#111) with memory losses and the number of open files compatibility was resolved. The context manager was added to handle pool closure after its use.

BRISQUE Data Range (#110)

The issue (#108) with pixel values range to be used was resolved. The functionality is aligned with MATLAB implementation.

VIF (#94)

• Tests were added to cover bug behaviour;
• EPS constant was added before division to avoid NaN;
• Some checks were introduced from original MATLAB implementation that have been missed;
• Reduction parameter was added;
• Another small bug was fixed.

SonarCloud (#117)

The code was refactored to meet the code style requirements and to remove sonarcloud code smells.

SSIM/MS-SSIM and BRISQUE Refactoring (#134)

• The implementations of the SSIM/MS-SSIM and BRISQUE metrics were optimised for better readability and utility;
• The implementations were compared with available TensorFlow and MATLAB implementation on real images using new tests;
• Fixed huge memory consumption during SSIM/MS-SSIM testing (#126);
• Other tests were refactored to improve readability and coverage. It allows flexible testing on all available devices (cpu and/or gpu) depending on the workstation for deployment.

Contributors: @bonlime, @denproc, @zakajd, @snk4tr.

New Title and BugFix

PyTorch Image Quality (PIQ) v0.4.1 Release Notes

• New title
• New feature - PSNR
• Code quality assessment
• BugFix

New Title - PyTorch Image Quality (PIQ) (#107)

New easy to remember title "PyTorch Image Quality (PIQ)" is introduced for our library with this release. This change allows importing all the metrics easier using just three letters piq. Moreover, the new title resolves the consistency between naming in different services (GitHub, PyPi). As a result, one could use automatic tools of IDE to detect if the library is messed and to install it straight away.

New Feature - Peak Signal-to-Noise Ratio (PSNR) (#95)

The Peak Signal-to-Noise Ratio (PSNR) was added to the library in this release. The PSRN is compatible with greyscale and RGB images. In the case of an RGB image, one could use all three channels to compute the ratio or transfer image to YCbCr colour space and use luminance. Check out Examples at README.md for more details.

Code quality assessment (#99)

SonarCloud was introduced to the project to improve the quality of the contributed code to the library.

BugFixes

BRISQUE

• The issues with GPU compatibility were resolved (#87);
• Back-propagation was fixed (#88);
• New assertions were added for images with values of variance around zero (#88).

MS-GMSD

• The issues with GPU compatibility were resolved (#91).

FID

• The issues with GPU compatibility were resolved (#103);
• The inconsistent result of the method due to inlace operations was fixed (#103);
• The convergence issues of Newton-Schulz approach due to float precision were mended (#103).

Contributors: @denproc, @zakajd, @snk4tr.
Special thanks to @PavelOstyakov for valuable feedback.