Combining CNN and transformers for full-reference and no-reference image quality assessment

Most deep learning approaches for image quality assessment use regression from deep features extracted by CNN (Convolutional Neural Networks). However, non-local information is usually neglected in existing methods. Motivated by the recent success of transformers in modeling contextual information, we propose a hybrid framework that utilizes a vision transformer backbone to extract features and a CNN decoder for quality estimation. We propose a shared feature extraction scheme for both FR and NR settings. A two-branch structured attentive quality predictor is devised for quality prediction. Evaluation experiments on various IQA datasets, including LIVE, CSIQ and TID2013, LIVE-Challenge, KADID-10 K, and KONIQ-10 K, show that our proposed models achieve outstanding performance for both FR and NR settings. © 2023 Published by Elsevier B.V.