A Novel Patch Variance Biased Convolutional Neural Network for No-Reference Image Quality Assessment

Lai-Man Po; Mengyang Liu; Wilson Y. F. Yuen; Yuming Li; Xuyuan Xu; Chang Zhou; Peter H. W. Wong; Kin Wai Lau; Hon-Tung Luk

doi:10.1109/TCSVT.2019.2891159

A Novel Patch Variance Biased Convolutional Neural Network for No-Reference Image Quality Assessment

Lai-Man Po^*
, Mengyang Liu
, Wilson Y. F. Yuen
, Yuming Li
, Xuyuan Xu
, Chang Zhou
, Peter H. W. Wong
, Kin Wai Lau
, Hon-Tung Luk

^*Corresponding author for this work

Department of Electrical Engineering

Research output: Journal Publications and Reviews › RGC 21 - Publication in refereed journal › peer-review

53 Citations (Scopus)

Abstract

Deep Convolutional Neural Networks (CNNs) have been successfully applied on no-reference image quality assessment (NR-IQA) with respect to human perception. Most of these methods deal with small image patches and use the average score of the test patches for predicting the whole image quality. We discovered that image patches from homogenous regions are unreliable for both neural network training and final image quality score estimation. In addition, image patches with complex structures have much higher chances to achieve better image quality prediction. Based on these findings, we enhanced the conventional CNN-based NR-IQA algorithm to avoid homogenous patches for the network training and quality score estimation. Moreover, we also use a variance-based weighting average to bias the final image quality score to the patches with complex structure. Experimental results show that this simple approach can achieve state-of-the-art performance as compared with well-known NR-IQA algorithms.

Original language	English
Pages (from-to)	1223-1229
Journal	IEEE Transactions on Circuits and Systems for Video Technology
Volume	29
Issue number	4
Online published	7 Jan 2019
DOIs	https://doi.org/10.1109/TCSVT.2019.2891159
Publication status	Published - Apr 2019

Funding

Manuscript received March 18, 2018; revised October 22, 2018; accepted January 1, 2019. Date of publication January 7, 2019; date of current version April 3, 2019. This work was supported by a grant from the Innovation and Technology Fund (ITF) of Hong Kong Government in the City University of Hong Kong under Project 9440172. This paper was recommended by Associate Editor B. Li. (Corresponding author: Lai-Man Po.) L.-M. Po, M. Liu, and C. Zhou are with the Department of Electronic Engineering, City University of Hong Kong, Hong Kong (e-mail: [email protected]). W. Y. F. Yuen, P. H. W. Wong, K. W. Lau, and H.-T. Luk are with TFI Digital Media Ltd., Hong Kong. Y. Li is with Minieye Company, Shenzhen, China. X. Xu is with Tencent Video, Tencent Holdings Ltd., Shenzhen 518057, China. Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TCSVT.2019.2891159

Research Keywords

Convolution
convolution neural network
Convolutional neural networks
deep learning
Estimation
Image color analysis
Image quality
no-reference image quality assessment
Training

Access Output

10.1109/TCSVT.2019.2891159

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

@article{970fa2910c2d4b55a80a2823e089cb18,

title = "A Novel Patch Variance Biased Convolutional Neural Network for No-Reference Image Quality Assessment",

abstract = "Deep Convolutional Neural Networks (CNNs) have been successfully applied on no-reference image quality assessment (NR-IQA) with respect to human perception. Most of these methods deal with small image patches and use the average score of the test patches for predicting the whole image quality. We discovered that image patches from homogenous regions are unreliable for both neural network training and final image quality score estimation. In addition, image patches with complex structures have much higher chances to achieve better image quality prediction. Based on these findings, we enhanced the conventional CNN-based NR-IQA algorithm to avoid homogenous patches for the network training and quality score estimation. Moreover, we also use a variance-based weighting average to bias the final image quality score to the patches with complex structure. Experimental results show that this simple approach can achieve state-of-the-art performance as compared with well-known NR-IQA algorithms.",

keywords = "Convolution, convolution neural network, Convolutional neural networks, deep learning, Estimation, Image color analysis, Image quality, no-reference image quality assessment, Training",

author = "Lai-Man Po and Mengyang Liu and Yuen, \{Wilson Y. F.\} and Yuming Li and Xuyuan Xu and Chang Zhou and Wong, \{Peter H. W.\} and Lau, \{Kin Wai\} and Hon-Tung Luk",

year = "2019",

month = apr,

doi = "10.1109/TCSVT.2019.2891159",

language = "English",

volume = "29",

pages = "1223--1229",

journal = "IEEE Transactions on Circuits and Systems for Video Technology",

issn = "1051-8215",

publisher = "IEEE",

number = "4",

}

A Novel Patch Variance Biased Convolutional Neural Network for No-Reference Image Quality Assessment. / Po, Lai-Man ; Liu, Mengyang; Yuen, Wilson Y. F. et al.
In: IEEE Transactions on Circuits and Systems for Video Technology, Vol. 29, No. 4, 04.2019, p. 1223-1229.

Research output: Journal Publications and Reviews › RGC 21 - Publication in refereed journal › peer-review

TY - JOUR

T1 - A Novel Patch Variance Biased Convolutional Neural Network for No-Reference Image Quality Assessment

AU - Po, Lai-Man

AU - Liu, Mengyang

AU - Yuen, Wilson Y. F.

AU - Li, Yuming

AU - Xu, Xuyuan

AU - Zhou, Chang

AU - Wong, Peter H. W.

AU - Lau, Kin Wai

AU - Luk, Hon-Tung

PY - 2019/4

Y1 - 2019/4

N2 - Deep Convolutional Neural Networks (CNNs) have been successfully applied on no-reference image quality assessment (NR-IQA) with respect to human perception. Most of these methods deal with small image patches and use the average score of the test patches for predicting the whole image quality. We discovered that image patches from homogenous regions are unreliable for both neural network training and final image quality score estimation. In addition, image patches with complex structures have much higher chances to achieve better image quality prediction. Based on these findings, we enhanced the conventional CNN-based NR-IQA algorithm to avoid homogenous patches for the network training and quality score estimation. Moreover, we also use a variance-based weighting average to bias the final image quality score to the patches with complex structure. Experimental results show that this simple approach can achieve state-of-the-art performance as compared with well-known NR-IQA algorithms.

AB - Deep Convolutional Neural Networks (CNNs) have been successfully applied on no-reference image quality assessment (NR-IQA) with respect to human perception. Most of these methods deal with small image patches and use the average score of the test patches for predicting the whole image quality. We discovered that image patches from homogenous regions are unreliable for both neural network training and final image quality score estimation. In addition, image patches with complex structures have much higher chances to achieve better image quality prediction. Based on these findings, we enhanced the conventional CNN-based NR-IQA algorithm to avoid homogenous patches for the network training and quality score estimation. Moreover, we also use a variance-based weighting average to bias the final image quality score to the patches with complex structure. Experimental results show that this simple approach can achieve state-of-the-art performance as compared with well-known NR-IQA algorithms.

KW - Convolution

KW - convolution neural network

KW - Convolutional neural networks

KW - deep learning

KW - Estimation

KW - Image color analysis

KW - Image quality

KW - no-reference image quality assessment

KW - Training

UR - https://www.scopus.com/pages/publications/85059632228

UR - https://www.scopus.com/record/pubmetrics.uri?eid=2-s2.0-85059632228&origin=recordpage

U2 - 10.1109/TCSVT.2019.2891159

DO - 10.1109/TCSVT.2019.2891159

M3 - RGC 21 - Publication in refereed journal

SN - 1051-8215

VL - 29

SP - 1223

EP - 1229

JO - IEEE Transactions on Circuits and Systems for Video Technology

JF - IEEE Transactions on Circuits and Systems for Video Technology

IS - 4

ER -

A Novel Patch Variance Biased Convolutional Neural Network for No-Reference Image Quality Assessment

Abstract

Funding

Research Keywords

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

Cite this