The goal of objective image quality assessment (IQA) is the construction of computational models that predict the perceived quality of visual images, which is of paramount importance in a variety of real-world applications, such as image restoration, compression, and rendering. Objective full-reference measures of image quality generally operate by comparing pixels of a degraded image to those of the original. Relative to human observers, the most existing IQA measures are overly sensitive to resampling of texture regions (exchanging the content of a texture region with a new sample of the same texture). Moreover, how to fairly compare the relative performance of IQA models is a challenging problem. Generally, it is evaluated by comparing model predictions to human quality judgments, but the heavy use of existing perceptual datasets creates a risk of overfitting.

In this thesis, we first develop a novel full-reference image quality metric - Deep Image Structure and Texture Similarity (DISTS) model, which shows explicit tolerance to texture resampling. Specifically, based on a convolutional neural network, we construct an injective and differentiable function that transforms images to multi-scale overcomplete representations. We then describe an image quality metric that combines correlations of the global spatial averages (“texture similarity”) with correlations of the whole feature maps (“structure similarity”). Experiments show that the proposed DISTS model explains human perceptual scores, both on conventional image quality databases, as well as on texture databases. DISTS also offers competitive performance on related tasks such as texture classification and retrieval. Moreover, DISTS is relatively insensitive to geometric transformations (e.g., translation and dilation), without use of any specialized training or data augmentation.

We then perform a large-scale comparison of IQA models in terms of their use as objectives for the optimization of image processing algorithms. Specifically, we use eleven full-reference IQA models to train deep neural networks (DNNs) for four low-level vision tasks: denoising, deblurring, super-resolution, and compression. Subjective testing on the optimized images allows us to rank the competing models in terms of their perceptual performance, elucidate their relative advantages and disadvantages in these tasks, and propose a set of desirable properties for incorporation into future IQA models.

Inspired by the optimization results, we further improve the DISTS model to a locally adaptive structure and texture similarity (A-DISTS) index for IQA. DISTS makes global quality measurements on the whole feature maps, ignoring the fact that natural photographic images are locally structured and textured across space and scale. We rely on a single statistical feature, namely the dispersion index, to localize texture regions at different scales. The estimated probability (of one patch being texture) is in turn used to adaptively pool local structure and texture measurements. The resulting A-DISTS is adapted to local image content, and is free of expensive human perceptual scores for supervised training. A-DISTS also shows better performance on quality predictions and perceptual optimization of single image super-resolution.

Finally, we turn image quality assessment (a low-level vision task) to intrinsic image popularity assessment (IIPA, a semantic-level vision task), aiming to explore how the image content affects the popularity on social media. Specifically, we first describe a probabilistic method to generate massive popularity-discriminable image pairs, based on which the first large-scale image database for IIPA is established. We then develop a DNN-based computational model, optimized by the ranking consistency with millions of popularity-discriminable image pairs. Experiments on Instagram and other social platforms demonstrate that the optimized model performs favorably against existing methods, exhibits reasonable generalizability on different databases, and even surpasses human-level performance on Instagram. In addition, we conduct a psychophysical experiment to analyze various aspects of human behavior in IIPA.