Perceptual Assessment and Optimization of High Dynamic Range Image Processing

Abstract

High dynamic range (HDR) images aim to faithfully reproduce the large luminance variations of natural scenes that low dynamic range (LDR) images are not capable of. In recent years, a multitude of HDR imaging technologies have been developed and brought to market in response to the growing demand for HDR images in various fields, including photography, gaming, cinema, and virtual reality. As a result, HDR image quality assessment (IQA) has emerged as a critical tool for monitoring, ensuring, and optimizing the perceptual quality of HDR images throughout the imaging, compression, communication, and rendering procedures.

Subjective quality assessment is so far the most straightforward and reliable way for evaluating HDR images because the ultimate receiver in most such applications is the human eye, while conventional subjective assessment suffers from sampling bias, algorithmic bias, and subjective bias. Also, there is a noticeable scarcity of objective HDR IQA methods, mainly stemming from the prevalent belief that most LDR quality models can be straightforwardly applied to HDR images. Consequently, existing deep neural networks (DNNs) based HDR image applications, such as single image high dynamic range (SI-HDR) reconstruction techniques, HDR novel view synthesis, and HDR image compression, tend to rely on makeshift combinations of quality metrics and image regularizers.

In this thesis, we first focus on developing perceptual quality assessment for HDR images, including both subjective and objective assessment. For subjective assessment, we strive to address biases inherent in conventional subjective assessments by implementing an automated process for sampling adaptive and diverse images for testing. To accomplish this, we formulate the sample selection task as a joint maximization problem, aiming to optimize both the discrepancy between enhancers and the diversity among the selected input images. This approach enables a more robust and unbiased evaluation of different reconstruction methods, ensuring a comprehensive assessment that accounts for both performance differences and content diversity. This approach is employed in the subjective user study in the following study of this thesis. For objective assessment, we develop a family of full-reference HDR quality metrics, which leverage the recent advancements in LDR IQA, relying on a simple inverse display model. The model decomposes an HDR image in (uncalibrated) luminance values to an LDR image stack comprising varying exposures in digital pixel values. The LDR image stack is subsequently evaluated using existing LDR quality models. The local quality scores of each exposure are then aggregated with the help of a simple well-exposedness measure into a global quality score, which will be further weighted across exposures to obtain the overall quality score. When assessing LDR images, the proposed HDR quality models reduce gracefully to the original LDR ones with the same performance. Experimental results from four human-rated HDR IQA datasets confirm the superior performance of the proposed metrics in assessing HDR image quality, compared to existing models.

Based on the proposed HDR quality metric, we then investigate two HDR rendering applications - SI-HDR reconstruction and HDR novel view synthesis and present a new HDR compression method. For SI-HDR reconstruction, we introduce a two-stage method that first jointly performs LDR image enhancement tasks relevant to SI-HDR reconstruction and then focuses primarily on (nonlinear) dynamic range expansion from the enhanced LDR image. Such stage design enables direct transfer of the recent advances in LDR image enhancement to SI-HDR reconstruction. Both stages are supervised by perceptual quality metrics - adaptive deep image structure and texture similarity (A-DISTS) index and the proposed HDR metric, respectively. In HDR novel view synthesis, we employ an eight-layer multi-layer perceptron (MLP) to model the radiance field of an HDR scene, guided by the proposed HDR quality metric. For HDR compression, we propose a system that compresses HDR images into two bitstreams for storage. One bitstream generates LDR images for display purposes based on the scene's maximum luminance, while the other serves as side information to aid HDR image reconstruction from the generated LDR image. To measure the perceptual quality of the displayable LDR images and the reconstructed HDR images, we employ the normalized Laplacian pyramid distance (NLPD) and the proposed HDR metric, respectively.

Finally, we propose a two-stage, self-calibrated, and perceptually optimized Tone Mapping Operator (TMO). In the first stage, we decompose the HDR image into a normalized Laplacian pyramid, serving as a multi-scale representation. Two lightweight DNNs are then employed to process the normalized representation and estimate the Laplacian pyramid of the corresponding LDR image. We optimize the tone mapping network by minimizing NLPD. In the second stage, we first feed the same HDR image but rescaled with different maximum luminances to the learned tone mapping network, generating a pseudo-multi-exposure image stack with different detail visibility and color saturation. Subsequently, another lightweight DNN is trained to fuse this LDR image stack into the desired output LDR image by maximizing a variant of the structural similarity index for multi-exposure image fusion (MEF-SSIM).

Overall, we first introduce perceptual quality assessment of HDR images, including a debiased subjective assessment method using the MAD competition and an objective assessment method based on a simple inverse display model. Then, we redesign a two-stage network for SI-HDR reconstruction and employ an MLP to effectively model the HDR scene’s radiance field. Next, we introduce an end-to-end framework to compress HDR images for perceptually optimal storage and display. At last, we present a computational method for tone mapping HDR images, which utilizes lightweight tone mapping and fusion networks. These contributions significantly advance the field of HDR image processing, providing novel methods and frameworks for evaluating and enhancing HDR image quality.

Date of Award	26 Aug 2024
Original language	English
Awarding Institution	City University of Hong Kong
Supervisor	Kede MA (Supervisor)