Perceptual Optimization of End-to-end Video Codec

In the era of artificial intelligence, the proliferation of 4K and 8K ultra-high-definition videos has led to an exponential increase in video data size, characterized by heightened complexity and variety. Traditional video codecs, which rely on handcrafted algorithms and manual parameter adjustments, are proving inadequate for the efficient transmission and storage of such high-fidelity content. This has necessitated a fundamental shift in video encoding paradigms towards Al-driven end-to-end video codecs that dynamicallyadapt to content, significantly enhancing compression efficiency and video quality. Perceptual optimization is pivotal in this context as it prioritizes the viewer's experience. Despite its advantages, including enhanced subjective quality and optimized bandwidth usage, perceptual optimization remains underexplored in end-to-end video codecs. This project endeavors to explore and advance perceptual optimization techniques to improve the performance of the codec that is constructed by neural networks. The objective is to develop an end-to-end video codec system that maximizes perceptual quality under the constraint of bandwidth, focusing on areas most impactful to the viewer's experience. To achieve this, the project will first develop a spatio~temporal adaptive coding frameworkthat optimizes the perceptual quality across various content types and improves the coding efficiency. Subsequently, it will investigate the rate-d istortion (R-O) relationship from a perceptual standpoint, aiming to establish a perceptual quality control mechanism that adjusts encoding parameters for an optimal viewer experience. Finally, the project will integrate these advancements as a first perceptual optimized end-to-end coding platform, setting a new benchmark for end-to-end video coding efficiency towardsperceptual quality. Furthermore, this project will advance the progress of ultra-highdefinition video and innovative industrialization of Hong Kong.