Perceptually Weighted Mean Squared Error Based Rate-Distortion Optimization for HEVC

The process of rate-distortion (RD) optimization plays a key role for video coding, which aims to achieve a tradeoff between compression efficiency and video quality distortion. Although the conventional objective distortion metric mean squared error performs well in computational complexity, it is not always in accordance with the perceptual quality perceived by human visual system (HVS). Taking the characteristics of HVS into consideration, a perceptually weighted mean squared error (PWMSE) based RD model is proposed in this work. First, a low-pass filter is employed to process the original distortion information in order to simulate visual signal processing and obtain the perceptual distortion. Then, masking modulation in both temporal and spatial domains is introduced into distortion model, and a novel Lagrange multiplier is derived accordingly. The proposed PWMSE based RD model is applied to the high efficiency video coding standard, and comparative experimental results demonstrate its effectiveness. The project page can be found in https://mic.tongji.edu.cn/ab/40/c9778a174912/page.htm.