Super-resolution for Very Low Quality Facial Images via Consistent Neighbor Relationship
基於近鄰一致關係的極低質量人臉圖像超分辨率算法研究
Student thesis: Doctoral Thesis
Author(s)
Related Research Unit(s)
Detail(s)
| Awarding Institution | |
|---|---|
| Supervisors/Advisors |
|
| Award date | 16 Mar 2018 |
Link(s)
| Permanent Link | https://scholars.cityu.edu.hk/en/theses/theses(64186a75-87d2-4b2c-8fec-5efb1ed566cc).html |
|---|---|
| Other link(s) | Links |
Abstract
With the widely application of the Closed Circuit Television (CCTV), the surveillance video plays more and more important role in the process of criminal investigation and forensic evidence. As one of the most important object, face images in surveillance video captured the most attention of the policemen. But the faces captured from surveillance video are usually in very low quality, which result from video compression, large distance between the camera and the object, and the environment with low illumination, .etc. In order to improve the effective pixels of low quality face images, face super resolution technique is proposed. FSR technique is to generate high resolution face images from given low resolution one(s) with the assistance of high resolution and low resolution external sample pairs in terms of prior knowledge contained among high/low resolution face images.
In last decade, consistent manifold learning based FSR approaches have captured enormous attention of researchers. These approaches have an important consistency premise: the local geometric structures of manifold spaces formed by high resolution and low resolution samples are similar. The local manifold structure is formed by neighbor relationship of samples. But they still face three challenges when the degradation process of facial images have complex resources: (1) the distance measurement and neighbor selection by high frequency visual feature has few robustness in representation; (2) the individual local patch representation has few continuity which lead to over-locality; (3) inaccurate neighbor relationship caused by neighbor mapping from low resolution to high resolution. To solve these problems, we do the research and present several solutions from the perspective of robust facial representation, facial contextual prior knowledge, and neighbor relationship learning from training set:
1. Face super resolution approach based on robust facial contour constraint
To improve the inconsistency of high-/low- neighbor relationship caused by the weak noise robustness of high-frequency visual feature measurement, we introduce the robust mid-frequency visual feature to FSR approach. The high-frequency feature is the most likely to be affected by degradations, in contrast, the mid-frequency component is less affected. Therefore, through using the cascade of mid-frequency facial contour and pixel intensity as the new robust feature, the robustness to low quality (such as noise) is effectively improved, thus the problem of robust feature representation is solved accordingly. In the experiments of CAS-PEAL-R1 dataset, our method outperformed the Chang 1.6dB and 0.14 in terms of PSNR and SSIM respectively, and the better visual performance is obtained in our approach in the reconstruction of facial images.
2. Face super resolution based on contextual relation for VLQ Scenarios
To avoid the low distinctiveness of local patch based FSR that confuses the local manifold relationship, in this work, we propose to use contextual relationship of local patch to supply the texture and structure information. In this way, the distinctiveness between the local patches is effectively improved and the confusing local manifold relationship is corrected accordingly. In the experiments of CAS-PEAL-R1 dataset, our method outperformed the Chang 2.7dB and 0.17 in terms of PSNR and SSIM respectively, and the better visual performance is obtained in our approach in the reconstruction of facial images.
3. Face super resolution method via reverse neighbor mapping and High-order neighbor structure learning
We define the mapping from low to high resolution as the forward direction, in contrast, the mapping process from high to low resolution as the reverse direction. Because of the inherent dimension difference exiting in high and low resolution space, the forward direction mapping in common FSR methods are most inaccurate. To solve the inaccuracy problem of the forward mapping, we proposed to use the accurate relationship existing in high resolution space to constrain inaccurate relationship in low resolution space, i.e., the mapping with reverse direction. In this way, the accurate neighbor mapping effectively ensures the accurate neighbor mapping and thus satisfy the consistency assumption. In the experiments of CAS-PEAL-R1 dataset, our method outperformed the Chang 2.4dB and 0.15 in terms of PSNR and SSIM respectively, and the better visual performance is obtained in our approach in the reconstruction of facial images.
To sum up, this thesis learns from human visual perception to achieve resenting robust facial feature for representation, mining the effective structure of facial image, and learning the neighbor relationship of facial example set. The study results provide new ways for super resolution of low quality facial images problem in terms of both basic theory and key techniques.
In last decade, consistent manifold learning based FSR approaches have captured enormous attention of researchers. These approaches have an important consistency premise: the local geometric structures of manifold spaces formed by high resolution and low resolution samples are similar. The local manifold structure is formed by neighbor relationship of samples. But they still face three challenges when the degradation process of facial images have complex resources: (1) the distance measurement and neighbor selection by high frequency visual feature has few robustness in representation; (2) the individual local patch representation has few continuity which lead to over-locality; (3) inaccurate neighbor relationship caused by neighbor mapping from low resolution to high resolution. To solve these problems, we do the research and present several solutions from the perspective of robust facial representation, facial contextual prior knowledge, and neighbor relationship learning from training set:
1. Face super resolution approach based on robust facial contour constraint
To improve the inconsistency of high-/low- neighbor relationship caused by the weak noise robustness of high-frequency visual feature measurement, we introduce the robust mid-frequency visual feature to FSR approach. The high-frequency feature is the most likely to be affected by degradations, in contrast, the mid-frequency component is less affected. Therefore, through using the cascade of mid-frequency facial contour and pixel intensity as the new robust feature, the robustness to low quality (such as noise) is effectively improved, thus the problem of robust feature representation is solved accordingly. In the experiments of CAS-PEAL-R1 dataset, our method outperformed the Chang 1.6dB and 0.14 in terms of PSNR and SSIM respectively, and the better visual performance is obtained in our approach in the reconstruction of facial images.
2. Face super resolution based on contextual relation for VLQ Scenarios
To avoid the low distinctiveness of local patch based FSR that confuses the local manifold relationship, in this work, we propose to use contextual relationship of local patch to supply the texture and structure information. In this way, the distinctiveness between the local patches is effectively improved and the confusing local manifold relationship is corrected accordingly. In the experiments of CAS-PEAL-R1 dataset, our method outperformed the Chang 2.7dB and 0.17 in terms of PSNR and SSIM respectively, and the better visual performance is obtained in our approach in the reconstruction of facial images.
3. Face super resolution method via reverse neighbor mapping and High-order neighbor structure learning
We define the mapping from low to high resolution as the forward direction, in contrast, the mapping process from high to low resolution as the reverse direction. Because of the inherent dimension difference exiting in high and low resolution space, the forward direction mapping in common FSR methods are most inaccurate. To solve the inaccuracy problem of the forward mapping, we proposed to use the accurate relationship existing in high resolution space to constrain inaccurate relationship in low resolution space, i.e., the mapping with reverse direction. In this way, the accurate neighbor mapping effectively ensures the accurate neighbor mapping and thus satisfy the consistency assumption. In the experiments of CAS-PEAL-R1 dataset, our method outperformed the Chang 2.4dB and 0.15 in terms of PSNR and SSIM respectively, and the better visual performance is obtained in our approach in the reconstruction of facial images.
To sum up, this thesis learns from human visual perception to achieve resenting robust facial feature for representation, mining the effective structure of facial image, and learning the neighbor relationship of facial example set. The study results provide new ways for super resolution of low quality facial images problem in terms of both basic theory and key techniques.
