Motion-Aware Temporal Coherence for Video Resizing

Description

Appropriate resizing of images or videos is of high demand nowadays due to the ubiquitous display devices with significantly different aspect ratios and resolutions, e.g., wide-screen TVs and smartphones. Content-aware image resizing, which adapts images to target display devices while preserving important content, has enjoyed a certain degree of success and some techniques have even been incorporated into professional image editing packages like Photoshop. Following the popularity of image resizing, content-aware video resizing has gained significant attention in both research and industry. However, the existing video resizing systems have been demonstrated only on a small set of video examples and are still far from mature.This project aims for a practical content-aware video resizing system. We identify the main problem with the existing systems as the lack of appropriate ways to preserve temporal coherence during resizing. To date, the problem of temporal incoherence has been addressed by constraining temporally adjacent pixels to be transformed coherently. However, due to the motion-oblivious nature of this simple constraint, the resized videos often exhibit flickering and waving artifacts, especially when significant camera or object motions are involved. Since the feature correspondence across frames changes spatially with both camera and object motion, motion-aware treatment of features is required for video resizing. This motivates us to design a motion-aware video resizing framework.To design effective motion-aware constraints for temporal coherence, which have not been studied before, we need to address several challenges such as what type of motion content is needed, how to extract the related motion information robustly, how to effectively integrate the extracted motion content into the resizing process, and how to address the scalability problem for long videos. This project plans to study various issues related to the idea of motion-aware constraints and to build a complete deformationbased video resizing system which will handle challenging inputs that are difficult to handle with previous methods.We are confident that we can complete the project and deliver the expected outcomes. We have had extensive research experiences in the field of interactive shape deformation, which we believe have paved the way for the proposed ideas. We expect that the new project will follow the success of our completed projects and make continuous contributions to both research and industry.