Stereo Image and Video Editing

Abstract

With the success of the 3D movie “Avatar”, stereo videos have become very popular in recent years. In general, each stereo image contains two regular 2D images captured from the same scene at the same time but from slightly different viewing locations. When a stereo image/video is displayed on the screen, with appropriate devices, viewers see one 2D regular image/frame with the left eye and the other with the right eye. The human brain will then fuse the two images/frames together to produce 3D scene depth information. As stereo images can convey more visual information, stereo media are considered as one of the main research directions of future development.

Although there are a lot of tools available for editing traditional 2D imags/videos, tools for editing 3D media are very limited. In general, editing and processing stereo images/videos are more difficult than those of 2D images/videos, due to three major reasons. First, it is difficult to obtain noise-free and accurate disparity/depth maps for stereo images/videos. Stereo matching methods, which aim at finding correspondences between pixels in the left and right images, generally do not perform very well, especially for stereo images of natural scenes. Even we use a depth camera, to obtain high resolution and noise-free depth maps from the low resolution and noisy output is still difficult. Second, it is difficult to ensure the spacial coherence of left and right images of stereo image pair, which is very important for minimizing distortion and producing high quality results. In practice, the left and right images usually need to be simultaneously processed in order to enforce the coherence of left and right images, such as processing by a global optimization. Thus, algorithms for processing stereo media are usually more complex than those for 2D media, with high computational and memory costs. Third, we need to ensure both motion and depth coherences across neighboring frames. In this thesis, our aim is to develop a 3D image/movie editor, which attempts to address these three technical difficulties and provides the users with a number of editing functions for processing stereo images/videos. We mainly introduce three editing functions as follows.

First, we propose a novel depth mapping method for stereo video depth mapping. Most stereo videos are developed primarily for viewing on large screens located at some distance away from the viewer. If we watch these videos on a small screen located near to us, the depth range of the videos will be seriously reduced, which can significantly degrade their 3D effects. In order to address this problem, we propose a linear depth mapping method to adjust the depth range of a stereo video according to the viewing configuration, including pixel density and distance to the screen. We also consider characters of human binocular vision, such as relative depth among objects to depth perception, human eyes sensitivity to straight lines and planes. Our method tries to minimize the distortion of stereo image contents, by preserving the relationship of neighboring features and preventing line and plane bending. It also considers motion and depth coherences across neighboring frames. While depth coherence ensures smooth changes of the depth field across frames, motion coherence ensures smooth content changes across frames. Our experimental results show that the proposed method can improve the stereoscopic effects while maintaining the quality of the output videos.

Second, in order to obtain high quality depth mapping and other stereo editing effect, we extend the shift-map method for stereo image editing. Our method simultaneously processes the left and right images on pixel level using a global optimization algorithm. It enforces photo consistence between the two images and preserves 3D scene structures. It also addresses the occlusion and disocclusion problems, which may enable many stereo image editing functions, such as depth mapping, object depth adjustment and non-homogeneous image resizing. Our experimental results show that the proposed method produces high quality results with a number of editing functions.

Third, we propose a method for creating infinite stereo panoramas. A stereo infinite panorama is a panoramic image that can be infinitely extended by continuously stitching together stereo images that depict similar scenes, but may be taken from different geographic locations. It can be used to create interesting walkthrough environment. An important issue underlying this application is to seamlessly stitch two stereo images together. Although many methods have been proposed for stitching 2D images, they may not work well on stereo images, due to the difficulty in ensuring disparity consistency. In this thesis, we propose a novel method to stitch two stereo images seamlessly. We first apply the graph cut algorithm to compute a seam for stitching, with a novel disparity-aware energy function to both ensure disparity continuity and suppress visual artifacts around the seam. We then apply a modified warping-based disparity scaling algorithm to suppress the seam in the depth domain. Our experimental results show that the proposed stitching method is capable of producing high quality stereo infinite panoramas.

Date of Award	5 Dec 2013
Original language	English
Awarding Institution	City University of Hong Kong
Supervisor	Rynson W H LAU (Supervisor)