Content-aware image editing based on interpolation methods

Abstract

Image editing, as a fundamental tool for image processing, has been widely applied in fields such as computer animation, film making and photography. Among a variety of image editing patterns, color transfer, image retargeting and image composition are three hot spots attracting much attention in recent years. However, all of them have to face the problem of how to trade off between preserving image contents and satisfying new requirements so as to generate new images of high quality. Essentially, all these topics can be reduced to an optimization problem under some specified constraints. We study them from the viewpoint of data interpolation and propagation in a content-aware manner. It results in four novel content-aware and interpolation based methods for addressing these problems. Specifically, our main contributions can be summarized as follows: (1) We investigate the efficiency issue of color transfer based on edit propagation. Instead of solving an optimization equation, we renew the color attributes of each pixel in the original image according to those in the specified strokes using approximation mean value coordinates, which is a closed-form solution and can achieve instant feedback. Furthermore, we introduce an optimization equation to ensure that the specified strokes in the original image can preserve their shape well even if the structured information of the original strokes and the corresponding reference strokes is unmatched. The solution is still efficient due to the fact that the number of unknown variables (the pixels in the strokes) is small. (2) We greatly simplify the interactive process for color transfer with a new interactive scheme in which only changed regions require interactively marking strokes and the unchanged regions are calculated automatically. It is very convenient especially when the color values are plentiful in an image and most of them will be preserved after transferred. Our policy is that similar colors to those in the strokes follow the similar transformation and dissimilar colors conduct an identity transformation. The strategy is modeled using moving least squares technique, which also yields a closed-form solution. In addition, we define a new metric to evaluate the similarity of two pixels, which not only has interpolatory property but also can adjust the similarity in the color space and the spatial space separately. (3) Our image retargeting approach addresses the resolution problem of the salient objects. Compared to previous methods, it is able to provide more space for significant objects. It can enlarge these objects even when the image remains unchanged or is just resized with the same ratio in both vertical and horizontal directions where many shape distortion based methods fail. The strategy is to define an additional quadric energy term in the optimization equation based on the assumption that the scaling factor of a local region is linearly proportional to its corresponding saliency. Furthermore, a triangle similarity quadric energy term is introduced to prevent important regions from distortion. The method can achieve multilevel scaling for different important objects by assigning saliency values to them independently. (4) We present a new image composition technique to deal with the inhomogenous problem that two boundaries of the composite region in the original image patch and the target image seriously mismatch in appearance. Previous techniques require the initial pixels to be a closed boundary and usually fail to work for this case. Our solution is to select a part of matched pixels from two boundaries and propagate their corresponding differences to the inner of the composite region using approximation mean value coordinates. Finally, we not only theoretically show the feasibility of our new methods but also demonstrate their efficiency and effectiveness using extensive experiments and comparisons.

Date of Award	16 Jul 2012
Original language	English
Awarding Institution	City University of Hong Kong
Supervisor	Wei Jia JIA (Supervisor)