Deep Exemplar-based Colorization

Research output: Journal Publications and Reviews (RGC: 21, 22, 62)21_Publication in refereed journal

27 Scopus Citations
View graph of relations

Author(s)

  • Mingming HE
  • Dongdong CHEN
  • Jing LIAO
  • Pedro V. SANDER
  • Lu YUAN

Detail(s)

Original languageEnglish
Article number47
Journal / PublicationACM Transactions on Graphics
Volume37
Issue number4
Publication statusPublished - Aug 2018
Externally publishedYes

Abstract

We propose the first deep learning approach for exemplar-based local colorization. Given a reference color image, our convolutional neural network directly maps a grayscale image to an output colorized image. Rather than using hand-crafted rules as in traditional exemplar-based methods, our end-to-end colorization network learns how to select, propagate, and predict colors from the large-scale data. The approach performs robustly and generalizes well even when using reference images that are unrelated to the input grayscale image. More importantly, as opposed to other learning-based colorization methods, our network allows the user to achieve customizable results by simply feeding different references. In order to further reduce manual effort in selecting the references, the system automatically recommends references with our proposed image retrieval algorithm, which considers both semantic and luminance information. The colorization can be performed fully automatically by simply picking the top reference suggestion. Our approach is validated through a user study and favorable quantitative comparisons to the-state-of-the-art methods. Furthermore, our approach can be naturally extended to video colorization. Our code and models are freely available for public use.

Research Area(s)

  • Colorization, Exemplar-based colorization, Deep learning, Vision for graphics

Citation Format(s)

Deep Exemplar-based Colorization. / HE, Mingming; CHEN, Dongdong; LIAO, Jing; SANDER, Pedro V.; YUAN, Lu.

In: ACM Transactions on Graphics, Vol. 37, No. 4, 47, 08.2018.

Research output: Journal Publications and Reviews (RGC: 21, 22, 62)21_Publication in refereed journal