Semantic Example Guided Image-to-Image Translation
Research output: Journal Publications and Reviews (RGC: 21, 22, 62) › 21_Publication in refereed journal › peer-review
Author(s)
Related Research Unit(s)
Detail(s)
Original language | English |
---|---|
Article number | 9115302 |
Pages (from-to) | 1654-1665 |
Journal / Publication | IEEE Transactions on Multimedia |
Volume | 23 |
Online published | 11 Jun 2020 |
Publication status | Published - 2021 |
Link(s)
Abstract
Many image-to-image (I2I) translation problems are in nature of high diversity that a single input may have various counterparts. Prior works proposed the multi-modal network that can build a many-to-many mapping between two visual domains. However, most of them are guided by sampled noises. Some others encode the reference images into a latent vector, by which the semantic information of the reference image will be washed away. In this work, we aim to provide a solution to control the output based on references semantically. Given a reference image and an input in another domain, a semantic matching is first performed between the two visual contents and generates the auxiliary image, which is explicitly encouraged to preserve semantic characteristics of the reference. A deep network then is used for I2I translation and the final outputs are expected to be semantically similar to both the input and the reference; however, no such paired data can satisfy that dual-similarity in a supervised fashion, so we build up a self-supervised framework to serve the training purpose. We improve the quality and diversity of the outputs by employing non-local blocks and a multi-task architecture. We assess the proposed method through extensive qualitative and quantitative evaluations and also presented comparisons with several state-of-art models.
Research Area(s)
- Artificial neural networks, image generation, image representation
Citation Format(s)
Semantic Example Guided Image-to-Image Translation. / Huang, Jialu; Liao, Jing; Kwong, Sam.
In: IEEE Transactions on Multimedia, Vol. 23, 9115302, 2021, p. 1654-1665.Research output: Journal Publications and Reviews (RGC: 21, 22, 62) › 21_Publication in refereed journal › peer-review