Coupled learning for image generation and latent representation inference using mmd

For modeling the data distribution or the latent representation distribution in the image domain, deep learning methods such as the variational autoencoder (VAE) and the generative adversarial network (GAN) have been proposed. However, despite its capability of modeling these two distributions, VAE tends to learn less meaningful latent representations; GAN can only model the data distribution using the challenging and unstable adversarial training. To address these issues, we propose an unsupervised learning framework to perform coupled learning of these two distributions based on kernel maximum mean discrepancy (MMD). Specifically, the proposed framework consists of (1) an inference network and a generation network for mapping between the data space and the latent space, and (2) a latent tester and a data tester for performing two-sample tests in these two spaces, respectively. On one hand, we perform a two-sample test between stochastic representations from the prior distribution and inferred representations from the inference network. On the other hand, we perform a two-sample test between the real data and generated data. In addition, we impose structural regularization that the two networks are inverses of each other, so that the learning of these two distributions can be coupled. Experimental results on benchmark image datasets demonstrate that the proposed framework is competitive on image generation and latent representation inference of images compared with representative approaches.