Deep Unsupervised Domain Adaptation and its Application to Generative Models

Abstract

Deep neural network demonstrates its impressive success across various machine learning problems. However, its impressive performance is attributed to supervised training on large-scale, well-annotated datasets, and it often implicitly assumes that the test data have the same distribution as the training data. In real world applications, this assumption can be easily violated and the corresponding performance may be seriously degraded for cases where the training and test data sets follow different distributions. This is known as the data bias or domain shift problem, which is one of the key factors that prevent the transfer of research results into real-world applications. To alleviate this problem, we focus on unsupervised domain adaptation, which attempts to leverage the learned knowledge from a labeled source domain to adapt the deep model to a completely unlabeled target domain. We observe that the main previous domain adaptation methods minimize the discrepancy between the source and target domains within a discriminative modeling framework. Consequently, it is difficult to take into consideration the class-level distributions during discrepancy minimization. In this thesis, we propose several unsupervised domain adaptation methods based on the generative models, which can help us better understand the real target data distribution or class-conditional distribution. Besides, by simulating the target distribution, we can generate training samples in the target domain directly, which is more interpretable. The model augmented with the generated data is expected to generalize to the target domain. Specifically, we introduce the following works that solve a number of specific problems within the domain adaptation applications:

1. Several existing domain adaptation approaches generalize models trained on the labeled source domain data to the unlabeled target domain data by forcing feature distributions of the two domains to be closer. However, these approaches are likely to ignore the semantic information during the feature alignment process between source and target domain. To address this problem, we propose to learn the semantic cross-domain features by achieving cross-domain generation. Specifically, we integrate a domain adversarial learning process and a within-domain reconstruction process to disentangle the semantic information from the domain information. Then, we apply a cross-domain generation process to further refine the task classifier. Experimental results show that our model achieves superior performance when compared with competing approaches.

2. Image-to-image translation provides another alternative for domain adaptation, since it allows the generation of target data conditioned on the source domain. However, these methods often rely on the cycle-consistency constraint, which is not efficient especially in semantic segmentation tasks which require large computation memory. Therefore, we propose a Simplified Unsupervised Image Translation (SUIT) model for domain adaptation on semantic segmentation. We adopt adversarial training for image generation, and design a novel semantic-content loss function to enhance visual appearance preservation. Specifically, the semantic-content loss contains two components, which focus on preserving label- and content-consistency, respectively. Both of them can be derived from existing modules of SUIT, which makes it simple yet effective for domain adaptation on semantic segmentation tasks. Meanwhile, since the transformation network (generator) is decoupled from the segmentation network, the former can be easily transplanted to other semantic segmentation models. Extensive experimental results demonstrate that these translated images within SUIT can significantly improve performance of the segmentation model on the target domain.

3. For adaptation tasks with large domain shift, we observe that it is difficult for image-to-image translation to generate reliable target training instances. To address this issue, we propose to generate target data conditioned on the shared label space, which is referred to as Label2Image-DA. Specifically, we integrate generative adversarial networks (GAN) into the model predictor, where the generator fed with labels aims to produce corresponding target domain images with a well-designed semantic loss. Compared to previous methods which focus on discrepancy reduction across domains, {i.e.}, image-to-image translation, our model focuses on target image generation which is less affected by the large domain gap. We demonstrate that Label2Image-DA can achieve class-level transfer by showing the plausible generated target image-label pairs. Experimental results show that Label2Image-DA is effective on both image classification and semantic segmentation tasks.

4. Most unsupervised domain adaptation methods require labeled source data during adaptation training. However, training with labeled source data is not always feasible or convenient in some real-world situations due to data privacy issues. For this purpose, we investigate a new and challenging unsupervised domain adaptation setting --- unsupervised model adaptation. We aim to explore how to rely only on unlabeled target data to improve performance of an existing pre-trained model on the target domain. We propose a new framework, which is referred to as Collaborative Class Conditional Generative Adversarial Net (3C-GAN) to bypass the dependence on the source data. Specifically, the prediction model is to be improved through generated target-style data, which provides more accurate guidance for the generator. As a result, the generator and the prediction model can collaborate with each other without source data. Furthermore, we include two additional regularization terms to stabilize training and improve the model's generalization. Compared to conventional domain adaptation methods, our model achieves superior performance on multiple adaptation tasks with only unlabeled target data, which verifies its effectiveness in this challenging setting.

Date of Award	20 Aug 2020
Original language	English
Awarding Institution	City University of Hong Kong
Supervisor	Hau San WONG (Supervisor)