Dual student : Breaking the Limits of the Teacher in Semi-supervised Learning

Research output: Chapters, Conference Papers, Creative and Literary WorksRGC 32 - Refereed conference paper (with host publication)peer-review

147 Scopus Citations
View graph of relations

Author(s)

Related Research Unit(s)

Detail(s)

Original languageEnglish
Title of host publicationProceedings - 2019 International Conference on Computer Vision (ICCV 2019)
PublisherIEEE
Pages6727-6735
ISBN (electronic)978-1-7281-4803-8
Publication statusPublished - Oct 2019

Publication series

NameProceedings of the IEEE International Conference on Computer Vision
Volume2019-October
ISSN (Print)1550-5499

Conference

Title17th IEEE/CVF International Conference on Computer Vision (ICCV 2019)
LocationCOEX Convention Center
PlaceKorea, Republic of
CitySeoul
Period27 October - 2 November 2019

Abstract

Recently, consistency-based methods have achieved state-of-the-art results in semi-supervised learning (SSL). These methods always involve two roles, an explicit or implicit teacher model and a student model, and penalize predictions under different perturbations by a consistency constraint. However, the weights of these two roles are tightly coupled since the teacher is essentially an exponential moving average (EMA) of the student. In this work, we show that the coupled EMA teacher causes a performance bottleneck. To address this problem, we introduce Dual Student, which replaces the teacher with another student. We also define a novel concept, stable sample, following which a stabilization constraint is designed for our structure to be trainable. Further, we discuss two variants of our method, which produce even higher performance. Extensive experiments show that our method improves the classification performance significantly on several main SSL benchmarks. Specifically, it reduces the error rate of the 13-layer CNN from 16.84% to 12.39% on CIFAR-10 with 1k labels and from 34.10% to 31.56% on CIFAR-100 with 10k labels. In addition, our method also achieves a clear improvement in domain adaptation.

Bibliographic Note

Full text of this publication does not contain sufficient affiliation information. With consent from the author(s) concerned, the Research Unit(s) information for this record is based on the existing academic department affiliation of the author(s).

Citation Format(s)

Dual student: Breaking the Limits of the Teacher in Semi-supervised Learning. / Ke, Zhanghan; Wang, Daoye; Yan, Qiong et al.
Proceedings - 2019 International Conference on Computer Vision (ICCV 2019). IEEE, 2019. p. 6727-6735 9009457 (Proceedings of the IEEE International Conference on Computer Vision; Vol. 2019-October).

Research output: Chapters, Conference Papers, Creative and Literary WorksRGC 32 - Refereed conference paper (with host publication)peer-review