TWINS : A Fine-Tuning Framework for Improved Transferability of Adversarial Robustness and Generalization

Research output: Chapters, Conference Papers, Creative and Literary WorksRGC 32 - Refereed conference paper (with host publication)peer-review

View graph of relations

Author(s)

Related Research Unit(s)

Detail(s)

Original languageEnglish
Title of host publicationProceedings - 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023
PublisherInstitute of Electrical and Electronics Engineers, Inc.
Pages16436-16446
ISBN (print)979-8-3503-0129-8
Publication statusPublished - Jun 2023

Conference

Title2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2023)
LocationVancouver Convention Center
PlaceCanada
CityVancouver
Period18 - 22 June 2023

Abstract

Recent years have seen the ever-increasing importance of pre-trained models and their downstream training in deep learning research and applications. At the same time, the defense for adversarial examples has been mainly investigated in the context of training from random initialization on simple classification tasks. To better exploit the potential of pre-trained models in adversarial robustness, this paper focuses on the fine-tuning of an adversarially pre-trained model in various classification tasks. Existing research has shown that since the robust pre-trained model has already learned a robust feature extractor, the crucial question is how to maintain the robustness in the pre-trained model when learning the downstream task. We study the model-based and data-based approaches for this goal and find that the two common approaches cannot achieve the objective of improving both generalization and adversarial robustness. Thus, we propose a novel statistics-based approach, Two-WIng NormliSation (TWINS) fine-tuning framework, which consists of two neural networks where one of them keeps the population means and variances of pre-training data in the batch normalization layers. Besides the robust information transfer, TWINS increases the effective learning rate without hurting the training stability since the relationship between a weight norm and its gradient norm in standard batch normalization layer is broken, resulting in a faster escape from the sub-optimal initialization and alleviating the robust overfitting. Finally, TWINS is shown to be effective on a wide range of image classification datasets in terms of both generalization and robustness.

©2023 IEEE

Citation Format(s)

TWINS: A Fine-Tuning Framework for Improved Transferability of Adversarial Robustness and Generalization. / Liu, Ziquan; Xu, Yi; Ji, Xiangyang et al.
Proceedings - 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023. Institute of Electrical and Electronics Engineers, Inc., 2023. p. 16436-16446.

Research output: Chapters, Conference Papers, Creative and Literary WorksRGC 32 - Refereed conference paper (with host publication)peer-review