SpecFormer: Guarding Vision Transformer Robustness via Maximum Singular Value Penalization

Xixu Hu; Runkai Zheng; Jindong Wang; Cheuk Hang Leung; Qi Wu; Xing Xie

doi:10.1007/978-3-031-72949-2_20

SpecFormer: Guarding Vision Transformer Robustness via Maximum Singular Value Penalization

Xixu Hu
, Runkai Zheng
, Jindong Wang^*
, Cheuk Hang Leung
, Qi Wu^*
, Xing Xie

^*Corresponding author for this work

Department of Data Science

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

2 Citations (Scopus)

Abstract

Vision Transformers (ViTs) are increasingly used in computer vision due to their high performance, but their vulnerability to adversarial attacks is a concern. Existing methods lack a solid theoretical basis, focusing mainly on empirical training adjustments. This study introduces SpecFormer, tailored to fortify ViTs against adversarial attacks, with theoretical underpinnings. We establish local Lipschitz bounds for the self-attention layer and propose the Maximum Singular Value Penalization (MSVP) to precisely manage these bounds By incorporating MSVP into ViTs’ attention layers, we enhance the model’s robustness without compromising training efficiency. SpecFormer, the resulting model, outperforms other state-of-the-art models in defending against adversarial attacks, as proven by experiments on CIFAR and ImageNet datasets. Code is released at https://github.com/microsoft/robustlearn. © The Author(s),

Original language	English
Title of host publication	Computer Vision
Subtitle of host publication	ECCV 2024
Editors	Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, Gül Varol
Publisher	Springer, Cham
Pages	345–362
Number of pages	18
ISBN (Electronic)	978-3-031-72949-2
ISBN (Print)	978-3-031-72948-5
DOIs	https://doi.org/10.1007/978-3-031-72949-2_20
Publication status	Published - 31 Oct 2024
Event	18th European Conference on Computer Vision (ECCV 2024) - MiCo Milano, Milan, Italy Duration: 29 Sept 2024 → 4 Oct 2024 https://eccv.ecva.net/

Publication series

Name	Lecture Notes in Computer Science
Publisher	Springer
Volume	15112
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	18th European Conference on Computer Vision (ECCV 2024)
Abbreviated title	ECCV2024
Place	Italy
City	Milan
Period	29/09/24 → 4/10/24
Internet address	https://eccv.ecva.net/

Bibliographical note

Full text of this publication does not contain sufficient affiliation information. With consent from the author(s) concerned, the Research Unit(s) information for this record is based on the existing academic department affiliation of the author(s)

Funding

Qi WU acknowledges the support from The CityU-JD Digits Joint Laboratory in Financial Technology and Engineering and The Hong Kong Research Grants Council [General Research Fund 11219420/9043008 ]. The work described in this paper was partially supported by the InnoHK initiative, the Government of the HKSAR, and the Laboratory for AI-Powered Financial Technologies.

Research Keywords

Vision Transformer
Adversarial Robustness
Lipschitz Continuity

RGC Funding Information

RGC-funded

Access Output

10.1007/978-3-031-72949-2_20

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

Hu, X., Zheng, R., Wang, J., Leung, C. H., Wu, Q., & Xie, X. (2024). SpecFormer: Guarding Vision Transformer Robustness via Maximum Singular Value Penalization. In A. Leonardis, E. Ricci, S. Roth, O. Russakovsky, T. Sattler, & G. Varol (Eds.), Computer Vision: ECCV 2024 (pp. 345–362). (Lecture Notes in Computer Science; Vol. 15112). Springer, Cham. https://doi.org/10.1007/978-3-031-72949-2_20

@inproceedings{abc8e22e884d4a16823844a2303a085e,

title = "SpecFormer: Guarding Vision Transformer Robustness via Maximum Singular Value Penalization",

abstract = "Vision Transformers (ViTs) are increasingly used in computer vision due to their high performance, but their vulnerability to adversarial attacks is a concern. Existing methods lack a solid theoretical basis, focusing mainly on empirical training adjustments. This study introduces SpecFormer, tailored to fortify ViTs against adversarial attacks, with theoretical underpinnings. We establish local Lipschitz bounds for the self-attention layer and propose the Maximum Singular Value Penalization (MSVP) to precisely manage these bounds By incorporating MSVP into ViTs{\textquoteright} attention layers, we enhance the model{\textquoteright}s robustness without compromising training efficiency. SpecFormer, the resulting model, outperforms other state-of-the-art models in defending against adversarial attacks, as proven by experiments on CIFAR and ImageNet datasets. Code is released at https://github.com/microsoft/robustlearn. {\textcopyright} The Author(s),",

keywords = "Vision Transformer, Adversarial Robustness, Lipschitz Continuity",

author = "Xixu Hu and Runkai Zheng and Jindong Wang and Leung, \{Cheuk Hang\} and Qi Wu and Xing Xie",

note = "Full text of this publication does not contain sufficient affiliation information. With consent from the author(s) concerned, the Research Unit(s) information for this record is based on the existing academic department affiliation of the author(s); 18th European Conference on Computer Vision (ECCV 2024), ECCV2024 ; Conference date: 29-09-2024 Through 04-10-2024",

year = "2024",

month = oct,

day = "31",

doi = "10.1007/978-3-031-72949-2\_20",

language = "English",

isbn = "978-3-031-72948-5",

series = "Lecture Notes in Computer Science",

publisher = "Springer, Cham",

pages = "345–362",

editor = "Ale{\v s} Leonardis and Elisa Ricci and Stefan Roth and Russakovsky, \{Olga \} and Torsten Sattler and G{\"u}l Varol",

booktitle = "Computer Vision",

url = "https://eccv.ecva.net/",

}

Hu, X, Zheng, R, Wang, J, Leung, CH, Wu, Q & Xie, X 2024, SpecFormer: Guarding Vision Transformer Robustness via Maximum Singular Value Penalization. in A Leonardis, E Ricci, S Roth, O Russakovsky, T Sattler & G Varol (eds), Computer Vision: ECCV 2024. Lecture Notes in Computer Science, vol. 15112, Springer, Cham, pp. 345–362, 18th European Conference on Computer Vision (ECCV 2024), Milan, Italy, 29/09/24. https://doi.org/10.1007/978-3-031-72949-2_20

SpecFormer: Guarding Vision Transformer Robustness via Maximum Singular Value Penalization. / Hu, Xixu; Zheng, Runkai; Wang, Jindong et al.
Computer Vision: ECCV 2024. ed. / Aleš Leonardis; Elisa Ricci; Stefan Roth; Olga Russakovsky; Torsten Sattler; Gül Varol. Springer, Cham, 2024. p. 345–362 (Lecture Notes in Computer Science; Vol. 15112).

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

TY - GEN

T1 - SpecFormer

T2 - 18th European Conference on Computer Vision (ECCV 2024)

AU - Hu, Xixu

AU - Zheng, Runkai

AU - Wang, Jindong

AU - Leung, Cheuk Hang

AU - Wu, Qi

AU - Xie, Xing

N1 - Full text of this publication does not contain sufficient affiliation information. With consent from the author(s) concerned, the Research Unit(s) information for this record is based on the existing academic department affiliation of the author(s)

PY - 2024/10/31

Y1 - 2024/10/31

N2 - Vision Transformers (ViTs) are increasingly used in computer vision due to their high performance, but their vulnerability to adversarial attacks is a concern. Existing methods lack a solid theoretical basis, focusing mainly on empirical training adjustments. This study introduces SpecFormer, tailored to fortify ViTs against adversarial attacks, with theoretical underpinnings. We establish local Lipschitz bounds for the self-attention layer and propose the Maximum Singular Value Penalization (MSVP) to precisely manage these bounds By incorporating MSVP into ViTs’ attention layers, we enhance the model’s robustness without compromising training efficiency. SpecFormer, the resulting model, outperforms other state-of-the-art models in defending against adversarial attacks, as proven by experiments on CIFAR and ImageNet datasets. Code is released at https://github.com/microsoft/robustlearn. © The Author(s),

AB - Vision Transformers (ViTs) are increasingly used in computer vision due to their high performance, but their vulnerability to adversarial attacks is a concern. Existing methods lack a solid theoretical basis, focusing mainly on empirical training adjustments. This study introduces SpecFormer, tailored to fortify ViTs against adversarial attacks, with theoretical underpinnings. We establish local Lipschitz bounds for the self-attention layer and propose the Maximum Singular Value Penalization (MSVP) to precisely manage these bounds By incorporating MSVP into ViTs’ attention layers, we enhance the model’s robustness without compromising training efficiency. SpecFormer, the resulting model, outperforms other state-of-the-art models in defending against adversarial attacks, as proven by experiments on CIFAR and ImageNet datasets. Code is released at https://github.com/microsoft/robustlearn. © The Author(s),

KW - Vision Transformer

KW - Adversarial Robustness

KW - Lipschitz Continuity

UR - https://arxiv.org/abs/2402.03317

U2 - 10.1007/978-3-031-72949-2_20

DO - 10.1007/978-3-031-72949-2_20

M3 - RGC 32 - Refereed conference paper (with host publication)

SN - 978-3-031-72948-5

T3 - Lecture Notes in Computer Science

SP - 345

EP - 362

BT - Computer Vision

A2 - Leonardis, Aleš

A2 - Ricci, Elisa

A2 - Roth, Stefan

A2 - Russakovsky, Olga

A2 - Sattler, Torsten

A2 - Varol, Gül

PB - Springer, Cham

Y2 - 29 September 2024 through 4 October 2024

ER -

SpecFormer: Guarding Vision Transformer Robustness via Maximum Singular Value Penalization

Abstract

Publication series

Conference

Bibliographical note

Funding

Research Keywords

RGC Funding Information

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

GRF: Generative Models of Multivariate Dependence for Asset Returns

Towards More Adaptive and Reliable Artificial Intelligence: Exploration in Sequential Data, Images and Large Language Models

Cite this

SpecFormer: Guarding Vision Transformer Robustness via Maximum Singular Value Penalization

Abstract

Publication series

Conference

Bibliographical note

Funding

Research Keywords

RGC Funding Information

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

Projects

GRF: Generative Models of Multivariate Dependence for Asset Returns

Student theses

Towards More Adaptive and Reliable Artificial Intelligence: Exploration in Sequential Data, Images and Large Language Models

Cite this