The CUR Decomposition of Self-Attention Matrices in Vision Transformers

Chong Wu; Maolin Che; Hong Yan

doi:10.1109/TPAMI.2025.3646452

The CUR Decomposition of Self-Attention Matrices in Vision Transformers

Chong Wu
, Maolin Che^*
, Hong Yan

^*Corresponding author for this work

Department of Electrical Engineering

Research output: Journal Publications and Reviews › RGC 21 - Publication in refereed journal › peer-review

1 Citation (Scopus)

Abstract

Transformers have achieved great success in natural language processing and computer vision. The core and basic technique of transformers is the self-attention mechanism. The vanilla self-attention mechanism has quadratic complexity, which limits its applications to vision tasks. Most of the existing linear self-attention mechanisms will sacrifice performance to some extent to reduce complexity. In this paper, we propose a novel linear approximation of the vanilla self-attention mechanism named CURSA to achieve both high performance and low complexity at the same time. CURSA is based on the CUR decomposition to decompose the multiplication of large matrices into the multiplication of several small matrices to achieve almost linear complexity. Experiment results of CURSA in image classification tasks, semantic segmentation tasks, object detection tasks, and long-range arena show that it outperforms state-of-the-art self-attention mechanisms with better data efficiency, faster speed, and higher accuracy.

© 2025 IEEE

Original language	English
Pages (from-to)	4792-4809
Journal	IEEE Transactions on Pattern Analysis and Machine Intelligence
Volume	48
Issue number	4
Online published	19 Dec 2025
DOIs	https://doi.org/10.1109/TPAMI.2025.3646452
Publication status	Published - Apr 2026

Funding

This work is supported by the Hong Kong Innovation and Technology Commission (InnoHK Project CIMDA), the Institute of Digital Medicine, City University of Hong Kong (Projects 9229503 and 9610460), the National Natural Science Foundation of China (No. 12561095), and the Special Posts of Guizhou University (No. [2025]06).

Research Keywords

Attention mechanism
CUR decomposition
linear approximation
vision transformer

Access Output

10.1109/TPAMI.2025.3646452

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

@article{6172d0046c744639bd25ee4101a440c3,

title = "The CUR Decomposition of Self-Attention Matrices in Vision Transformers",

abstract = "Transformers have achieved great success in natural language processing and computer vision. The core and basic technique of transformers is the self-attention mechanism. The vanilla self-attention mechanism has quadratic complexity, which limits its applications to vision tasks. Most of the existing linear self-attention mechanisms will sacrifice performance to some extent to reduce complexity. In this paper, we propose a novel linear approximation of the vanilla self-attention mechanism named CURSA to achieve both high performance and low complexity at the same time. CURSA is based on the CUR decomposition to decompose the multiplication of large matrices into the multiplication of several small matrices to achieve almost linear complexity. Experiment results of CURSA in image classification tasks, semantic segmentation tasks, object detection tasks, and long-range arena show that it outperforms state-of-the-art self-attention mechanisms with better data efficiency, faster speed, and higher accuracy. {\textcopyright} 2025 IEEE",

keywords = "Attention mechanism, CUR decomposition, linear approximation, vision transformer",

author = "Chong Wu and Maolin Che and Hong Yan",

year = "2026",

month = apr,

doi = "10.1109/TPAMI.2025.3646452",

language = "English",

volume = "48",

pages = "4792--4809",

journal = "IEEE Transactions on Pattern Analysis and Machine Intelligence",

issn = "0162-8828",

publisher = "IEEE Computer Society",

number = "4",

}

TY - JOUR

T1 - The CUR Decomposition of Self-Attention Matrices in Vision Transformers

AU - Wu, Chong

AU - Che, Maolin

AU - Yan, Hong

PY - 2026/4

Y1 - 2026/4

N2 - Transformers have achieved great success in natural language processing and computer vision. The core and basic technique of transformers is the self-attention mechanism. The vanilla self-attention mechanism has quadratic complexity, which limits its applications to vision tasks. Most of the existing linear self-attention mechanisms will sacrifice performance to some extent to reduce complexity. In this paper, we propose a novel linear approximation of the vanilla self-attention mechanism named CURSA to achieve both high performance and low complexity at the same time. CURSA is based on the CUR decomposition to decompose the multiplication of large matrices into the multiplication of several small matrices to achieve almost linear complexity. Experiment results of CURSA in image classification tasks, semantic segmentation tasks, object detection tasks, and long-range arena show that it outperforms state-of-the-art self-attention mechanisms with better data efficiency, faster speed, and higher accuracy. © 2025 IEEE

AB - Transformers have achieved great success in natural language processing and computer vision. The core and basic technique of transformers is the self-attention mechanism. The vanilla self-attention mechanism has quadratic complexity, which limits its applications to vision tasks. Most of the existing linear self-attention mechanisms will sacrifice performance to some extent to reduce complexity. In this paper, we propose a novel linear approximation of the vanilla self-attention mechanism named CURSA to achieve both high performance and low complexity at the same time. CURSA is based on the CUR decomposition to decompose the multiplication of large matrices into the multiplication of several small matrices to achieve almost linear complexity. Experiment results of CURSA in image classification tasks, semantic segmentation tasks, object detection tasks, and long-range arena show that it outperforms state-of-the-art self-attention mechanisms with better data efficiency, faster speed, and higher accuracy. © 2025 IEEE

KW - Attention mechanism

KW - CUR decomposition

KW - linear approximation

KW - vision transformer

UR - https://www.scopus.com/pages/publications/105025479274

UR - https://www.scopus.com/record/pubmetrics.uri?eid=2-s2.0-105025479274&origin=recordpage

UR - https://www.webofscience.com/wos/woscc/full-record/WOS:001708169500042

U2 - 10.1109/TPAMI.2025.3646452

DO - 10.1109/TPAMI.2025.3646452

M3 - RGC 21 - Publication in refereed journal

C2 - 41418007

SN - 0162-8828

VL - 48

SP - 4792

EP - 4809

JO - IEEE Transactions on Pattern Analysis and Machine Intelligence

JF - IEEE Transactions on Pattern Analysis and Machine Intelligence

IS - 4

ER -

The CUR Decomposition of Self-Attention Matrices in Vision Transformers

Abstract

Funding

Research Keywords

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

DON_RMG: IDM Cluster 2 : Health Data and Models - RMGS

Cite this