FastViT: Real-Time Linear Attention Accelerator for Dense Predictions of Vision Transformer (ViT)

Zhuoheng Ran; Zewen Ye; Chong Wu; Ray C.C. Cheung; Hong Yan

doi:10.1109/ISCAS56072.2025.11043624

FastViT: Real-Time Linear Attention Accelerator for Dense Predictions of Vision Transformer (ViT)

Zhuoheng Ran^*, Zewen Ye, Chong Wu, Ray C.C. Cheung, Hong Yan

^*Corresponding author for this work

Department of Electrical Engineering

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

2 Citations (Scopus)

Abstract

The commercial success of generative artificial intelligence (GenAI) has driven an exponential surge in demand for real-time inference in Vision Transformer (ViT) applications, including latency-sensitive domains in autonomous driving, medical imaging and computational photography. This paper introduces FastViT, a high-performance and energy-efficient hardware accelerator for emerging kernel function-based linear attention mechanisms. By leveraging cost-efficient multiplication, mixed-precision quantisation and optimised data flow, FastViT improves real-time performance for high-resolution dense prediction tasks. Compared to existing approaches, experiments demonstrate that FastViT achieves higher throughput and energy efficiency while maintaining negligible accuracy degradation and balanced resource allocation. In the future, we will improve its scalability for next-generation hardware equipped with advanced DSP cores.

© 2025 IEEE.

Original language	English
Title of host publication	IEEE ISCAS 2025 SYMPOSIUM PROCEEDINGS
Publisher	IEEE
Number of pages	5
ISBN (Electronic)	979-8-3503-5683-0
ISBN (Print)	979-8-3503-5684-7
DOIs	https://doi.org/10.1109/ISCAS56072.2025.11043624
Publication status	Published - 2025
Event	2025 IEEE International Symposium on Circuits and Systems - London, United Kingdom Duration: 25 May 2025 → 28 May 2025 https://2025.ieee-iscas.org/

Publication series

Name
ISSN (Print)	0271-4302
ISSN (Electronic)	2158-1525

Conference

Conference	2025 IEEE International Symposium on Circuits and Systems
Abbreviated title	ISCAS 2025
Place	United Kingdom
City	London
Period	25/05/25 → 28/05/25
Internet address	https://2025.ieee-iscas.org/

Funding

This work is supported by Hong Kong Innovation and Technology Commission (InnoHK Project CIMDA), Hong Kong Research Grants Council (Project 11204821), and City University of Hong Kong (Project 9610460).

Research Keywords

vision transformer (ViT)
mixed-precision quantisation
kernel-based linear attention
hardware acceleration

RGC Funding Information

RGC-funded

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access Output

10.1109/ISCAS56072.2025.11043624

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

@inproceedings{6d61a28e901b438b929c09ce1b039736,

title = "FastViT: Real-Time Linear Attention Accelerator for Dense Predictions of Vision Transformer (ViT)",

abstract = "The commercial success of generative artificial intelligence (GenAI) has driven an exponential surge in demand for real-time inference in Vision Transformer (ViT) applications, including latency-sensitive domains in autonomous driving, medical imaging and computational photography. This paper introduces FastViT, a high-performance and energy-efficient hardware accelerator for emerging kernel function-based linear attention mechanisms. By leveraging cost-efficient multiplication, mixed-precision quantisation and optimised data flow, FastViT improves real-time performance for high-resolution dense prediction tasks. Compared to existing approaches, experiments demonstrate that FastViT achieves higher throughput and energy efficiency while maintaining negligible accuracy degradation and balanced resource allocation. In the future, we will improve its scalability for next-generation hardware equipped with advanced DSP cores.{\textcopyright} 2025 IEEE.",

keywords = "vision transformer (ViT), mixed-precision quantisation, kernel-based linear attention, hardware acceleration",

author = "Zhuoheng Ran and Zewen Ye and Chong Wu and Cheung, {Ray C.C.} and Hong Yan",

year = "2025",

doi = "10.1109/ISCAS56072.2025.11043624",

language = "English",

isbn = "979-8-3503-5684-7",

publisher = "IEEE",

booktitle = "IEEE ISCAS 2025 SYMPOSIUM PROCEEDINGS",

address = "United States",

note = "2025 IEEE International Symposium on Circuits and Systems, ISCAS 2025 ; Conference date: 25-05-2025 Through 28-05-2025",

url = "https://2025.ieee-iscas.org/",

}

TY - GEN

T1 - FastViT

T2 - 2025 IEEE International Symposium on Circuits and Systems

AU - Ran, Zhuoheng

AU - Ye, Zewen

AU - Wu, Chong

AU - Cheung, Ray C.C.

AU - Yan, Hong

PY - 2025

Y1 - 2025

N2 - The commercial success of generative artificial intelligence (GenAI) has driven an exponential surge in demand for real-time inference in Vision Transformer (ViT) applications, including latency-sensitive domains in autonomous driving, medical imaging and computational photography. This paper introduces FastViT, a high-performance and energy-efficient hardware accelerator for emerging kernel function-based linear attention mechanisms. By leveraging cost-efficient multiplication, mixed-precision quantisation and optimised data flow, FastViT improves real-time performance for high-resolution dense prediction tasks. Compared to existing approaches, experiments demonstrate that FastViT achieves higher throughput and energy efficiency while maintaining negligible accuracy degradation and balanced resource allocation. In the future, we will improve its scalability for next-generation hardware equipped with advanced DSP cores.© 2025 IEEE.

AB - The commercial success of generative artificial intelligence (GenAI) has driven an exponential surge in demand for real-time inference in Vision Transformer (ViT) applications, including latency-sensitive domains in autonomous driving, medical imaging and computational photography. This paper introduces FastViT, a high-performance and energy-efficient hardware accelerator for emerging kernel function-based linear attention mechanisms. By leveraging cost-efficient multiplication, mixed-precision quantisation and optimised data flow, FastViT improves real-time performance for high-resolution dense prediction tasks. Compared to existing approaches, experiments demonstrate that FastViT achieves higher throughput and energy efficiency while maintaining negligible accuracy degradation and balanced resource allocation. In the future, we will improve its scalability for next-generation hardware equipped with advanced DSP cores.© 2025 IEEE.

KW - vision transformer (ViT)

KW - mixed-precision quantisation

KW - kernel-based linear attention

KW - hardware acceleration

UR - http://www.scopus.com/inward/record.url?scp=105010585388&partnerID=8YFLogxK

UR - https://www.scopus.com/record/pubmetrics.uri?eid=2-s2.0-105010585388&origin=recordpage

U2 - 10.1109/ISCAS56072.2025.11043624

DO - 10.1109/ISCAS56072.2025.11043624

M3 - RGC 32 - Refereed conference paper (with host publication)

SN - 979-8-3503-5684-7

BT - IEEE ISCAS 2025 SYMPOSIUM PROCEEDINGS

PB - IEEE

Y2 - 25 May 2025 through 28 May 2025

ER -

FastViT: Real-Time Linear Attention Accelerator for Dense Predictions of Vision Transformer (ViT)