Unlocking the Full Potential of Separable Convolutions on Tensor Cores

Aodie Cui; Chuangxin Zhao; Xiaoyu Deng; Gaozhe Jiang; Yifan Yang; Guangzhen Yao; Renda Han; Wenxin Zhang; Xi Xuan

doi:10.1007/978-981-96-9921-6_4

Unlocking the Full Potential of Separable Convolutions on Tensor Cores

Aodie Cui (Co-first Author)
, Chuangxin Zhao (Co-first Author)
, Xiaoyu Deng
, Gaozhe Jiang
, Yifan Yang
, Guangzhen Yao
, Renda Han
, Wenxin Zhang^*
, Xi Xuan

^*Corresponding author for this work

City University of Hong Kong

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

1 Citation (Scopus)

Abstract

While separable convolutions have demonstrated great performance in network design, they suffer from poor efficiency on Tensor Cores-equipped GPUs. This paper proposes TensorFuse, which exploits the Tensor Cores by transforming nested-loops into hierarchy matrix multiplications for kernel fusion. TensorFuse minimizes redundant memory accesses by efficiently lowering GEMM-based convolution along execution hierarchy: from shared memory to register files. Compared with the state-of-the-art, it achieves up to 2.60 × inference speed on Tensor Cores. Furthermore, we explore the performance of network decoupling with multiple separable convolutions. TensorFuse consistently outperforms state-of-the-art libraries with up to 2.76 × acceleration on modern CNN benchmarks. © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2025

Original language	English
Title of host publication	Advanced Intelligent Computing Technology and Applications
Subtitle of host publication	21st International Conference, ICIC 2025, Ningbo, China, July 26–29, 2025, Proceedings, Part XVI
Editors	De-Shuang Huang, Wei Chen, Yijie Pan, Haiming Chen
Place of Publication	Singapore
Publisher	Springer
Pages	39-50
ISBN (Electronic)	978-981-96-9921-6
ISBN (Print)	978-981-96-9920-9
DOIs	https://doi.org/10.1007/978-981-96-9921-6_4
Publication status	Published - 2025
Event	2025 International Conference on Intelligent Computing (ICIC 2025) - Ningbo, China Duration: 26 Jul 2025 → 29 Jul 2025

Publication series

Name	Lecture Notes in Computer Science
Volume	15857
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	2025 International Conference on Intelligent Computing (ICIC 2025)
Abbreviated title	ICIC2025
Place	China
City	Ningbo
Period	26/07/25 → 29/07/25

Funding

This research was funded by the National Natural Science Foundation of China under Grant 72210107001, the Beijing Natural Science Foundation under Grant IS23128, the Fundamental Research Funds for the Central Universities, and the CAS PIFI International Outstanding Team Project (2024PG0013).

Access Output

10.1007/978-981-96-9921-6_4

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

Cui, A., Zhao, C., Deng, X., Jiang, G., Yang, Y., Yao, G., Han, R., Zhang, W., & Xuan, X. (2025). Unlocking the Full Potential of Separable Convolutions on Tensor Cores. In D.-S. Huang, W. Chen, Y. Pan, & H. Chen (Eds.), Advanced Intelligent Computing Technology and Applications: 21st International Conference, ICIC 2025, Ningbo, China, July 26–29, 2025, Proceedings, Part XVI (pp. 39-50). (Lecture Notes in Computer Science; Vol. 15857). Springer . https://doi.org/10.1007/978-981-96-9921-6_4

Cui, Aodie ; Zhao, Chuangxin ; Deng, Xiaoyu et al. / Unlocking the Full Potential of Separable Convolutions on Tensor Cores. Advanced Intelligent Computing Technology and Applications: 21st International Conference, ICIC 2025, Ningbo, China, July 26–29, 2025, Proceedings, Part XVI. editor / De-Shuang Huang ; Wei Chen ; Yijie Pan ; Haiming Chen. Singapore : Springer , 2025. pp. 39-50 (Lecture Notes in Computer Science).

@inproceedings{576367a09fa74ccdaf03619e454bbf00,

title = "Unlocking the Full Potential of Separable Convolutions on Tensor Cores",

abstract = "While separable convolutions have demonstrated great performance in network design, they suffer from poor efficiency on Tensor Cores-equipped GPUs. This paper proposes TensorFuse, which exploits the Tensor Cores by transforming nested-loops into hierarchy matrix multiplications for kernel fusion. TensorFuse minimizes redundant memory accesses by efficiently lowering GEMM-based convolution along execution hierarchy: from shared memory to register files. Compared with the state-of-the-art, it achieves up to 2.60 × inference speed on Tensor Cores. Furthermore, we explore the performance of network decoupling with multiple separable convolutions. TensorFuse consistently outperforms state-of-the-art libraries with up to 2.76 × acceleration on modern CNN benchmarks. {\textcopyright} The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2025",

author = "Aodie Cui and Chuangxin Zhao and Xiaoyu Deng and Gaozhe Jiang and Yifan Yang and Guangzhen Yao and Renda Han and Wenxin Zhang and Xi Xuan",

year = "2025",

doi = "10.1007/978-981-96-9921-6\_4",

language = "English",

isbn = "978-981-96-9920-9",

series = "Lecture Notes in Computer Science",

publisher = "Springer ",

pages = "39--50",

editor = "Huang, \{De-Shuang \} and Chen, \{Wei \} and Pan, \{Yijie \} and Chen, \{Haiming \}",

booktitle = "Advanced Intelligent Computing Technology and Applications",

note = "2025 International Conference on Intelligent Computing (ICIC 2025), ICIC2025 ; Conference date: 26-07-2025 Through 29-07-2025",

}

Cui, A, Zhao, C, Deng, X, Jiang, G, Yang, Y, Yao, G, Han, R, Zhang, W & Xuan, X 2025, Unlocking the Full Potential of Separable Convolutions on Tensor Cores. in D-S Huang, W Chen, Y Pan & H Chen (eds), Advanced Intelligent Computing Technology and Applications: 21st International Conference, ICIC 2025, Ningbo, China, July 26–29, 2025, Proceedings, Part XVI. Lecture Notes in Computer Science, vol. 15857, Springer , Singapore, pp. 39-50, 2025 International Conference on Intelligent Computing (ICIC 2025), Ningbo, China, 26/07/25. https://doi.org/10.1007/978-981-96-9921-6_4

Unlocking the Full Potential of Separable Convolutions on Tensor Cores. / Cui, Aodie (Co-first Author); Zhao, Chuangxin (Co-first Author); Deng, Xiaoyu et al.
Advanced Intelligent Computing Technology and Applications: 21st International Conference, ICIC 2025, Ningbo, China, July 26–29, 2025, Proceedings, Part XVI. ed. / De-Shuang Huang; Wei Chen; Yijie Pan; Haiming Chen. Singapore: Springer , 2025. p. 39-50 (Lecture Notes in Computer Science; Vol. 15857).

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

TY - GEN

T1 - Unlocking the Full Potential of Separable Convolutions on Tensor Cores

AU - Cui, Aodie

AU - Zhao, Chuangxin

AU - Deng, Xiaoyu

AU - Jiang, Gaozhe

AU - Yang, Yifan

AU - Yao, Guangzhen

AU - Han, Renda

AU - Zhang, Wenxin

AU - Xuan, Xi

PY - 2025

Y1 - 2025

N2 - While separable convolutions have demonstrated great performance in network design, they suffer from poor efficiency on Tensor Cores-equipped GPUs. This paper proposes TensorFuse, which exploits the Tensor Cores by transforming nested-loops into hierarchy matrix multiplications for kernel fusion. TensorFuse minimizes redundant memory accesses by efficiently lowering GEMM-based convolution along execution hierarchy: from shared memory to register files. Compared with the state-of-the-art, it achieves up to 2.60 × inference speed on Tensor Cores. Furthermore, we explore the performance of network decoupling with multiple separable convolutions. TensorFuse consistently outperforms state-of-the-art libraries with up to 2.76 × acceleration on modern CNN benchmarks. © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2025

AB - While separable convolutions have demonstrated great performance in network design, they suffer from poor efficiency on Tensor Cores-equipped GPUs. This paper proposes TensorFuse, which exploits the Tensor Cores by transforming nested-loops into hierarchy matrix multiplications for kernel fusion. TensorFuse minimizes redundant memory accesses by efficiently lowering GEMM-based convolution along execution hierarchy: from shared memory to register files. Compared with the state-of-the-art, it achieves up to 2.60 × inference speed on Tensor Cores. Furthermore, we explore the performance of network decoupling with multiple separable convolutions. TensorFuse consistently outperforms state-of-the-art libraries with up to 2.76 × acceleration on modern CNN benchmarks. © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2025

UR - https://www.scopus.com/pages/publications/105012922833

UR - https://www.scopus.com/record/pubmetrics.uri?eid=2-s2.0-105012922833&origin=recordpage

UR - https://www.webofscience.com/wos/woscc/full-record/WOS:001568983600004

U2 - 10.1007/978-981-96-9921-6_4

DO - 10.1007/978-981-96-9921-6_4

M3 - RGC 32 - Refereed conference paper (with host publication)

SN - 978-981-96-9920-9

T3 - Lecture Notes in Computer Science

SP - 39

EP - 50

BT - Advanced Intelligent Computing Technology and Applications

A2 - Huang, De-Shuang

A2 - Chen, Wei

A2 - Pan, Yijie

A2 - Chen, Haiming

PB - Springer

CY - Singapore

T2 - 2025 International Conference on Intelligent Computing (ICIC 2025)

Y2 - 26 July 2025 through 29 July 2025

ER -

Cui A, Zhao C, Deng X, Jiang G, Yang Y, Yao G et al. Unlocking the Full Potential of Separable Convolutions on Tensor Cores. In Huang DS, Chen W, Pan Y, Chen H, editors, Advanced Intelligent Computing Technology and Applications: 21st International Conference, ICIC 2025, Ningbo, China, July 26–29, 2025, Proceedings, Part XVI. Singapore: Springer . 2025. p. 39-50. (Lecture Notes in Computer Science). Epub 2025 Jul 26. doi: 10.1007/978-981-96-9921-6_4

Unlocking the Full Potential of Separable Convolutions on Tensor Cores

Abstract

Publication series

Conference

Funding

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

Cite this