ReLU Network with Width d + O (1) Can Achieve Optimal Approximation Rate

Chenghao Liu; Minghua Chen

ReLU Network with Width d + O (1) Can Achieve Optimal Approximation Rate

Chenghao Liu
, Minghua Chen^*

^*Corresponding author for this work

Department of Data Science

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

Abstract

The prevalent employment of narrow neural networks, characterized by their minimal parameter count per layer, has led to a surge in research exploring their potential as universal function approximators. A notable result in this field states that networks with just a width of d + 1 can approximate any continuous function for input dimension d arbitrarily well. However, the optimal approximation rate for these narrowest networks, i.e., the optimal relation between the count of tunable parameters and the approximation error, remained unclear. In this paper, we address this gap by proving that ReLU networks with width d + 1 can achieve the optimal approximation rate for continuous functions over the domain [0, 1]^d under L^p norm for p ∈ [1, ∞). We further show that for the uniform norm, a width of d + 11 is sufficient. We also extend the results to narrow feed-forward networks with various activations, confirming their capability to approximate at the optimal rate. This work adds to the understanding of universal approximation of narrow networks.

© 2024 by the author(s)

Original language	English
Title of host publication	Proceedings of the 41^st International Conference on Machine Learning
Publisher	ML Research Press
Pages	30755-30788
Publication status	Published - 2024
Event	41st International Conference on Machine Learning (ICML 2024) - Messe Wien Exhibition Congress Center, Vienna, Austria Duration: 21 Jul 2024 → 27 Jul 2024 https://proceedings.mlr.press/v235/ https://icml.cc/

Publication series

Name	Proceedings of Machine Learning Research
Volume	235
ISSN (Print)	2640-3498

Conference

Conference	41st International Conference on Machine Learning (ICML 2024)
Place	Austria
City	Vienna
Period	21/07/24 → 27/07/24
Internet address	https://proceedings.mlr.press/v235/ https://icml.cc/

Funding

This work is supported in part by a General Research Fund from Research Grants Council, Hong Kong (Project No. 11203122), an InnoHK initiative, The Government of the HKSAR, Laboratory for AI-Powered Financial Technologies, and a Shenzhen-Hong Kong-Macau Science & Technology Project (Category C, Project No. SGDX20220530111203026). The authors would also like to thank the anonymous reviewers for their helpful comments.

RGC Funding Information

RGC-funded

Access Output

https://proceedings.mlr.press/v235/liu24g.html

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

@inproceedings{e886ea90563d4b32b520ad7276f93c1d,

title = "ReLU Network with Width d + O (1) Can Achieve Optimal Approximation Rate",

abstract = "The prevalent employment of narrow neural networks, characterized by their minimal parameter count per layer, has led to a surge in research exploring their potential as universal function approximators. A notable result in this field states that networks with just a width of d + 1 can approximate any continuous function for input dimension d arbitrarily well. However, the optimal approximation rate for these narrowest networks, i.e., the optimal relation between the count of tunable parameters and the approximation error, remained unclear. In this paper, we address this gap by proving that ReLU networks with width d + 1 can achieve the optimal approximation rate for continuous functions over the domain [0, 1]d under Lp norm for p ∈ [1, ∞). We further show that for the uniform norm, a width of d + 11 is sufficient. We also extend the results to narrow feed-forward networks with various activations, confirming their capability to approximate at the optimal rate. This work adds to the understanding of universal approximation of narrow networks. {\textcopyright} 2024 by the author(s)",

author = "Chenghao Liu and Minghua Chen",

year = "2024",

language = "English",

series = "Proceedings of Machine Learning Research",

publisher = "ML Research Press",

pages = "30755--30788",

booktitle = "Proceedings of the 41st International Conference on Machine Learning",

note = "41st International Conference on Machine Learning (ICML 2024) ; Conference date: 21-07-2024 Through 27-07-2024",

url = "https://proceedings.mlr.press/v235/, https://icml.cc/",

}

Liu, C & Chen, M 2024, ReLU Network with Width d + O (1) Can Achieve Optimal Approximation Rate. in Proceedings of the 41^st International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 235, ML Research Press, pp. 30755-30788, 41st International Conference on Machine Learning (ICML 2024), Vienna, Austria, 21/07/24. <https://proceedings.mlr.press/v235/liu24g.html>

ReLU Network with Width d + O (1) Can Achieve Optimal Approximation Rate. / Liu, Chenghao ; Chen, Minghua.
Proceedings of the 41^st International Conference on Machine Learning. ML Research Press, 2024. p. 30755-30788 (Proceedings of Machine Learning Research; Vol. 235).

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

TY - GEN

T1 - ReLU Network with Width d + O (1) Can Achieve Optimal Approximation Rate

AU - Liu, Chenghao

AU - Chen, Minghua

PY - 2024

Y1 - 2024

N2 - The prevalent employment of narrow neural networks, characterized by their minimal parameter count per layer, has led to a surge in research exploring their potential as universal function approximators. A notable result in this field states that networks with just a width of d + 1 can approximate any continuous function for input dimension d arbitrarily well. However, the optimal approximation rate for these narrowest networks, i.e., the optimal relation between the count of tunable parameters and the approximation error, remained unclear. In this paper, we address this gap by proving that ReLU networks with width d + 1 can achieve the optimal approximation rate for continuous functions over the domain [0, 1]d under Lp norm for p ∈ [1, ∞). We further show that for the uniform norm, a width of d + 11 is sufficient. We also extend the results to narrow feed-forward networks with various activations, confirming their capability to approximate at the optimal rate. This work adds to the understanding of universal approximation of narrow networks. © 2024 by the author(s)

AB - The prevalent employment of narrow neural networks, characterized by their minimal parameter count per layer, has led to a surge in research exploring their potential as universal function approximators. A notable result in this field states that networks with just a width of d + 1 can approximate any continuous function for input dimension d arbitrarily well. However, the optimal approximation rate for these narrowest networks, i.e., the optimal relation between the count of tunable parameters and the approximation error, remained unclear. In this paper, we address this gap by proving that ReLU networks with width d + 1 can achieve the optimal approximation rate for continuous functions over the domain [0, 1]d under Lp norm for p ∈ [1, ∞). We further show that for the uniform norm, a width of d + 11 is sufficient. We also extend the results to narrow feed-forward networks with various activations, confirming their capability to approximate at the optimal rate. This work adds to the understanding of universal approximation of narrow networks. © 2024 by the author(s)

UR - https://www.scopus.com/pages/publications/85203817347

UR - https://www.scopus.com/record/pubmetrics.uri?eid=2-s2.0-85203817347&origin=recordpage

M3 - RGC 32 - Refereed conference paper (with host publication)

T3 - Proceedings of Machine Learning Research

SP - 30755

EP - 30788

BT - Proceedings of the 41st International Conference on Machine Learning

PB - ML Research Press

T2 - 41st International Conference on Machine Learning (ICML 2024)

Y2 - 21 July 2024 through 27 July 2024

ER -

ReLU Network with Width d + O (1) Can Achieve Optimal Approximation Rate

Abstract

Publication series

Conference

Funding

RGC Funding Information

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

GRF: Developing Deep Neural Network Schemes for Solving Optimal Power Flow Problems: Solution Feasibility and Multiple Load-Solution Mappings

Cite this

ReLU Network with Width d + O (1) Can Achieve Optimal Approximation Rate

Abstract

Publication series

Conference

Funding

RGC Funding Information

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

Projects

GRF: Developing Deep Neural Network Schemes for Solving Optimal Power Flow Problems: Solution Feasibility and Multiple Load-Solution Mappings

Cite this