Realization of Spatial Sparseness by Deep ReLU Nets with Massive Data

Research output: Journal Publications and ReviewsRGC 21 - Publication in refereed journalpeer-review

20 Scopus Citations
View graph of relations

Author(s)

  • Charles K. Chui
  • Shao-Bo Lin
  • Bo Zhang
  • Ding-Xuan Zhou

Detail(s)

Original languageEnglish
Pages (from-to)229-243
Journal / PublicationIEEE Transactions on Neural Networks and Learning Systems
Volume33
Issue number1
Online published16 Oct 2020
Publication statusPublished - Jan 2022

Abstract

The great success of deep learning poses urgent challenges for understanding its working mechanism and rationality. The depth, structure, and massive size of the data are recognized to be three key ingredients for deep learning. Most of the recent theoretical studies for deep learning focus on the necessity and advantages of depth and structures of neural networks. In this article, we aim at rigorous verification of the importance of massive data in embodying the outperformance of deep learning. In particular, we prove that the massiveness of data is necessary for realizing the spatial sparseness, and deep nets are crucial tools to make full use of massive data in such an application. All these findings present the reasons why deep learning achieves great success in the era of big data though deep nets and numerous network structures have been proposed at least 20 years ago.

Research Area(s)

  • Deep nets, learning theory, massive data, spatial sparseness

Citation Format(s)

Realization of Spatial Sparseness by Deep ReLU Nets with Massive Data. / Chui, Charles K.; Lin, Shao-Bo; Zhang, Bo et al.
In: IEEE Transactions on Neural Networks and Learning Systems, Vol. 33, No. 1, 01.2022, p. 229-243.

Research output: Journal Publications and ReviewsRGC 21 - Publication in refereed journalpeer-review