Impact of Mathematical Norms on Convergence of Gradient Descent Algorithms for Deep Neural Networks Learning

Linzhe Cai; Xinghuo Yu; Chaojie Li; Andrew Eberhard; Lien Thuy Nguyen; Chuong Thai Doan

doi:10.1007/978-3-031-22695-3_10

Impact of Mathematical Norms on Convergence of Gradient Descent Algorithms for Deep Neural Networks Learning

Linzhe Cai^*
, Xinghuo Yu
, Chaojie Li
, Andrew Eberhard
, Lien Thuy Nguyen
, Chuong Thai Doan

^*Corresponding author for this work

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

1 Citation (Scopus)

Abstract

To improve the performance of gradient descent learning algorithms, the impact of different types of norms is studied for deep neural network training. The performance of different norm types used on both finite-time and fixed-time convergence algorithms are compared. The accuracy of the multiclassification task realized by three typical algorithms using different types of norms is given, and the improvement of Jorge’s finite time algorithm with momentum or Nesterov accelerated gradient is also studied. Numerical experiments show that the infinity norm can provide better performance in finite time gradient descent algorithms and give strong robustness under different network structures. © 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.

Original language	English
Title of host publication	AI 2022: Advances in Artificial Intelligence
Subtitle of host publication	35th Australasian Joint Conference, AI 2022, Perth, WA, Australia, December 5–8, 2022, Proceedings
Editors	Haris Aziz, Débora Corrêa, Tim French
Place of Publication	Cham
Publisher	Springer
Pages	131-144
ISBN (Electronic)	978-3-031-22695-3
ISBN (Print)	9783031226946
DOIs	https://doi.org/10.1007/978-3-031-22695-3_10
Publication status	Published - 2022
Externally published	Yes
Event	35th Australasian Joint Conference on Artificial Intelligence (AI 2022) - Perth, Australia Duration: 5 Dec 2022 → 8 Dec 2022

Publication series

Name	Lecture Notes in Computer Science
Volume	13728
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	35th Australasian Joint Conference on Artificial Intelligence (AI 2022)
Place	Australia
City	Perth
Period	5/12/22 → 8/12/22

Funding

The authors were supported by the Australian Research Council (ARC) under Discovery Program Grant DP200101197.

Research Keywords

Deep neural network
Finite-time convergence
Infinity norm
Norms equivalence

Access Output

10.1007/978-3-031-22695-3_10

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

Cai, L., Yu, X., Li, C., Eberhard, A., Nguyen, L. T., & Doan, C. T. (2022). Impact of Mathematical Norms on Convergence of Gradient Descent Algorithms for Deep Neural Networks Learning. In H. Aziz, D. Corrêa, & T. French (Eds.), AI 2022: Advances in Artificial Intelligence: 35th Australasian Joint Conference, AI 2022, Perth, WA, Australia, December 5–8, 2022, Proceedings (pp. 131-144). (Lecture Notes in Computer Science; Vol. 13728). Springer . https://doi.org/10.1007/978-3-031-22695-3_10

Cai, Linzhe ; Yu, Xinghuo ; Li, Chaojie et al. / Impact of Mathematical Norms on Convergence of Gradient Descent Algorithms for Deep Neural Networks Learning. AI 2022: Advances in Artificial Intelligence: 35th Australasian Joint Conference, AI 2022, Perth, WA, Australia, December 5–8, 2022, Proceedings. editor / Haris Aziz ; Débora Corrêa ; Tim French. Cham : Springer , 2022. pp. 131-144 (Lecture Notes in Computer Science).

@inproceedings{ccfc3574f6824a7e96c520165df670c6,

title = "Impact of Mathematical Norms on Convergence of Gradient Descent Algorithms for Deep Neural Networks Learning",

abstract = "To improve the performance of gradient descent learning algorithms, the impact of different types of norms is studied for deep neural network training. The performance of different norm types used on both finite-time and fixed-time convergence algorithms are compared. The accuracy of the multiclassification task realized by three typical algorithms using different types of norms is given, and the improvement of Jorge{\textquoteright}s finite time algorithm with momentum or Nesterov accelerated gradient is also studied. Numerical experiments show that the infinity norm can provide better performance in finite time gradient descent algorithms and give strong robustness under different network structures. {\textcopyright} 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.",

keywords = "Deep neural network, Finite-time convergence, Infinity norm, Norms equivalence",

author = "Linzhe Cai and Xinghuo Yu and Chaojie Li and Andrew Eberhard and Nguyen, \{Lien Thuy\} and Doan, \{Chuong Thai\}",

year = "2022",

doi = "10.1007/978-3-031-22695-3\_10",

language = "English",

isbn = "9783031226946",

series = "Lecture Notes in Computer Science",

publisher = "Springer ",

pages = "131--144",

editor = "Aziz, \{Haris \} and Corr{\^e}a, \{D{\'e}bora \} and French, \{Tim \}",

booktitle = "AI 2022: Advances in Artificial Intelligence",

note = "35th Australasian Joint Conference on Artificial Intelligence (AI 2022) ; Conference date: 05-12-2022 Through 08-12-2022",

}

Cai, L, Yu, X, Li, C, Eberhard, A, Nguyen, LT & Doan, CT 2022, Impact of Mathematical Norms on Convergence of Gradient Descent Algorithms for Deep Neural Networks Learning. in H Aziz, D Corrêa & T French (eds), AI 2022: Advances in Artificial Intelligence: 35th Australasian Joint Conference, AI 2022, Perth, WA, Australia, December 5–8, 2022, Proceedings. Lecture Notes in Computer Science, vol. 13728, Springer , Cham, pp. 131-144, 35th Australasian Joint Conference on Artificial Intelligence (AI 2022) , Perth, Australia, 5/12/22. https://doi.org/10.1007/978-3-031-22695-3_10

Impact of Mathematical Norms on Convergence of Gradient Descent Algorithms for Deep Neural Networks Learning. / Cai, Linzhe; Yu, Xinghuo; Li, Chaojie et al.
AI 2022: Advances in Artificial Intelligence: 35th Australasian Joint Conference, AI 2022, Perth, WA, Australia, December 5–8, 2022, Proceedings. ed. / Haris Aziz; Débora Corrêa; Tim French. Cham: Springer , 2022. p. 131-144 (Lecture Notes in Computer Science; Vol. 13728).

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

TY - GEN

T1 - Impact of Mathematical Norms on Convergence of Gradient Descent Algorithms for Deep Neural Networks Learning

AU - Cai, Linzhe

AU - Yu, Xinghuo

AU - Li, Chaojie

AU - Eberhard, Andrew

AU - Nguyen, Lien Thuy

AU - Doan, Chuong Thai

PY - 2022

Y1 - 2022

N2 - To improve the performance of gradient descent learning algorithms, the impact of different types of norms is studied for deep neural network training. The performance of different norm types used on both finite-time and fixed-time convergence algorithms are compared. The accuracy of the multiclassification task realized by three typical algorithms using different types of norms is given, and the improvement of Jorge’s finite time algorithm with momentum or Nesterov accelerated gradient is also studied. Numerical experiments show that the infinity norm can provide better performance in finite time gradient descent algorithms and give strong robustness under different network structures. © 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.

AB - To improve the performance of gradient descent learning algorithms, the impact of different types of norms is studied for deep neural network training. The performance of different norm types used on both finite-time and fixed-time convergence algorithms are compared. The accuracy of the multiclassification task realized by three typical algorithms using different types of norms is given, and the improvement of Jorge’s finite time algorithm with momentum or Nesterov accelerated gradient is also studied. Numerical experiments show that the infinity norm can provide better performance in finite time gradient descent algorithms and give strong robustness under different network structures. © 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.

KW - Deep neural network

KW - Finite-time convergence

KW - Infinity norm

KW - Norms equivalence

UR - https://www.scopus.com/pages/publications/85144814665

UR - https://www.scopus.com/pages/publications/85144814665#tab=impact

U2 - 10.1007/978-3-031-22695-3_10

DO - 10.1007/978-3-031-22695-3_10

M3 - RGC 32 - Refereed conference paper (with host publication)

SN - 9783031226946

T3 - Lecture Notes in Computer Science

SP - 131

EP - 144

BT - AI 2022: Advances in Artificial Intelligence

A2 - Aziz, Haris

A2 - Corrêa, Débora

A2 - French, Tim

PB - Springer

CY - Cham

T2 - 35th Australasian Joint Conference on Artificial Intelligence (AI 2022)

Y2 - 5 December 2022 through 8 December 2022

ER -

Cai L, Yu X, Li C, Eberhard A, Nguyen LT, Doan CT. Impact of Mathematical Norms on Convergence of Gradient Descent Algorithms for Deep Neural Networks Learning. In Aziz H, Corrêa D, French T, editors, AI 2022: Advances in Artificial Intelligence: 35th Australasian Joint Conference, AI 2022, Perth, WA, Australia, December 5–8, 2022, Proceedings. Cham: Springer . 2022. p. 131-144. (Lecture Notes in Computer Science). Epub 2022 Dec 3. doi: 10.1007/978-3-031-22695-3_10

Impact of Mathematical Norms on Convergence of Gradient Descent Algorithms for Deep Neural Networks Learning

Abstract

Publication series

Conference

Funding

Research Keywords

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

Cite this