Global Convergence of Block Coordinate Descent in Deep Learning

Jinshan Zeng; Tim Tsz-Kit Lau; Shao-Bo Lin; Yuan Yao

Global Convergence of Block Coordinate Descent in Deep Learning

Jinshan Zeng (Co-first Author)
, Tim Tsz-Kit Lau (Co-first Author)
, Shao-Bo Lin
, Yuan Yao^*

^*Corresponding author for this work

Department of Mathematics

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

46 Citations (Scopus)

Abstract

Deep learning has aroused extensive attention due to its great empirical success. The efficiency of the block coordinate descent (BCD) methods has been recently demonstrated in deep neural network (DNN) training. However, theoretical studies on their convergence properties are limited due to the highly nonconvex nature of DNN training. In this paper, we aim at providing a general methodology for provable convergence guarantees for this type of methods. In particular, for most of the commonly used DNN training models involving both two-and three-splitting schemes, we establish the global convergence to a critical point at a rate of O(1/k), where k is the number of iterations. The results extend to general loss functions which have Lipschitz continuous gradients and deep residual networks (ResNets). Our key development adds several new elements to the Kurdyka-Łojasiewicz inequality framework that enables us to carry out the global convergence analysis of BCD in the general scenario of deep learning. © 2019 by the author(s).

Original language	English
Title of host publication	Proceedings of the 36th International Conference on Machine Learning
Publisher	ML Research Press
Pages	7313-7323
Publication status	Published - Jun 2019
Event	36th International Conference on Machine Learning (ICML 2019) - Long Beach, United States Duration: 9 Jun 2019 → 15 Jun 2019 https://icml.cc/

Publication series

Name	Proceedings of Machine Learning Research
Volume	97
ISSN (Print)	2640-3498

Conference

Conference	36th International Conference on Machine Learning (ICML 2019)
Place	United States
City	Long Beach
Period	9/06/19 → 15/06/19
Internet address	https://icml.cc/

Funding

The work of Jinshan Zeng is supported in part by the National Natural Science Foundation (NNSF) of China (No. 61603162, 61876074). The work of Tim Tsz-Kit Lau is supported by the University Fellowship for first-year Ph.D. students by The Graduate School at Northwestern University. The work of Shao-Bo Lin is supported in part by the NNSF of China (No. 61876133). The work of Yuan Yao is supported in part by HKRGC grant 16303817, 973 Program of China (No. 2015CB85600), NNSF of China (No. 61370004, 11421110001), as well as grants from Tencent AI Lab, Si Family Foundation, Baidu BDI, and Microsoft Research-Asia.

RGC Funding Information

RGC-funded

Access Output

http://proceedings.mlr.press/v97/zeng19a.html

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

@inproceedings{d050eed0634b4c459ba43150da2a2ddc,

title = "Global Convergence of Block Coordinate Descent in Deep Learning",

abstract = "Deep learning has aroused extensive attention due to its great empirical success. The efficiency of the block coordinate descent (BCD) methods has been recently demonstrated in deep neural network (DNN) training. However, theoretical studies on their convergence properties are limited due to the highly nonconvex nature of DNN training. In this paper, we aim at providing a general methodology for provable convergence guarantees for this type of methods. In particular, for most of the commonly used DNN training models involving both two-and three-splitting schemes, we establish the global convergence to a critical point at a rate of O(1/k), where k is the number of iterations. The results extend to general loss functions which have Lipschitz continuous gradients and deep residual networks (ResNets). Our key development adds several new elements to the Kurdyka-{\L}ojasiewicz inequality framework that enables us to carry out the global convergence analysis of BCD in the general scenario of deep learning. {\textcopyright} 2019 by the author(s).",

author = "Jinshan Zeng and Lau, \{Tim Tsz-Kit\} and Shao-Bo Lin and Yuan Yao",

year = "2019",

month = jun,

language = "English",

series = "Proceedings of Machine Learning Research",

publisher = "ML Research Press",

pages = "7313--7323",

booktitle = "Proceedings of the 36th International Conference on Machine Learning",

note = "36th International Conference on Machine Learning (ICML 2019) ; Conference date: 09-06-2019 Through 15-06-2019",

url = "https://icml.cc/",

}

Zeng, J, Lau, TT-K, Lin, S-B & Yao, Y 2019, Global Convergence of Block Coordinate Descent in Deep Learning. in Proceedings of the 36th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 97, ML Research Press, pp. 7313-7323, 36th International Conference on Machine Learning (ICML 2019), Long Beach, California, United States, 9/06/19. <http://proceedings.mlr.press/v97/zeng19a.html>

Global Convergence of Block Coordinate Descent in Deep Learning. / Zeng, Jinshan (Co-first Author); Lau, Tim Tsz-Kit (Co-first Author); Lin, Shao-Bo et al.
Proceedings of the 36th International Conference on Machine Learning. ML Research Press, 2019. p. 7313-7323 (Proceedings of Machine Learning Research; Vol. 97).

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

TY - GEN

T1 - Global Convergence of Block Coordinate Descent in Deep Learning

AU - Zeng, Jinshan

AU - Lau, Tim Tsz-Kit

AU - Lin, Shao-Bo

AU - Yao, Yuan

PY - 2019/6

Y1 - 2019/6

N2 - Deep learning has aroused extensive attention due to its great empirical success. The efficiency of the block coordinate descent (BCD) methods has been recently demonstrated in deep neural network (DNN) training. However, theoretical studies on their convergence properties are limited due to the highly nonconvex nature of DNN training. In this paper, we aim at providing a general methodology for provable convergence guarantees for this type of methods. In particular, for most of the commonly used DNN training models involving both two-and three-splitting schemes, we establish the global convergence to a critical point at a rate of O(1/k), where k is the number of iterations. The results extend to general loss functions which have Lipschitz continuous gradients and deep residual networks (ResNets). Our key development adds several new elements to the Kurdyka-Łojasiewicz inequality framework that enables us to carry out the global convergence analysis of BCD in the general scenario of deep learning. © 2019 by the author(s).

AB - Deep learning has aroused extensive attention due to its great empirical success. The efficiency of the block coordinate descent (BCD) methods has been recently demonstrated in deep neural network (DNN) training. However, theoretical studies on their convergence properties are limited due to the highly nonconvex nature of DNN training. In this paper, we aim at providing a general methodology for provable convergence guarantees for this type of methods. In particular, for most of the commonly used DNN training models involving both two-and three-splitting schemes, we establish the global convergence to a critical point at a rate of O(1/k), where k is the number of iterations. The results extend to general loss functions which have Lipschitz continuous gradients and deep residual networks (ResNets). Our key development adds several new elements to the Kurdyka-Łojasiewicz inequality framework that enables us to carry out the global convergence analysis of BCD in the general scenario of deep learning. © 2019 by the author(s).

UR - https://www.scopus.com/pages/publications/105020261964

UR - https://www.scopus.com/pages/publications/105020261964#tab=impact

M3 - RGC 32 - Refereed conference paper (with host publication)

T3 - Proceedings of Machine Learning Research

SP - 7313

EP - 7323

BT - Proceedings of the 36th International Conference on Machine Learning

PB - ML Research Press

T2 - 36th International Conference on Machine Learning (ICML 2019)

Y2 - 9 June 2019 through 15 June 2019

ER -

Global Convergence of Block Coordinate Descent in Deep Learning

Abstract

Publication series

Conference

Funding

RGC Funding Information

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

Cite this