A New Initialization Method for Neural Networks with Weight Sharing

Xiaofeng Ding; Hongfei Yang; Raymond H. Chan; Hui Hu; Yaxin Peng; Tieyong Zeng

doi:10.1007/978-981-16-2701-9_9

A New Initialization Method for Neural Networks with Weight Sharing

Xiaofeng Ding
, Hongfei Yang
, Raymond H. Chan
, Hui Hu
, Yaxin Peng
, Tieyong Zeng^*

^*Corresponding author for this work

Department of Mathematics

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

5 Citations (Scopus)

Abstract

A proper initialization of parameters in a neural network can facilitate its training. The Xavier initialization introduced by Glorot and Bengio which is later generalized to Kaiming initialization by He, Zhang, Ren and Sun are now widely used. However, from experiments we find that networks with heavy weight sharing are difficulty to train even with the Xavier or the Kaiming initialization. We also notice that a certain simple network can be decomposed in two ways, where one is difficult to train while the other is easy to train, when both are properly initialized by the Xavier or the Kaiming initialization. In this paper we will propose a new initialization method which will increase training speed and training stability of neural networks with heavy weight sharing. We will also propose a simple yet efficient method to adjust learning rates layer by layer which is indispensable to our initialization.

Original language	English
Title of host publication	Mathematical Methods in Image Processing and Inverse Problems, IPIP 2018
Editors	Xue-Cheng Tai, Suhua Wei, Haiguang Liu
Publisher	Springer Nature Singapore Pte Ltd.
Pages	165-179
ISBN (Electronic)	9789811627019
ISBN (Print)	9789811627002
DOIs	https://doi.org/10.1007/978-981-16-2701-9_9
Publication status	Published - 2021
Event	International Workshop on Image Processing and Inverse Problems (IPIP 2018) - Beijing Computational Science Research Center, Beijing, China Duration: 21 Apr 2018 → 24 Apr 2018 https://www.csrc.ac.cn/en/event/workshop/2017-12-27/78.html

Publication series

Name	Springer Proceedings in Mathematics and Statistics
Volume	360
ISSN (Print)	2194-1009
ISSN (Electronic)	2194-1017

Conference

Conference	International Workshop on Image Processing and Inverse Problems (IPIP 2018)
Place	China
City	Beijing
Period	21/04/18 → 24/04/18
Internet address	https://www.csrc.ac.cn/en/event/workshop/2017-12-27/78.html

Research Keywords

Learning rate
Neural networks
Weight sharing
Xavier initialization

RGC Funding Information

RGC-funded

Access Output

10.1007/978-981-16-2701-9_9

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

Ding, X., Yang, H., Chan, R. H., Hu, H., Peng, Y., & Zeng, T. (2021). A New Initialization Method for Neural Networks with Weight Sharing. In X.-C. Tai, S. Wei, & H. Liu (Eds.), Mathematical Methods in Image Processing and Inverse Problems, IPIP 2018 (pp. 165-179). (Springer Proceedings in Mathematics and Statistics; Vol. 360). Springer Nature Singapore Pte Ltd.. https://doi.org/10.1007/978-981-16-2701-9_9

@inproceedings{ddf7c759d558483caee3fa922096210e,

title = "A New Initialization Method for Neural Networks with Weight Sharing",

abstract = "A proper initialization of parameters in a neural network can facilitate its training. The Xavier initialization introduced by Glorot and Bengio which is later generalized to Kaiming initialization by He, Zhang, Ren and Sun are now widely used. However, from experiments we find that networks with heavy weight sharing are difficulty to train even with the Xavier or the Kaiming initialization. We also notice that a certain simple network can be decomposed in two ways, where one is difficult to train while the other is easy to train, when both are properly initialized by the Xavier or the Kaiming initialization. In this paper we will propose a new initialization method which will increase training speed and training stability of neural networks with heavy weight sharing. We will also propose a simple yet efficient method to adjust learning rates layer by layer which is indispensable to our initialization.",

keywords = "Learning rate, Neural networks, Weight sharing, Xavier initialization",

author = "Xiaofeng Ding and Hongfei Yang and Chan, \{Raymond H.\} and Hui Hu and Yaxin Peng and Tieyong Zeng",

year = "2021",

doi = "10.1007/978-981-16-2701-9\_9",

language = "English",

isbn = "9789811627002",

series = "Springer Proceedings in Mathematics and Statistics",

publisher = "Springer Nature Singapore Pte Ltd.",

pages = "165--179",

editor = "Xue-Cheng Tai and Suhua Wei and Haiguang Liu",

booktitle = "Mathematical Methods in Image Processing and Inverse Problems, IPIP 2018",

note = "International Workshop on Image Processing and Inverse Problems (IPIP 2018) ; Conference date: 21-04-2018 Through 24-04-2018",

url = "https://www.csrc.ac.cn/en/event/workshop/2017-12-27/78.html",

}

Ding, X, Yang, H, Chan, RH, Hu, H, Peng, Y & Zeng, T 2021, A New Initialization Method for Neural Networks with Weight Sharing. in X-C Tai, S Wei & H Liu (eds), Mathematical Methods in Image Processing and Inverse Problems, IPIP 2018. Springer Proceedings in Mathematics and Statistics, vol. 360, Springer Nature Singapore Pte Ltd., pp. 165-179, International Workshop on Image Processing and Inverse Problems (IPIP 2018), Beijing, China, 21/04/18. https://doi.org/10.1007/978-981-16-2701-9_9

A New Initialization Method for Neural Networks with Weight Sharing. / Ding, Xiaofeng; Yang, Hongfei; Chan, Raymond H. et al.
Mathematical Methods in Image Processing and Inverse Problems, IPIP 2018. ed. / Xue-Cheng Tai; Suhua Wei; Haiguang Liu. Springer Nature Singapore Pte Ltd., 2021. p. 165-179 (Springer Proceedings in Mathematics and Statistics; Vol. 360).

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

TY - GEN

T1 - A New Initialization Method for Neural Networks with Weight Sharing

AU - Ding, Xiaofeng

AU - Yang, Hongfei

AU - Chan, Raymond H.

AU - Hu, Hui

AU - Peng, Yaxin

AU - Zeng, Tieyong

PY - 2021

Y1 - 2021

N2 - A proper initialization of parameters in a neural network can facilitate its training. The Xavier initialization introduced by Glorot and Bengio which is later generalized to Kaiming initialization by He, Zhang, Ren and Sun are now widely used. However, from experiments we find that networks with heavy weight sharing are difficulty to train even with the Xavier or the Kaiming initialization. We also notice that a certain simple network can be decomposed in two ways, where one is difficult to train while the other is easy to train, when both are properly initialized by the Xavier or the Kaiming initialization. In this paper we will propose a new initialization method which will increase training speed and training stability of neural networks with heavy weight sharing. We will also propose a simple yet efficient method to adjust learning rates layer by layer which is indispensable to our initialization.

AB - A proper initialization of parameters in a neural network can facilitate its training. The Xavier initialization introduced by Glorot and Bengio which is later generalized to Kaiming initialization by He, Zhang, Ren and Sun are now widely used. However, from experiments we find that networks with heavy weight sharing are difficulty to train even with the Xavier or the Kaiming initialization. We also notice that a certain simple network can be decomposed in two ways, where one is difficult to train while the other is easy to train, when both are properly initialized by the Xavier or the Kaiming initialization. In this paper we will propose a new initialization method which will increase training speed and training stability of neural networks with heavy weight sharing. We will also propose a simple yet efficient method to adjust learning rates layer by layer which is indispensable to our initialization.

KW - Learning rate

KW - Neural networks

KW - Weight sharing

KW - Xavier initialization

UR - https://www.scopus.com/pages/publications/85116465564

UR - https://www.scopus.com/record/pubmetrics.uri?eid=2-s2.0-85116465564&origin=recordpage

U2 - 10.1007/978-981-16-2701-9_9

DO - 10.1007/978-981-16-2701-9_9

M3 - RGC 32 - Refereed conference paper (with host publication)

SN - 9789811627002

T3 - Springer Proceedings in Mathematics and Statistics

SP - 165

EP - 179

BT - Mathematical Methods in Image Processing and Inverse Problems, IPIP 2018

A2 - Tai, Xue-Cheng

A2 - Wei, Suhua

A2 - Liu, Haiguang

PB - Springer Nature Singapore Pte Ltd.

T2 - International Workshop on Image Processing and Inverse Problems (IPIP 2018)

Y2 - 21 April 2018 through 24 April 2018

ER -

Ding X, Yang H, Chan RH, Hu H, Peng Y, Zeng T. A New Initialization Method for Neural Networks with Weight Sharing. In Tai XC, Wei S, Liu H, editors, Mathematical Methods in Image Processing and Inverse Problems, IPIP 2018. Springer Nature Singapore Pte Ltd. 2021. p. 165-179. (Springer Proceedings in Mathematics and Statistics). Epub 2021 Sept 26. doi: 10.1007/978-981-16-2701-9_9

A New Initialization Method for Neural Networks with Weight Sharing

Abstract

Publication series

Conference

Research Keywords

RGC Funding Information

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

Cite this