A NEW INITIALIZATION METHOD BASED ON NORMED STATISTICAL SPACES IN DEEP NETWORKS

Research output: Journal Publications and Reviews (RGC: 21, 22, 62)21_Publication in refereed journalpeer-review

6 Scopus Citations
View graph of relations

Author(s)

  • Hongfei YANG
  • Xiaofeng DING
  • Hui HU
  • Yaxin PENG
  • Tieyong ZENG

Related Research Unit(s)

Detail(s)

Original languageEnglish
Pages (from-to)147-158
Journal / PublicationInverse Problems and Imaging
Volume15
Issue number1
Online publishedAug 2020
Publication statusPublished - Feb 2021

Abstract

Training deep neural networks can be difficult. For classical neural networks, the initialization method by Xavier and Yoshua which is later generalized by He, Zhang, Ren and Sun can facilitate stable training. How-ever, with the recent development of new layer types, we find that the above mentioned initialization methods may fail to lead to successful training. Based on these two methods, we will propose a new initialization by studying the parameter space of a network. Our principal is to put constrains on the growth of parameters in different layers in a consistent way. In order to do so, we introduce a norm to the parameter space and use this norm to measure the growth of parameters. Our new method is suitable for a wide range of layer types, especially for layers with parameter-sharing weight matrices.

Research Area(s)

  • Deep learning, Model training, Neural networks, Parameters initialization, Parameters sharing

Citation Format(s)

A NEW INITIALIZATION METHOD BASED ON NORMED STATISTICAL SPACES IN DEEP NETWORKS. / YANG, Hongfei; DING, Xiaofeng; CHAN, Raymond et al.
In: Inverse Problems and Imaging, Vol. 15, No. 1, 02.2021, p. 147-158.

Research output: Journal Publications and Reviews (RGC: 21, 22, 62)21_Publication in refereed journalpeer-review