Convergence analysis of multiplicative weight noise injection during training

Kevin Ho; Chi-Sing Leung; John Sum; Siu-Chung Lau

doi:10.1109/TAAI.2010.64

Convergence analysis of multiplicative weight noise injection during training

Kevin Ho
, Chi-Sing Leung
, John Sum
, Siu-Chung Lau

Department of Electrical Engineering

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

1 Citation (Scopus)

Abstract

Injecting weight noise during training has been proposed for almost two decades as a simple technique to improve fault tolerance and generalization of a multilayer perceptron (MLP). However, little has been done regarding their convergence behaviors. Therefore, we presents in this paper the convergence proofs of two of these algorithms for MLPs. One is based on combining injecting multiplicative weight noise and weight decay (MWN-WD) during training. The other is based on combining injecting additive weight noise and weight decay (AWN-WD) during training. Let m be the number of hidden nodes of a MLP, α be the weight decay constant and S_b be the noise variance. It is showed that the convergence of MWN-WD algorithm is with probability one if α > √S_bm. While the convergence of the AWN-WD algorithm is with probability one if α > 0. © 2010 IEEE.

Original language	English
Title of host publication	Proceedings - International Conference on Technologies and Applications of Artificial Intelligence, TAAI 2010
Pages	358-365
DOIs	https://doi.org/10.1109/TAAI.2010.64
Publication status	Published - 2010
Event	2010 15th Conference on Technologies and Applications of Artificial Intelligence, TAAI 2010 - Hsinchu, Taiwan, China Duration: 18 Nov 2010 → 20 Nov 2010

Conference

Conference	2010 15th Conference on Technologies and Applications of Artificial Intelligence, TAAI 2010
Place	Taiwan, China
City	Hsinchu
Period	18/11/10 → 20/11/10

Research Keywords

Convergence
Learning
MLP
Weight noise

Access Output

10.1109/TAAI.2010.64

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

@inproceedings{710327fac7e84bd7a62074e709276990,

title = "Convergence analysis of multiplicative weight noise injection during training",

abstract = "Injecting weight noise during training has been proposed for almost two decades as a simple technique to improve fault tolerance and generalization of a multilayer perceptron (MLP). However, little has been done regarding their convergence behaviors. Therefore, we presents in this paper the convergence proofs of two of these algorithms for MLPs. One is based on combining injecting multiplicative weight noise and weight decay (MWN-WD) during training. The other is based on combining injecting additive weight noise and weight decay (AWN-WD) during training. Let m be the number of hidden nodes of a MLP, α be the weight decay constant and Sb be the noise variance. It is showed that the convergence of MWN-WD algorithm is with probability one if α > √Sbm. While the convergence of the AWN-WD algorithm is with probability one if α > 0. {\textcopyright} 2010 IEEE.",

keywords = "Convergence, Learning, MLP, Weight noise",

author = "Kevin Ho and Chi-Sing Leung and John Sum and Siu-Chung Lau",

year = "2010",

doi = "10.1109/TAAI.2010.64",

language = "English",

isbn = "9780769542539",

pages = "358--365",

booktitle = "Proceedings - International Conference on Technologies and Applications of Artificial Intelligence, TAAI 2010",

note = "2010 15th Conference on Technologies and Applications of Artificial Intelligence, TAAI 2010 ; Conference date: 18-11-2010 Through 20-11-2010",

}

Ho, K, Leung, C-S, Sum, J & Lau, S-C 2010, Convergence analysis of multiplicative weight noise injection during training. in Proceedings - International Conference on Technologies and Applications of Artificial Intelligence, TAAI 2010., 5695477, pp. 358-365, 2010 15th Conference on Technologies and Applications of Artificial Intelligence, TAAI 2010, Hsinchu, Taiwan, China, 18/11/10. https://doi.org/10.1109/TAAI.2010.64

Convergence analysis of multiplicative weight noise injection during training. / Ho, Kevin; Leung, Chi-Sing; Sum, John et al.
Proceedings - International Conference on Technologies and Applications of Artificial Intelligence, TAAI 2010. 2010. p. 358-365 5695477.

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

TY - GEN

T1 - Convergence analysis of multiplicative weight noise injection during training

AU - Ho, Kevin

AU - Leung, Chi-Sing

AU - Sum, John

AU - Lau, Siu-Chung

PY - 2010

Y1 - 2010

N2 - Injecting weight noise during training has been proposed for almost two decades as a simple technique to improve fault tolerance and generalization of a multilayer perceptron (MLP). However, little has been done regarding their convergence behaviors. Therefore, we presents in this paper the convergence proofs of two of these algorithms for MLPs. One is based on combining injecting multiplicative weight noise and weight decay (MWN-WD) during training. The other is based on combining injecting additive weight noise and weight decay (AWN-WD) during training. Let m be the number of hidden nodes of a MLP, α be the weight decay constant and Sb be the noise variance. It is showed that the convergence of MWN-WD algorithm is with probability one if α > √Sbm. While the convergence of the AWN-WD algorithm is with probability one if α > 0. © 2010 IEEE.

AB - Injecting weight noise during training has been proposed for almost two decades as a simple technique to improve fault tolerance and generalization of a multilayer perceptron (MLP). However, little has been done regarding their convergence behaviors. Therefore, we presents in this paper the convergence proofs of two of these algorithms for MLPs. One is based on combining injecting multiplicative weight noise and weight decay (MWN-WD) during training. The other is based on combining injecting additive weight noise and weight decay (AWN-WD) during training. Let m be the number of hidden nodes of a MLP, α be the weight decay constant and Sb be the noise variance. It is showed that the convergence of MWN-WD algorithm is with probability one if α > √Sbm. While the convergence of the AWN-WD algorithm is with probability one if α > 0. © 2010 IEEE.

KW - Convergence

KW - Learning

KW - MLP

KW - Weight noise

UR - https://www.scopus.com/pages/publications/79951730809

UR - https://www.scopus.com/record/pubmetrics.uri?eid=2-s2.0-79951730809&origin=recordpage

U2 - 10.1109/TAAI.2010.64

DO - 10.1109/TAAI.2010.64

M3 - RGC 32 - Refereed conference paper (with host publication)

SN - 9780769542539

SP - 358

EP - 365

BT - Proceedings - International Conference on Technologies and Applications of Artificial Intelligence, TAAI 2010

T2 - 2010 15th Conference on Technologies and Applications of Artificial Intelligence, TAAI 2010

Y2 - 18 November 2010 through 20 November 2010

ER -

Convergence analysis of multiplicative weight noise injection during training

Abstract

Conference

Research Keywords

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

Cite this