TY - CHAP
T1 - Analysis on Dropout Regularization
AU - Sum, John
AU - Leung, Chi-Sing
PY - 2019
Y1 - 2019
N2 - Dropout, including Bernoulli dropout (equivalently random node fault) and multiplicative Gaussian noise (MGN) dropout (equivalently multiplicative node noise), has been a technique in training a neural network (NN) to achieve better performance. While simulation results have demonstrated its success, not many work has been done to explain why it works (or why it does not work). In this paper, the objective functions L (w) of the learning algorithms with Bernoulli dropout and MGN dropout are derived and thus their regularization effects are analyzed. It is found that learning with Bernoulli dropout cannot improve the generalization of a NN if its weights are not scaled down after training. If we further let J (w) be the desired measure of a NN with such inherent dropout, we clarify a misconception that L (w) = J (w). The model attained by learning with dropout is not the desired model that can tolerate such inherent dropout.
AB - Dropout, including Bernoulli dropout (equivalently random node fault) and multiplicative Gaussian noise (MGN) dropout (equivalently multiplicative node noise), has been a technique in training a neural network (NN) to achieve better performance. While simulation results have demonstrated its success, not many work has been done to explain why it works (or why it does not work). In this paper, the objective functions L (w) of the learning algorithms with Bernoulli dropout and MGN dropout are derived and thus their regularization effects are analyzed. It is found that learning with Bernoulli dropout cannot improve the generalization of a NN if its weights are not scaled down after training. If we further let J (w) be the desired measure of a NN with such inherent dropout, we clarify a misconception that L (w) = J (w). The model attained by learning with dropout is not the desired model that can tolerate such inherent dropout.
KW - Dropout
KW - Multiplicative node noise
KW - Random node fault
UR - https://www.scopus.com/pages/publications/85078411781
UR - https://www.scopus.com/record/pubmetrics.uri?eid=2-s2.0-85078411781&origin=recordpage
U2 - 10.1007/978-3-030-36802-9_28
DO - 10.1007/978-3-030-36802-9_28
M3 - RGC 12 - Chapter in an edited book (Author)
SN - 9783030368012
T3 - Communications in Computer and Information Science
SP - 253
EP - 261
BT - Neural Information Processing - 26th International Conference, ICONIP 2019, Proceedings
A2 - Gedeon, Tom
A2 - Wong, Kok Wai
A2 - Lee, Minho
PB - Springer
T2 - 26th International Conference on Neural Information Processing (ICONIP 2019)
Y2 - 12 December 2019 through 15 December 2019
ER -