TY - GEN
T1 - Noise on Gradient Systems with Forgetting
AU - Su, Chang
AU - Sum, John
AU - Leung, Chi-Sing
AU - Ho, Kevin I.-J.
PY - 2015
Y1 - 2015
N2 - In this paper, we study the effect of noise on a gradient system with forgetting. The noise include multiplicative noise, additive noise and chaotic noise. For multiplicative or additive noise, the noise is a mean zero Gaussian noise. It is added to the state vector of the system. For chaotic noise, it is added to the gradient vector. Let x be the state vector of a system, Sb be the variance of the Gaussian noise, k’ is average noise level of the chaotic noise, λ is a positive constant, V (x) be the energy function of the original gradient system, V⊗(x), V⊕(x) and V⊙(x) be the energy functions of the gradient systems, if multiplicative, additive and chaotic noises are introduced. Suppose (Formula presented.) It is shown that (Formula presented.), (Formula presented.), and (Formula presented.). The first two results imply thatmultiplicative or additive noise has no effect on the system if F(x) is quadratic. While the third result implies that adding chaotic noise can have no effect on the system if k’ is zero. As many learning algorithms are developed based on the method of gradient descent, these results can be applied in analyzing the effect of noise on those algorithms. © Springer International Publishing Switzerland 2015.
AB - In this paper, we study the effect of noise on a gradient system with forgetting. The noise include multiplicative noise, additive noise and chaotic noise. For multiplicative or additive noise, the noise is a mean zero Gaussian noise. It is added to the state vector of the system. For chaotic noise, it is added to the gradient vector. Let x be the state vector of a system, Sb be the variance of the Gaussian noise, k’ is average noise level of the chaotic noise, λ is a positive constant, V (x) be the energy function of the original gradient system, V⊗(x), V⊕(x) and V⊙(x) be the energy functions of the gradient systems, if multiplicative, additive and chaotic noises are introduced. Suppose (Formula presented.) It is shown that (Formula presented.), (Formula presented.), and (Formula presented.). The first two results imply thatmultiplicative or additive noise has no effect on the system if F(x) is quadratic. While the third result implies that adding chaotic noise can have no effect on the system if k’ is zero. As many learning algorithms are developed based on the method of gradient descent, these results can be applied in analyzing the effect of noise on those algorithms. © Springer International Publishing Switzerland 2015.
UR - http://www.scopus.com/inward/record.url?scp=84952056434&partnerID=8YFLogxK
UR - https://www.scopus.com/record/pubmetrics.uri?eid=2-s2.0-84952056434&origin=recordpage
U2 - 10.1007/978-3-319-26555-1_54
DO - 10.1007/978-3-319-26555-1_54
M3 - RGC 32 - Refereed conference paper (with host publication)
SN - 9783319265544
VL - Part III
T3 - Lecture Notes in Computer Science
SP - 479
EP - 487
BT - Neural Information Processing - 22nd International Conference, ICONIP 2015 - Proceedings
A2 - Sabri Arik, null
A2 - Tingwen Huang, null
A2 - Weng Kin Lai, null
A2 - Qingshan Liu, null
PB - Springer, Cham
T2 - 22nd International Conference on Neural Information Processing (ICONIP 2015)
Y2 - 9 November 2015 through 12 November 2015
ER -