Online training and its convergence for faulty networks with multiplicative weight noise

A recent article showed that the objective function of the online weight noise injection algorithm is not equal to the training set error of faulty radial basis function (RBF) networks under the weight noise situation (Ho et al., 2010). Hence the online weight noise injection algorithm is not able to optimize the training set error of faulty networks with multiplicative weight noise. This paper proposes an online learning algorithm to tolerate multiplicative weight noise. Two learning rate cases, namely fixed learning rate and adaptive learning rate, are investigated. For the fixed learning rate case, we show that if the learning rate μ is less than 2/(σb2+maxi{norm of matrix}ϕ(xi)|2), then the online algorithm converges, where x_i's are the training input vectors, σ²_b is the variance of the multiplicative weight noise, ϕ(x_i)=[ϕ₁(x_i), ., ϕ_M(x_i)]^T, and ϕ_j(·) is the output of the j-th RBF node. In addition, as μ→0, the trained weight vector tends to the optimal solution. For the adaptive learning rate case, let the learning rates {μ_k} be a decreasing sequence and lim_k→∞μ_k=0, where k is the index of learning cycles. We prove that if ∑k=1∞μk=∞ and ∑k=1∞μk2