Analysis of Recurrent Neural Networks under Imperfect Conditions
對於有缺陷的遞歸神經網絡的分析
Student thesis: Doctoral Thesis
Author(s)
Related Research Unit(s)
Detail(s)
Awarding Institution  

Supervisors/Advisors 

Award date  10 Jan 2022 
Link(s)
Permanent Link  https://scholars.cityu.edu.hk/en/theses/theses(027dbe3ce99b48b1b7d9791ea1176c2f).html 

Other link(s)  Links 
Abstract
The dual neural networkbased kwinnertakeall (DNNkWTA) model, the Boltzmann machine (BM), and the restricted Boltzmann machine (RBM) are representative recurrent neural models. They have many applications, including face and biometric recognition, rankorder filtering, and classification. However, when we implement them in analog circuits, there are many imperfections. These imperfections lead to the networks operating incorrectly and greatly degrading their performance. This thesis investigates the performance degradation of these three models when imperfections exist.
The DNNkWTA model is an analog neural network that can find the k largest numbers from n inputs. Two imperfect situations are discussed. First, we consider that there are timeinvariant and timevarying offset voltage drifts in the inputoutput (I/O) neurons, and only the drift range is available. For the timeinvariant case, we show that the network state converges to a unique equilibrium point. A sufficient condition of generating correct outputs is given. Furthermore, a lower bound on the probability that the network works properly is presented for the inputs with uniform distribution. The aforementioned results are also extended to the timevarying case. Besides, given the sets of inputs, we provide a way to compute the exact convergence time of the DNNkWTA model with timeinvariant drifts. When the inputs are uniformly distributed, the mean and variance of the convergence time are further derived. Simulations are performed to verify our theoretical results.
Second, we consider that the activation function of the I/O neuron is a logistic function rather than an ideal step function, and input noise exists in the I/O neurons. These two imperfections coexist in the network realization. A set of equivalent equations are built to describe the dynamics of the imperfect DNNkWTA model. The aim of introducing equivalent dynamics is to help us to analyze the behavior of the DNNkWTA under the two mentioned imperfections. From the equivalent dynamics, the sufficient and necessary conditions of operational correctness are derived. These conditions provide an efficient way to determine whether the imperfect model can identify correct winners and losers for the given set of inputs without simulating the timeconsuming neural dynamics. They can also be used to estimate the chance of the imperfect model producing correct outputs. Furthermore, a lower bound on the probability that the imperfect model works properly is presented for the uniformly distributed inputs. Finally, our analysis method is extended to the nonGaussian input noise case. Experiments are conducted to validate our theoretical results.
BMs and RBMs are stochastic neural networks in which each neuron state is governed by a stochastic activation function. We consider that there is input noise in the neurons of BMs or RBMs. The behaviors of BMs and RBMs under this imperfect condition are analyzed. We find that the effect of additive input noise is the same as elevating the temperature factor of the activation function in the BM and RBM models. Since the input noise may lead to the wrong neurons’ stochastic
behaviors, there exists Kullback Leibler (KL) divergence loss in noisy BMs and noisy RBMs. With the Gaussiandistributed noise assumption, we propose a compensation method to suppress the effect of the noise. Experimental results present that with our method, the KL divergence loss is greatly decreased. Besides, our method also works for nonGaussian noise cases.
For the RBM model, another imperfect situation is that there is additive weight noise during operation. The effect of noise on the behavior of RBM is analyzed. We present that the additive weight noise raises the temperature factors of the stochastic activation functions. Also, the raised temperature factor in the visible layer is different from that in the hidden layer. Thus, the state distribution of a noisy RBM does not follow the Boltzmann distribution. We make the state distribution be Boltzmanndistributed again by equalizing the raised temperature factors in the visible and hidden layer. Since the effective temperatures of the noisy RBM and the noisefree RBM are different, some KL divergence loss exists in the state distribution of noisy RBM. Therefore, we provide a compensation method to suppress the effect of noise. Experiments show that with our noise compensation scheme, the performance of noisy RBM is effectively improved.
The DNNkWTA model is an analog neural network that can find the k largest numbers from n inputs. Two imperfect situations are discussed. First, we consider that there are timeinvariant and timevarying offset voltage drifts in the inputoutput (I/O) neurons, and only the drift range is available. For the timeinvariant case, we show that the network state converges to a unique equilibrium point. A sufficient condition of generating correct outputs is given. Furthermore, a lower bound on the probability that the network works properly is presented for the inputs with uniform distribution. The aforementioned results are also extended to the timevarying case. Besides, given the sets of inputs, we provide a way to compute the exact convergence time of the DNNkWTA model with timeinvariant drifts. When the inputs are uniformly distributed, the mean and variance of the convergence time are further derived. Simulations are performed to verify our theoretical results.
Second, we consider that the activation function of the I/O neuron is a logistic function rather than an ideal step function, and input noise exists in the I/O neurons. These two imperfections coexist in the network realization. A set of equivalent equations are built to describe the dynamics of the imperfect DNNkWTA model. The aim of introducing equivalent dynamics is to help us to analyze the behavior of the DNNkWTA under the two mentioned imperfections. From the equivalent dynamics, the sufficient and necessary conditions of operational correctness are derived. These conditions provide an efficient way to determine whether the imperfect model can identify correct winners and losers for the given set of inputs without simulating the timeconsuming neural dynamics. They can also be used to estimate the chance of the imperfect model producing correct outputs. Furthermore, a lower bound on the probability that the imperfect model works properly is presented for the uniformly distributed inputs. Finally, our analysis method is extended to the nonGaussian input noise case. Experiments are conducted to validate our theoretical results.
BMs and RBMs are stochastic neural networks in which each neuron state is governed by a stochastic activation function. We consider that there is input noise in the neurons of BMs or RBMs. The behaviors of BMs and RBMs under this imperfect condition are analyzed. We find that the effect of additive input noise is the same as elevating the temperature factor of the activation function in the BM and RBM models. Since the input noise may lead to the wrong neurons’ stochastic
behaviors, there exists Kullback Leibler (KL) divergence loss in noisy BMs and noisy RBMs. With the Gaussiandistributed noise assumption, we propose a compensation method to suppress the effect of the noise. Experimental results present that with our method, the KL divergence loss is greatly decreased. Besides, our method also works for nonGaussian noise cases.
For the RBM model, another imperfect situation is that there is additive weight noise during operation. The effect of noise on the behavior of RBM is analyzed. We present that the additive weight noise raises the temperature factors of the stochastic activation functions. Also, the raised temperature factor in the visible layer is different from that in the hidden layer. Thus, the state distribution of a noisy RBM does not follow the Boltzmann distribution. We make the state distribution be Boltzmanndistributed again by equalizing the raised temperature factors in the visible and hidden layer. Since the effective temperatures of the noisy RBM and the noisefree RBM are different, some KL divergence loss exists in the state distribution of noisy RBM. Therefore, we provide a compensation method to suppress the effect of noise. Experiments show that with our noise compensation scheme, the performance of noisy RBM is effectively improved.