The dual neural network-based k-winner-take-all (DNN-kWTA) model, the Boltzmann machine (BM), and the restricted Boltzmann machine (RBM) are representative recurrent neural models. They have many applications, including face and biometric recognition, rank-order filtering, and classification. However, when we implement them in analog circuits, there are many imperfections. These imperfections lead to the networks operating incorrectly and greatly degrading their performance. This thesis investigates the performance degradation of these three models when imperfections exist.

The DNN-kWTA model is an analog neural network that can find the k largest numbers from n inputs. Two imperfect situations are discussed. First, we consider that there are time-invariant and time-varying offset voltage drifts in the input-output (I/O) neurons, and only the drift range is available. For the time-invariant case, we show that the network state converges to a unique equilibrium point. A sufficient condition of generating correct outputs is given. Furthermore, a lower bound on the probability that the network works properly is presented for the inputs with uniform distribution. The aforementioned results are also extended to the time-varying case. Besides, given the sets of inputs, we provide a way to compute the exact convergence time of the DNN-kWTA model with time-invariant drifts. When the inputs are uniformly distributed, the mean and variance of the convergence time are further derived. Simulations are performed to verify our theoretical results.

Second, we consider that the activation function of the I/O neuron is a logistic function rather than an ideal step function, and input noise exists in the I/O neurons. These two imperfections coexist in the network realization. A set of equivalent equations are built to describe the dynamics of the imperfect DNN-kWTA model. The aim of introducing equivalent dynamics is to help us to analyze the behavior of the DNN-kWTA under the two mentioned imperfections. From the equivalent dynamics, the sufficient and necessary conditions of operational correctness are derived. These conditions provide an efficient way to determine whether the imperfect model can identify correct winners and losers for the given set of inputs without simulating the time-consuming neural dynamics. They can also be used to estimate the chance of the imperfect model producing correct outputs. Furthermore, a lower bound on the probability that the imperfect model works properly is presented for the uniformly distributed inputs. Finally, our analysis method is extended to the non-Gaussian input noise case. Experiments are conducted to validate our theoretical results.

BMs and RBMs are stochastic neural networks in which each neuron state is governed by a stochastic activation function. We consider that there is input noise in the neurons of BMs or RBMs. The behaviors of BMs and RBMs under this imperfect condition are analyzed. We find that the effect of additive input noise is the same as elevating the temperature factor of the activation function in the BM and RBM models. Since the input noise may lead to the wrong neurons’ stochastic
behaviors, there exists Kullback Leibler (KL) divergence loss in noisy BMs and noisy RBMs. With the Gaussian-distributed noise assumption, we propose a compensation method to suppress the effect of the noise. Experimental results present that with our method, the KL divergence loss is greatly decreased. Besides, our method also works for non-Gaussian noise cases.

For the RBM model, another imperfect situation is that there is additive weight noise during operation. The effect of noise on the behavior of RBM is analyzed. We present that the additive weight noise raises the temperature factors of the stochastic activation functions. Also, the raised temperature factor in the visible layer is different from that in the hidden layer. Thus, the state distribution of a noisy RBM does not follow the Boltzmann distribution. We make the state distribution be Boltzmann-distributed again by equalizing the raised temperature factors in the visible and hidden layer. Since the effective temperatures of the noisy RBM and the noise-free RBM are different, some KL divergence loss exists in the state distribution of noisy RBM. Therefore, we provide a compensation method to suppress the effect of noise. Experiments show that with our noise compensation scheme, the performance of noisy RBM is effectively improved.