Two Investigations of Analog Neural Networks: Imperfect Implementation and Non-differential Optimization

兩項關於模擬神經網絡的研究:不完全的硬件實現和不可導的優化問題

Student thesis: Doctoral Thesis

View graph of relations

Author(s)

  • Ruibin FENG

Related Research Unit(s)

Detail(s)

Awarding Institution
Supervisors/Advisors
Award date19 Jan 2017

Abstract

In this thesis, analog neural networks are explored. As described by many neural network scholars, the analog neural network approach is more effective than digital algorithms, particularly when real-time solutions are required. However, there are two critical problems with this approach: the undesirable behaviors under imperfect situations and the optimization incapability for non-differential objective functions. For the first problem, the dual neural network (DNN)-based k-winner-take-all (kWTA) model was studied, which is an effective approach to finding the k largest inputs from n inputs. Two imperfect situations were investigated. First, the transfer function of threshold logic units (TLUs) was modelled as a logistic function. Second, the effects of uniform input noise and Gaussian input noise on the DNN-kWTA network were analyzed. These two imperfect implementations prove that, given any initial state, the network stabilizes at one equilibrium point. The energy functions of the models were also revealed. Based on the energy functions, formulae were derived to determine whether the network produces correct outputs. Furthermore, the lower boundaries of the probability that the network produces the correct outputs are presented for uniformly distributed inputs. When the minimum separation amongst inputs was provided, the conditions for the network to produce correct outputs were derived. Finally, experimental results are presented to verify the theoretical results.

Second, two problems with non-differential objective functions were considered: the center selection problem of fault-tolerant radial basis function (RBF) networks and the sparse approximation problem. One of the most important issues during the construction of an RBF network is the selection of RBF centers. However, many selection methods are designed for the fault-free situation only. It was assumed that all of the training samples were used to construct a fault-tolerant RBF network and an l1 norm regularizer was added to the fault tolerant objective function. According to the nature of the l1 norm regularizer, some unnecessary RBF nodes were removed automatically during training. An analog method called fault tolerant LCA (FTLCA) was proposed based on the local competition algorithm (LCA) concept to minimize the fault tolerant objective function. This paper also demonstrates that the proposed fault tolerant objective function has a unique, optimal solution and that the FTLCA converges to the global optimal solution. Simulation results show that the FTLCA is better than the orthogonal least square approach and the support vector regression approach.

Another non-differential problem that is discussed in this thesis is the sparse approximation problem, which is formulated as a constrained optimization problem in most cases. The Lagrange programming neural network (LPNN) was applied to solve this problem. However, since the major limitation of the LPNN approach is that the objective and the constraint functions should be twice differentiable, the original LPNN approach is not suitable for recovering sparse signals. Therefore, this thesis proposes a new formulation of the LPNN approach based on the concept of the LCA. Unlike the classical LCA approach, which can only solve unconstrained optimization problems, the proposed LPNN approach can solve constrained optimization problems. This thesis shows that the equilibrium points of the proposed approach are the optimal solutions for the original constrained sparse approximation problem, and that the optimal solutions of the problem are the equilibrium points of our model. In additional, the equilibrium points are stable and simulations were conducted to verify the effectiveness of our LPNN model.