Optimization Algorithm Development for Nonlinear Problem and Its Application

Abstract

This thesis addresses the following four nonlinear optimization problems.

Online Learning Algorithm for Fault-Tolerant Neural Networks
Some existing results claim that the fault injection approach is able to minimize the mean square error of faulty radial basis function (RBF) networks. This claim is not correct.

We propose a general online learning algorithm to handle the concurrent fault situation, where weight noise, node noise, weight fault and node fault may occur simultaneously in an RBF network. Two kinds of learning rates are considered. They are fixed learning rate and adaptive learning rate. In addition, this thesis provides the convergent conditions for the proposed online learning algorithm.

Center Selection Algorithms for Fault-Tolerant Neural Networks
In training an RBF network, selecting RBF centers is very important. However, many existing center selection algorithms focus on the fault-free situation only.

We propose an algorithm which is able to train and to select the RBF centers simultaneously. In our algorithm, an l₁-norm regularizer is added to the objective function of the fault-tolerant networks. The alternating direction method of multipliers (ADMM) framework is applied to minimize the modified objective function. Besides, the optimality and the convergent performance of the proposed algorithm are analyzed.

Simulation results demonstrate that the proposed center selection algorithm can achieve better results than the orthogonal least squares (OLS) algorithm. To further improve the performance, a new center selection algorithm is proposed by introducing an l₀-norm constraint. With an indicator function of the l₀-norm constraint, the constrained optimization problem can be converted to an unconstrained one which can then be solved effectively with the ADMM framework. The convergent analysis of the proposed algorithm is also provided. Besides, the results show that the proposed center selection algorithm with an l₀-norm constraint outperforms the center selection algorithm with an l₁-norm regularizer.

Tensor Completion Algorithm based on Sparse and Truncated Nuclear Norm
In tensor completion, the calculation of the tensor rank is one of the main challenges as it involves minimizing the singular values at the same time. Hence the tensor rank cannot be well approximated. In addition, the structure information of the tensor data is usually ignored in many existing algorithms.

To address these issues, a new tensor completion algorithm is proposed. In this algorithm, the tensor truncated nuclear norm (TTNN) is defined on the basis of the matrix truncated nuclear norm (MTNN). In addition, a sparse regularization term is introduced to maintain the structure information of the tensor data. Specifically, the sparse regularization term is defined as the l₁-norm of the multidimensional discrete cosine transform (DCT) coefficients of the tensor data so that the piece-wise smooth property of the tensor data can be preserved effectively. The ADMM framework is then adapted to solve this nonlinear optimization problem. Experimental results show that the proposed algorithm outperforms several state-of-the-art tensor completion schemes.

Localization Algorithm via Lagrange Programming Neural Network
Time difference ofarrival (TDOA)measurements are usually affected by Gaussian-distributed noises. A common method for solving the TDOA based source localization problem is to maximize the maximum likelihood (ML) cost function. However, the highly nonlinear property of the unconstrained objective function may cause some problems if we directly solve this optimization problem with a traditional gradient descent (GD) method. Furthermore, under poor initialization or bad geometry conditions, the ML estimator might fail to solve this TDOA based localization problem.

The Lagrange programming neural network (LPNN) approach provides a framework for solving highly nonlinear constrained optimization problems. Furthermore, with the introduction of an augmented term, the LPNN is able to solve some nonlinear constrained optimization problems with nonconvex objective functions. In this thesis, we apply the LPNN framework to tackle the TDOA based localization problem. Firstly, the ML optimization problem for the TDOA based localization is converted to a constrained optimization problem by introducing some dummy variables. An augmented Lagrangian function is then constructed to improve the convexity and the stability of the LPNN. Simulation results show that the proposed framework is superior to some state-of-the-art algorithms and is close to the Cramér-Rao Lower Bound (CRLB).

Date of Award	14 Oct 2016
Original language	English
Awarding Institution	City University of Hong Kong
Supervisor	Chi Sing Andrew LEUNG (Supervisor)