Learning Algorithms and Some Neural Network Models Under Imperfect Implementations


Student thesis: Doctoral Thesis

View graph of relations

Related Research Unit(s)


Awarding Institution
Award date14 Sept 2021


Function approximation is an important application of artificial neural networks(ANNs). In areas such as regression problems, image processing, and computer vision, function approximation is a commonly used technique. In function approximation, one of the common issues is the reliability of the estimated model. Besides, in conventional neural network training algorithms, whenever an estimated model is implemented on hardware such as field-programmable gate array (FPGA), the performance of the estimated model embedded in a hardware system is different from the one obtained after learning. Also, in hardware implementation of neural networks faults are unavoidable. The faults can render a well-trained neural network wasteful or valueless.

To address the problem, this thesis investigates some neural network models and algorithms under imperfect situations, where well-trained neural networks are affected by faults/noise, such as weight noise, weight fault, and node noise or the combination of these failures. We developed several learning algorithms to handle those imperfect conditions.

Extreme learning machine (ELM) is an efficient technique for constructing single layer feed-forward neural networks (SLFFNs). We explore and investigate single-layer feed-forward neural networks (SLFFNs) under imperfect conditions. A noise-tolerant objective function that can handle multiplicative weight noise is developed. Based on the developed objective function, we propose two noise-tolerant incremental extreme learning machine algorithms, namely weight deviation incremental extreme learning machine (WDT-IELM) and weight deviation convex incremental extreme learning machine (WDTC-IELM). Compared to the original extreme learning machine algorithms, the two proposed algorithms are much better able to tolerate the multiplicative weight noise.

Furthermore, in real-life applications, two or more faults can co-exist on a neural network. For instance, impulsive gaussian noise/outlier in the training samples and fault/noise can concurrently exist on a neural network. These kinds of failures may affect the neural network's performance greatly. In this case, a single fault-based learning algorithm cannot handle such a situation. In this thesis, to accommodate the aforementioned issue, a robust fault-aware ELM algorithm is proposed. We use the maximum correntropy criterion (MCC) to handle training data sets with outliers. In order to compensate for the effect of multiplicative noise in the input weights and the output weights of the ELM network, a fault-aware regularizer term is added to the objective function. Furthermore, due to the non-concave nature (or saying non-convex from another perspective) of the derived objective function, we employ some theories of convex conjugated functions to simplify and modify the function. Hence, a robust fault-aware ELM algorithm (MCC-FAELM) is developed. Several simulation results demonstrate the effectiveness of the proposed MCC-FAELM algorithm.

Lastly, we explore and study a broad layer neural network under imperfect conditions. Broad learning system (BLS) gives an approach to construct a broad layer neural network and its learning algorithm. First, we develop a fault-aware objective function for BLS, it contains a regularizer term that is able to handle the fault/noise in the trained network. Based on the developed objective function, a fault-aware BLS algorithm for the coexistence of network failure situations is proposed. It is designed to handle the coexistence of four kinds of fault/noise. They are open weight fault in output weights, multiplicative noise in output weights, multiplicative noise in feature nodes, and multiplicative noise in input weights of enhancement nodes of a BLS network. In this thesis, our approach is called fault-aware BLS (FABLS). Several numerical results prove that the proposed FABLS outperformed the classical BLS and two other state-of-the-art BLS algorithms in terms of mean square error(MSEs).