Kernel-based algorithms in statistical learning theory

Abstract

Statistical learning theory plays an important role in some subjects of various areas such as science, engineering, and finance. As a research field, it provides theoretical foundations for algorithms in machine learning. Generally speaking, statistical learning aims at learning function features or data structures from observations. By making use of kernel methods, data can be mapped into a high dimensional feature space, in which various methods can be employed to find relations. In this thesis, we mainly focus on several different kernel-based learning algorithms in the framework of statistical learning theory. Firstly, we study the q-norm regularized least-squares regression with dependent samples. We conduct error analysis of the least-squares regularized regression algorithm when the sampling sequence is weakly dependent satisfying an exponentially decaying α-mixing condition and when the regularizer takes the q-penalty with 0 < q ≤ 2. We use a covering number argument and derive learning rates in terms of the α-mixing decay, an approximation condition, and the capacity of balls of the reproducing kernel Hilbert space. Secondly, we concentrate on the coefficient-based regularized regression problem. The lq-regularized least-squares regression problem with 1 ≤ q ≤ 2 and data dependent hypothesis spaces is addressed. Algorithms in data dependent hypothesis spaces perform well with the property of flexibility. We conduct a unified error analysis by a stepping stone technique. An empirical covering number technique is also employed in our study to improve the sample error. Compared with existing results, we make a few improvements: First, we obtain a significantly sharper learning rate of type O(m−θ) with θ arbitrarily close to 1 under reasonable conditions, which is regarded as the best learning rate in learning theory. Second, our results cover the case q = 1, which is novel. Finally, our results hold under very general conditions. Finally, we address the pairwise ranking problem via a kernel-based learning approach. Various settings for the pair-wise ranking problem are compared. We adopt a preference-based two-stage setting while the empirical data is generated in a different manner. For the first learning stage, we learn a preference function by reducing ranking to classification. Learning results concerning the learnability of the ranking rule we learned are presented as in classification. For the second stage, we present an optimization algorithm to produce a scoring function that might be used to yield an ordering.

Date of Award	3 Oct 2012
Original language	English
Awarding Institution	City University of Hong Kong
Supervisor	Dingxuan ZHOU (Supervisor)