Learning Theory of Deep Structured Neural Networks

Deep learning algorithms using deep neural networks have brought us technological breakthroughs in many practical domains including speech recognition and computer vision. This has raised a strong demand of learning theory of deep neural networks, especially those with special structures, as a theoretical foundation of deep learning. In this project we plan to conduct rigorous mathematical analysis and establish a learning theory for some deep structured neural networks. First we propose to express the output function of a fully connected multi-layer neural network as that of a downsampled deep convolutional neural network, which would show that such deep structured neural networks have at least as good approximation abilities as the classical fully connected networks. Here the operation of downsampling in wavelet analysis plays a key role. Next we plan to show some advantages of deep structured neural networks by analyzing the problem of deconvolution for regression with convoluted inputs. The effect of filter length for approximation by deep convolutional neural networks and some other related approximation theory problems will also be studied. Then we aim at deriving error bounds and learning rates of learning with deep structured neural networks in terms of the capacity of the hypothesis space for batch learning algorithms and step sizes for stochastic gradient descent type online learning algorithms. For this complexity analysis part, some approaches from multivariate approximation theory and wavelets are essential.