Approximation Theory of Structured Deep Nets and Learning

Description

Deep learning has been developed very successfully as a super effective scheme forhandling big data practically. It is implemented by deep neural networks or deep netswhich often have special structures instead of being fully connected. But there is littlework on approximation properties of structured deep nets which is comparable to theclassical approximation theory for fully connected neural networks. This project aims ata rigorous approximation theory for some topics of structured deep nets and deeplearning. First we plan to establish an approximation theory for convolutional deep netsassociated with the rectified linear unit. Expected results include estimates for thecomplexity of the function space spanned by the components at the last level of hiddenunits in terms of the convolution kernels, rates of function approximation by thegenerated output functions, and analysis for the role of pooling in convolutional deepnets and deep learning. Then we plan to carry out wavelet analysis of learning with deepnets: estimating covering numbers of the involved hypothesis spaces to handle theredundancy of distributed representations of deep nets, and conducting error analysis forlearning with deep nets of special structures. Finally some stochastic composite mirrordescent algorithms including online mirror descent with differentiable mirror maps andcomposite mirror descent with non-differentiable mirror maps will be studied byanalyzing the induced Bregman distances, which will help the theoretical understandingof recurrent deep nets in deep learning.