A Novel Theoretical Framework for Analyzing Error Dynamics in Deep Learning

Machine learning, particularly through the use of deep neural networks (DNNs), hasgarnered significant attention due to its remarkable performance and successfulapplications across diverse fields, including life sciences, social sciences, and engineering.Despite these practical successes, our theoretical understanding of DNN properties isstill evolving.Initially, DNNs were often viewed as "black boxes," achieving input-output data fittingwithout clarity on underlying mechanisms. This opacity has led to challenges ininterpretability and performance guarantees, including issues related to transferability,accuracy, generalizability, and robustness. In recent years, various interpretable DNNmodels have been proposed, driven by new theories and insights. However, theseapproaches are primarily assessed through numerical experiments, which often lackrigorous error analysis and convergence guarantees. The central questions arise: 1) Howcan we guarantee a given level of accuracy based on the scale of networks and samplesize? 2) What factors influence the occurrence of errors? The existing theoreticalframeworks for deep learning are indeed sophisticated, yet they often treat neuralnetworks simply as large parameterized models. Error estimation is typically performedin a separate, coarse-grained manner. The insights derived from these analyticalframeworks often fail to address the complexities associated with high dimensionalityand overparameterization, which are prevalent in deep learning and contradict manypractical observations. As a result, these studies do not adequately capture the intricaciesof large-scale deep neural network architectures or the complexities of trainingprocesses that involve various optimization techniques. To bridge this gap and enhancethe applicability of theoretical insights in real-world scenarios, a more integratedapproach is required.In this proposal, we aim to develop a comprehensive and specific theoretical frameworkfor deep learning that emphasizes a deeper understanding of the interplay betweenapproximation ability, training processes, and generalization for large-scale deep neuralnetwork models. We will introduce a new mathematical framework for error analysisthat encompasses approximation error, optimization training error, and generalizationgap, as well as the interactions among these aspects. Our innovation lies in connectingdeep neural networks with minimization over features and dictionaries, treating themnot merely as end-to-end parameterized models but as systems influenced by thedynamics of forward and backward propagation. The anticipated results will be benefitfor both the theoretical analysis community and practitioners in industry. Furthermore,the insights gained may guide the development of more efficient and accurate networkdesigns and algorithms.