Explore Energy Landscape of Deep Learning to Understand Generalization Error

Deep learning has been a great practical success and has a profound impact on the machine learning. Understanding theoretic properties of deep learning is an active research area. In contrast to shallow networks, deep learning is equipped with much more effective parameters than the number of training examples. The loss function of the deep learning quickly becomes very high-dimensional and non-convex. The existence of a huge number of local minima makes the optimization task conceptually challenging. The recent research shows the major optimization difficulty may originate from the proliferation of saddle points and suggests there is a hierarchical structure of the critical points with the various index on the energy landscape (i.e., loss function surface) in deep learning.However, despite the complicated energy landscape and the optimization challenges, many of deep networks with learned locally optimal parameters from stochastic gradient descent exhibit remarkably small generalization error, i.e., the difference between training error and test error. The traditional statistical learning theory on complexity has intrinsic difficulty to explain why this high-capacity network can still obtain low complexity solutions and generalize well. One popular hypothesis is that the characteristic of the landscape of the loss function is one of the important elements to understand the optimization algorithm performance and to explain the good generalization capability in deep learning.This research plan is to conduct substantial investigations of this energy landscape perspective and aims to study the energy landscape of deep learning in an innovative way to gain a better understanding of the geometry of loss surface and its potential connection to the remarkable generalization performance. Our main methodology is based on the previous work of the PI on the rare-event study of random transitions on energy landscape arising from molecular system and energy functional in material sciences. The existing work has built a solid connection between the stochastic gradient descent on the empirical loss function and the modified stochastic differential equation, which is the same equation describing rare events. We propose to use the key concept, energy barrier, in rare-event study as the candidate to measure generalization capability of local minima in deep learning with well-justified reasons. The toolset previously used to explore energy landscape in physics, such as the numerical methods for index-1 saddle points and the minimum energy path, will be carefully modified and applied for the new purpose of deep learning.