Improve L2-normalized Softmax with Exponential Moving Average

In this paper, we propose an effective training method to improve the performance of L2-normalized softmax for convolutional neural networks. Recent studies of deep learning show that by L2-normalizing the input features of softmax, the accuracy of CNN can be increased. Several works proposed novel loss functions based on the L2-normalized softmax. A common property shared by these modified normalized softmax models is that an extra set of parameters is introduced as the class centers. Although the physical meaning of this parameter is clear, few attentions have been paid to how to learn these class centers, which limits futher improvement.
In this paper, we address the problem of learning the class centers in the L2-normalized softmax. By treating the CNN training process as a time series, we propose a novel learning algorithm that combines the generally used gradient descent with the exponential moving average. Extensive experiments show that our model not only achieves better performance but also has a higher tolerance to the imbalance data.