Co-clustering of Deep Features for Facial Expression Analysis

Gabor wavelets and other pre-defined features have been widely used for human facial expression analysis. These features have a high level of redundancy, and several methods can be used to select features for classifier training. The PI’s team has proposed a radically new method for feature selection based on co-clustering, which can detect coherent patterns consisting of subsets of samples and subsets of features. With as low as 0.9% of Gabor wavelet features selected, we were able to obtain a facial expression recognition accuracy even higher than that with the full set of features used. Deep neural networks (DNNs) have superior performance to traditional classifiers that use pre-defined features for facial expression recognition and many other pattern classification tasks. However, a DNN is often viewed as a black box, where we only deal with the input samples and output results, and the features and their properties inside the black box are largely hidden. In this project, we open the black box and analyze the deep features using co-clustering. Our preliminary work shows the middle layers in a DNN produce interesting co-cluster patterns. We will investigate these coherent patterns for facial expressions from images and video data. The co-clusters reveal simultaneous and consistent structures both spatially and temporally. They will be employed for facial expression and microexpression generation and classification. We will develop techniques to simplify deep neural networks by removing the weights that do not contribute to the selected features. Our method combines the classification power of deep neural networks and feature selection power of co-clustering. In addition to providing effective techniques for facial expression analysis, our work will lead to a deep understanding of the hidden layers in deep neural networks. The co-clusters can reveal the coherent patterns produced inside the black box and can help us build more powerful networks in terms of computational speed and accuracy. Robust facial expression analysis techniques will have many applications to medical diagnosis, human-machine interactions, sports, education and training, video gaming, and entertainment. Although our work focuses on facial expressions in this project, the research results can be employed in a wide range of other applications. Also, our work will help offer explainability of deep neural networks. The black box will no longer be mysterious, and this will open the door for us to develop a new generation of efficient intelligent data analysis systems for many practical applications.