4D facial expression recognition using multimodal time series analysis of geometric landmark-based deformations

One of the main challenges in dynamic facial expression recognition is how to capture temporal information in the system. In this study, a novel approach based on time series analysis is adapted for this problem. The proposed dynamic facial expression recognition system comprises four phases: head pose correction and normalization, feature extraction, feature selection and classification. Head pose detection and correction is the first phase to realign locations of the facial landmarks. A comprehensive set of geometric deformations including point, distance and angle deformations are extracted from the key points. The concept of facial action unit analysis is interlocked with this phase to identify related key points from the landmarks. A set of multimodal time series are then constructed from the extracted deformations by applying a sliding window to characterize the dynamics of mean deformations in a window. In the third phase, a feature selection method based on neighborhood component analysis is applied on the peak value of deformation features to select useful features and discard irrelevant ones. Finally, adaptive cost dynamic time warping is utilized to recognize six prototypic expressions from multimodal time series of selected features. Experimental results on BU-4DFE data set confirm that proposed algorithm is efficient in dynamic facial expression recognition compared with state of the art.