Understanding personality of portrait by social embedding visual features

Image understanding achieves unprecedented performance in content recognition and emotion rating recently. However, previous research mainly focused on visual features. In this paper, inspired by human cognitive activities, we discuss how to measure portrait distance in respective of high level semantic: personality, by employing features from both visual contents and behavior contents. Firstly, a new image distance metric, named Social and Visual Portrait Distance, is designed by jointly considering visual features and human social media behavior features. In portrait images, visual features are defined globally and locally utilizing theoretical and empirical concepts from psychology theory. While in social media, behavior features are designed with considerations of demographic factors, identity claims and behavioral residue. And the new distance is estimated referred to feature reliability. Secondly, we modify the proposed distance calculation formula in order to apply it on only visual features but preserving both visual and social relations by a new Social Embedding Portrait Distance Learning method. In this manner, we could measure the social embedding visual distance of common portrait images in absence of social media information, such as web portrait, or daily photos. Comprehensive experiments are employed to investigate the effectiveness of the new portrait distance and the metric learning method in representing personality distance compared with several baselines. Moreover, the learned distance matrix reveals a reasonable explanation of social preferred visual features with contribute partitions.