Unsupervised celebrity face naming in web videos

This paper investigates the problem of celebrity face naming in unconstrained videos with user-provided metadata. Instead of relying on accurate face labels for supervised learning, a rich set of relationships automatically derived from video content and knowledge from image domain and social cues is leveraged for unsupervised face labeling. The relationships refer to the appearances of faces under different spatio-temporal contexts and their visual similarities. The knowledge includes Web images weakly tagged with celebrity names and the celebrity social networks. The relationships and knowledge are elegantly encoded using conditional random field (CRF) for label inference. Two versions of face annotation are considered: within-video and between-video face labeling. The former addresses the problem of incomplete and noisy labels in metadata, where null assignment of names is allowed - a problem seldom been considered in the literature. The latter further rectifies the errors in metadata, specifically to correct false labels and annotate faces with missing names in the metadata of a video, by considering a group of socially connected videos for joint label inference. Experimental results on a large archive of Web videos show the robustness of the proposed approach in dealing with the problems of missing and false labels, leading to higher accuracy in face labeling than several existing approaches but with minor degradation in speed efficiency.