TY - GEN

T1 - Automatic Gender and Identity Recognition in Annotated Multimodal Face-to-face Conversations

AU - Navarretta, Costanza

PY - 2018

Y1 - 2018

N2 - This paper addresses the automatic recognition ofthe gender and identity of speakers in spontaneous dyadic conversationsusing information about the multimodal communicativebehavior of the participants. Identifying gender or individualspecific behaviors in face to face communication is relevant forconstructing advanced and robust interactive systems. This informationalso contributes to understanding how humans communicateface-to-face. In the present work, classifiers have been trainedon features extracted from an annotated multimodal corpus oftwelve first encounters in order to distinguish the gender and theidentity of the participants. The training features comprise speechduration and shape annotations of co-speech communicative headmovements, facial expressions, body postures and hand gesturesof six female and six male participants. Information about theemotions shown by the participants’ facial expressions was alsoadded to the training set. Differing from other studies addressingrecognition of individuals for security systems using databasesbuilt for the purpose, the multimodal training features in thisstudy are exclusively related to communication and the data arespontaneous occurring conversations since we study multimodalcommunication. A number of classifiers were trained on the dataand the best results were obtained by a multilayer perceptronfor gender recognition with a weighed F-score of 0.65 (accuracy64%) and by multinomial logistic regression for the classificationof 12 participants with an F-score of 0.31 (accuracy 30%). Themost useful features for gender recognition were informationabout the emotions shown by the participants, the type of headmovements and handedness, while the features which were mostuseful for the identification of individuals are emotions, headmovements, handedness and body direction. The results on bothtasks are significantly better than by chance accuracy and theresults obtained by a majority classifier. This is promising sincethis is a first pilot study on a corpus of limited size. The featuresaddressed in this study could in the future be combined to otherbiometric patterns such as those used in multimedia securitysystems.

AB - This paper addresses the automatic recognition ofthe gender and identity of speakers in spontaneous dyadic conversationsusing information about the multimodal communicativebehavior of the participants. Identifying gender or individualspecific behaviors in face to face communication is relevant forconstructing advanced and robust interactive systems. This informationalso contributes to understanding how humans communicateface-to-face. In the present work, classifiers have been trainedon features extracted from an annotated multimodal corpus oftwelve first encounters in order to distinguish the gender and theidentity of the participants. The training features comprise speechduration and shape annotations of co-speech communicative headmovements, facial expressions, body postures and hand gesturesof six female and six male participants. Information about theemotions shown by the participants’ facial expressions was alsoadded to the training set. Differing from other studies addressingrecognition of individuals for security systems using databasesbuilt for the purpose, the multimodal training features in thisstudy are exclusively related to communication and the data arespontaneous occurring conversations since we study multimodalcommunication. A number of classifiers were trained on the dataand the best results were obtained by a multilayer perceptronfor gender recognition with a weighed F-score of 0.65 (accuracy64%) and by multinomial logistic regression for the classificationof 12 participants with an F-score of 0.31 (accuracy 30%). Themost useful features for gender recognition were informationabout the emotions shown by the participants, the type of headmovements and handedness, while the features which were mostuseful for the identification of individuals are emotions, headmovements, handedness and body direction. The results on bothtasks are significantly better than by chance accuracy and theresults obtained by a majority classifier. This is promising sincethis is a first pilot study on a corpus of limited size. The featuresaddressed in this study could in the future be combined to otherbiometric patterns such as those used in multimedia securitysystems.

M3 - Article in proceedings

SN - 978-1-5386-7094-1 3

SP - 87

EP - 92

BT - 9th IEEE International Conference on Cognitive Infocommunications (CogInfoCom 2018)

PB - IEEE

CY - Budapest

ER -