Temporal Coordination of Facial Expressions and Head Movements in First Encounter Dialogues

Research output: Chapter in Book/Report/Conference proceedingArticle in proceedingsResearchpeer-review

This paper deals with the temporal coordination between facial expressions and co-occurring head movements in a multimodal corpus of first encounter conversations. In particular, we look at how the onset of facial expressions is coordinated with the first overlapping head movement, in other words which of the two modalities precedes the other and why. We find and discuss statistical main effects on the temporal delays between the two behaviours due to individual variation, type of head movement, and the communicative function of the multimodal signal. In particular, the analysis shows that when speakers give feedback, their facial expression becomes visible before the head starts to move, especially in the case of negative comments associated with frowning or scowling. The opposite is true when the multimodal signal is used as a comment to the speaker's own speech. The motivation for the analysis is to shed light on a less studied aspect of multimodal communication -- an aspect that is relevant to the generation of natural multimodal expressions in ECAs.
Original languageEnglish
Title of host publicationProceedings of the LREC 2018 Workshop "LB-ILR2018 and MMC2018 Joint Workshop" : Language and Body in Real Life
EditorsHanae Koiso, Patrizia Paggio
Number of pages5
PublisherEuropean Language Resources Association
Publication date2018
Pages40-44
ISBN (Electronic)979-10-95546-16-0
Publication statusPublished - 2018

ID: 201504986