GEstures and Head Movements in language (GEHM)

The GEHM network will support cooperation among eight leading research groups working in the area of gesture and language, and thereby foster new theoretical insights into the way hand gestures and head movements interact with speech in face-to-face multimodal communication.

The network has specific focus on three research strands:

  1. language-specific characteristics of gesture-speech interaction
  2. multimodal prominence
  3. multimodal behaviour modelling





  1. The first research strand, on language-specific characteristics of gesture-speech interaction, will work towards a theory that can account for how speakers’ ability to process and produce gesture and speech is affected and changed by their language profile. Speech-gesture profiles of monolingual and bilingual speakers’ production will be established by combining audio, video and sensor output from motion capture. These rich multimodal data will provide fine-grained information about cross-linguistic differences in native and non-native speech-gesture coordination.
  2. The second research strand, on multimodal prominence, investigates the theoretical question how linguistic prominence is expressed through combinations of kinematic and prosodic features. In general, it is not yet well understood how gestures and pitch accents might be combined to create different types of multimodal prominence, and how specifically visual prominence cues are used in spoken communication. Datasets will be created and analysed by this research to arrive at a fine-grained and largely documented theory of multimodal prominence.
  3. The third research strand, on modelling multimodal behaviour, aims at conceptual and statistical modelling of multimodal contributions, with particular regard to head movements and the use of gaze. This research strand will develop models of multimodal behaviour based on the datasets developed in the previous two strands, but also take advantage of existing corpora, including interaction data where eye-gaze has been tracked.


















Gullberg, M. (2021). Bimodal convergence: How languages interact in multicompetent language users’ speech and gestures. In A. Morgenstern & S. Goldin-Meadow (Eds.), Gesture in language: Development across the lifespan (pp. 318-333). Berlin: Mouton de Gruyter.

Mesh, K., Cruz, E., van de Weijer, J., Burenhult, N., & Gullberg, M. (2021). Effects of scale on multimodal deixis: Evidence from Quiahije Chatino. Frontiers in Psychology, 11(584231). doi:10.3389/fpsyg.2020.584231

Paggio, P., Navarretta, C., Jongejan, B., & Aguirrezabal Zabaleta, M. (2021). Towards a Methodology Supporting Semiautomatic Annotation of Head Movements in Video-recorded Conversations. I Proceedings of The Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop (s. 151-159). Association for Computational Linguistics.


Ambrazaitis, G., Frid, J. and House, D. (2020). Word prominence ratings in Swedish television news readings: effects of pitch accents and head movements. In Proceedings of Speech Prosody 2020, online

Ambrazaitis, G., Zellers, M., House, D. (2020) Compounds in interaction: patterns of synchronization between manual gestures and lexically stressed syllables in spontaneous Swedish. In: Proceedings of Gesture and Speech in Interaction (GESPIN2020).

Debreslioska, S., & Gullberg, M. (2020a). The semantic content of gestures varies with definiteness, information status and clause structure. Journal of Pragmatics, 168, 36-52. doi:10.1016/j.pragma.2020.06.005

Debreslioska, S., & Gullberg, M. (2020b). What’s new? Gestures accompany inferable rather than brand-new referents in discourse. Frontiers in Psychology, Gesture-speech integration: Combining gesture and speech to create understanding(11), 1935. doi:10.3389/fpsyg.2020.01935

McLaren, L., Koutsombogera, M. and Vogel, C. (2020) A Heuristic Method for Automatic Gaze Detection in Constrained Multi-Modal Dialogue Corpora. In Proceedings of the 11th IEEE International Conference on Cognitive Infocommunications (CogInfoCom), Mariehamn, Finland, 2020, pp. 55-60, doi: 10.1109/CogInfoCom50765.2020.9237883. 

McLaren, L., Koutsombogera, M. and Vogel, C. (2020) Gaze, Dominance and Dialogue Role in the MULTISIMO Corpus. In Proceedings of the 11th IEEE International Conference on Cognitive Infocommunications (CogInfoCom), Mariehamn, Finland, 2020, pp. 83-88, doi: 10.1109/CogInfoCom50765.2020.9237833.

Navarretta, C & Paggio, P 2020, Dialogue Act Annotation in a Multimodal Corpus of First Encounter Dialogues. i Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020). European Language Resources Association, s. 627-636. 

Paggio, P., Agirrezabal, M., Jongejan, B. and C. Navarretta (2020). Automatic Detection and Classification of Head Movements in Face-to-Face Conversations. In Proceedings of ONION 2020: Workshop on peOple in laNguage, vIsiOn and the miNd, pages 15–21 Language Resources and Evaluation Conference (LREC 2020), Marseille, 11–16 May 2020. 

Paggio, P, Gatt, A & Klinger, R (eds) 2020, Proceedings of LREC2020 Workshop "People in language, vision and the mind'' (ONION2020). European Language Resources Association.

Prieto, P., and Espinal, M.T. (2020). "Prosody, Gesture, and Negation". The Oxford Handbook of Negation, ed. by V. Deprez and M.Teresa Espinal. Oxford: Oxford University Press, pp.667-693

Vilà-Giménez, I., and Prieto, P. (in press, 2020). "Encouraging kids to beat: Children's beat gesture production boosts their narrative performance." Developmental Science. First online:

Vogel, Carl and Anna Esposito, "Interaction Analysis and Cognitive Infocommunications", Infocommunications Journal, Vol. XII, No 1, March 2020, pp. 2-9. DOI: 10.36244/ICJ.2020.1.1

Zhang,Y., Baills, F., and Prieto, P. (in press). "Hand-clapping to the rhythm of newly learned words improves L2 pronunciation: Evidence from training Chinese adolescents with French words". Language Teaching Research. First online:  


Cravotta, A., Busà, M. G., and Prieto, P. (2019). "Effects of Encouraging the Use of Gestures on Speech". Journal of Speech, Language, and Hearing Research, 62, 3204-3219.

Debreslioska, S., & Gullberg, M. (2019). Discourse is bimodal: How information status in speech interacts with presence and viewpoint of gestures. Discourse Processes, 56(1), 41-60. doi:10.1080/0163853X.2017.1351909

Debreslioska, S., van de Weijer, J., & Gullberg, M. (2019). Addressees are sensitive to the presence of gestures when tracking a single referent in discourse. Frontiers in Psychology, 10(1775). doi:10.3389/fpsyg.2019.01775

Hübscher, I., and Prieto, P. (2019). "Gestural and prosodic development act as sister systems and jointly pave the way for children’s sociopragmatic development". Frontiers in Psychology, 10:1259.

Nirme, J., Haake, M., Gulz, A., & Gullberg, M. (2019). Motion capture-based animated characters for the study of speech-gesture integration. Behaviour Research Methods. doi:10.3758/s13428-019-01319-w

Paggio, P., & Navarretta, C. (2019). Multimodal feedback i social interaktion. NyS, 56, 77-101.

Sandler, W., Gullberg, M., & Padden, C. (Eds.). (2019). Visual language. Lausanne: Frontiers Media.

Vilà-Giménez, I., Igualada, A., and Prieto, P. (2019). "Observing storytellers who use rhythmic beat gestures improves children’s narrative discourse performance". Developmental Psychology, 55(2), 250-262. A video abstract of this article can be viewed at:

Vogel, Carl and Anna Esposito , Linguistic and Behaviour Interaction Analysis within Cognitive Infocommunications, 10th IEEE International Conference on Cognitive Infocommunications (CogInfoCom), Naples, Italy, 23-25 Oct. 2019, 2019, pp47 - 52 Conference Paper, 2019 DOI:

Yuan, C., González-Fuente, S., Baills, F., and Prieto, P. (2019). "Observing pitch gestures favors the learning of Spanish intonation by Mandarin speakers". Studies in Second Language Acquisition, 41(1), 5-32.






























UCPH researchers

Name Title Phone E-mail
Bart Jongejan Software Developer +4535329075 E-mail
Costanza Navarretta Senior Researcher +4535329079 E-mail
Patrizia Paggio Associate Professor +4535329072 E-mail

Funded by

Independent Research Fund Denmark

The network is funded by the Independent Research Fund Denmark with grant 9055-00004B.

Project period: 1 September 2019 - 31 December 2023.


Other network members

Department of Linguistics and Phonetics at Kiel University:

Division of Speech, Music and Hearing at KTH Royal Institute of Technology:

MIDI group at KU Leuven:

Centre for IMS at Linnaeus University:

Centre for Languages and Literature and Lund University Humanities Lab at Lund University:

Computational Linguistics Group at Trinity College Dublin:

GrEP at Universitat Pompeu Fabra:

University of Malta, Institute of Linguistics and Language Technology: