GEstures and Head Movements in language (GEHM)

The GEHM network will support cooperation among eight leading research groups working in the area of gesture and language, and thereby foster new theoretical insights into the way hand gestures and head movements interact with speech in face-to-face multimodal communication.

The network has specific focus on three research strands:

  1. language-specific characteristics of gesture-speech interaction
  2. multimodal prominence
  3. multimodal behaviour modelling

 

 

 

 

  1. The first research strand, on language-specific characteristics of gesture-speech interaction, will work towards a theory that can account for how speakers’ ability to process and produce gesture and speech is affected and changed by their language profile. Speech-gesture profiles of monolingual and bilingual speakers’ production will be established by combining audio, video and sensor output from motion capture. These rich multimodal data will provide fine-grained information about cross-linguistic differences in native and non-native speech-gesture coordination.
        
  2. The second research strand, on multimodal prominence, investigates the theoretical question how linguistic prominence is expressed through combinations of kinematic and prosodic features. In general, it is not yet well understood how gestures and pitch accents might be combined to create different types of multimodal prominence, and how specifically visual prominence cues are used in spoken communication. Datasets will be created and analysed by this research to arrive at a fine-grained and largely documented theory of multimodal prominence.
         
  3. The third research strand, on modelling multimodal behaviour, aims at conceptual and statistical modelling of multimodal contributions, with particular regard to head movements and the use of gaze. This research strand will develop models of multimodal behaviour based on the datasets developed in the previous two strands, but also take advantage of existing corpora, including interaction data where eye-gaze has been tracked.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

2021

Gullberg, M. (2021). Bimodal convergence: How languages interact in multicompetent language users’ speech and gestures. In A. Morgenstern & S. Goldin-Meadow (Eds.), Gesture in language: Development across the lifespan (pp. 318-333). Berlin: Mouton de Gruyter.

Mesh, K., Cruz, E., van de Weijer, J., Burenhult, N., & Gullberg, M. (2021). Effects of scale on multimodal deixis: Evidence from Quiahije Chatino. Frontiers in Psychology, 11(584231). doi:10.3389/fpsyg.2020.584231

Paggio, P., Navarretta, C., Jongejan, B., & Aguirrezabal Zabaleta, M. (2021). Towards a Methodology Supporting Semiautomatic Annotation of Head Movements in Video-recorded Conversations. I Proceedings of The Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop (s. 151-159). Association for Computational Linguistics.

2020

Ambrazaitis, G., Frid, J. and House, D. (2020). Word prominence ratings in Swedish television news readings: effects of pitch accents and head movements. In Proceedings of Speech Prosody 2020, online https://sp2020.jpn.org/

Ambrazaitis, G., Zellers, M., House, D. (2020) Compounds in interaction: patterns of synchronization between manual gestures and lexically stressed syllables in spontaneous Swedish. In: Proceedings of Gesture and Speech in Interaction (GESPIN2020).

Debreslioska, S., & Gullberg, M. (2020a). The semantic content of gestures varies with definiteness, information status and clause structure. Journal of Pragmatics, 168, 36-52. doi:10.1016/j.pragma.2020.06.005

Debreslioska, S., & Gullberg, M. (2020b). What’s new? Gestures accompany inferable rather than brand-new referents in discourse. Frontiers in Psychology, Gesture-speech integration: Combining gesture and speech to create understanding(11), 1935. doi:10.3389/fpsyg.2020.01935

McLaren, L., Koutsombogera, M. and Vogel, C. (2020) A Heuristic Method for Automatic Gaze Detection in Constrained Multi-Modal Dialogue Corpora. In Proceedings of the 11th IEEE International Conference on Cognitive Infocommunications (CogInfoCom), Mariehamn, Finland, 2020, pp. 55-60, doi: 10.1109/CogInfoCom50765.2020.9237883. 

McLaren, L., Koutsombogera, M. and Vogel, C. (2020) Gaze, Dominance and Dialogue Role in the MULTISIMO Corpus. In Proceedings of the 11th IEEE International Conference on Cognitive Infocommunications (CogInfoCom), Mariehamn, Finland, 2020, pp. 83-88, doi: 10.1109/CogInfoCom50765.2020.9237833.

Navarretta, C & Paggio, P 2020, Dialogue Act Annotation in a Multimodal Corpus of First Encounter Dialogues. i Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020). European Language Resources Association, s. 627-636. 

Paggio, P., Agirrezabal, M., Jongejan, B. and C. Navarretta (2020). Automatic Detection and Classification of Head Movements in Face-to-Face Conversations. In Proceedings of ONION 2020: Workshop on peOple in laNguage, vIsiOn and the miNd, pages 15–21 Language Resources and Evaluation Conference (LREC 2020), Marseille, 11–16 May 2020. https://www.aclweb.org/anthology/2020.onion-1.3.pdf 

Paggio, P, Gatt, A & Klinger, R (eds) 2020, Proceedings of LREC2020 Workshop "People in language, vision and the mind'' (ONION2020). European Language Resources Association.

Prieto, P., and Espinal, M.T. (2020). "Prosody, Gesture, and Negation". The Oxford Handbook of Negation, ed. by V. Deprez and M.Teresa Espinal. Oxford: Oxford University Press, pp.667-693

Vilà-Giménez, I., and Prieto, P. (in press, 2020). "Encouraging kids to beat: Children's beat gesture production boosts their narrative performance." Developmental Science. First online:  https://doi.org/10.1111/desc.12967

Vogel, Carl and Anna Esposito, "Interaction Analysis and Cognitive Infocommunications", Infocommunications Journal, Vol. XII, No 1, March 2020, pp. 2-9. DOI: 10.36244/ICJ.2020.1.1

Zhang,Y., Baills, F., and Prieto, P. (in press). "Hand-clapping to the rhythm of newly learned words improves L2 pronunciation: Evidence from training Chinese adolescents with French words". Language Teaching Research. First online:  https://doi.org/10.1177/1362168818806531  

2019

Cravotta, A., Busà, M. G., and Prieto, P. (2019). "Effects of Encouraging the Use of Gestures on Speech". Journal of Speech, Language, and Hearing Research, 62, 3204-3219.

Debreslioska, S., & Gullberg, M. (2019). Discourse is bimodal: How information status in speech interacts with presence and viewpoint of gestures. Discourse Processes, 56(1), 41-60. doi:10.1080/0163853X.2017.1351909

Debreslioska, S., van de Weijer, J., & Gullberg, M. (2019). Addressees are sensitive to the presence of gestures when tracking a single referent in discourse. Frontiers in Psychology, 10(1775). doi:10.3389/fpsyg.2019.01775

Hübscher, I., and Prieto, P. (2019). "Gestural and prosodic development act as sister systems and jointly pave the way for children’s sociopragmatic development". Frontiers in Psychology, 10:1259.

Nirme, J., Haake, M., Gulz, A., & Gullberg, M. (2019). Motion capture-based animated characters for the study of speech-gesture integration. Behaviour Research Methods. doi:10.3758/s13428-019-01319-w

Paggio, P., & Navarretta, C. (2019). Multimodal feedback i social interaktion. NyS, 56, 77-101.

Sandler, W., Gullberg, M., & Padden, C. (Eds.). (2019). Visual language. Lausanne: Frontiers Media.

Vilà-Giménez, I., Igualada, A., and Prieto, P. (2019). "Observing storytellers who use rhythmic beat gestures improves children’s narrative discourse performance". Developmental Psychology, 55(2), 250-262. A video abstract of this article can be viewed at: https://www.youtube.com/watch?v=t-EKJIQt20g.

Vogel, Carl and Anna Esposito , Linguistic and Behaviour Interaction Analysis within Cognitive Infocommunications, 10th IEEE International Conference on Cognitive Infocommunications (CogInfoCom), Naples, Italy, 23-25 Oct. 2019, 2019, pp47 - 52 Conference Paper, 2019 DOI: https://doi.org/10.1109/CogInfoCom47531.2019.9089904

Yuan, C., González-Fuente, S., Baills, F., and Prieto, P. (2019). "Observing pitch gestures favors the learning of Spanish intonation by Mandarin speakers". Studies in Second Language Acquisition, 41(1), 5-32.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Researchers

UCPH researchers

Name Title Phone E-mail
Bart Jongejan Software Developer +4535329075 E-mail
Costanza Navarretta Senior Researcher +4535329079 E-mail
Elisabeth Engberg-Pedersen Professor +4530298664 E-mail
Patrizia Paggio Associate Professor +4535329072 E-mail

Funded by

Independent Research Fund Denmark

The network is funded by the Independent Research Fund Denmark with grant 9055-00004B.

Project period: 1 September 2019 - 28 February 2023.

Contact

The MultiModal MultiDimensional (M3D) labeling system

11-01-2021

The M3D labeling system represents a joint effort between three gesture labs (one of the GEHM labs, e.g., the Prosodic Studies Group at Universtitat Pompeu Fabra).

More details

2nd International Workshop on Language Acquisition

23-06-2021 08:25

The Second International Workshop on Language Acquisition, University of Copenhagen, Denmark.

More details

2021 GEHM plenary meeting

15-12-2021 13:00

The GEHM network on Gesture and Head Movements in Language is organising a plenary meeting open to the project participants and other interested on 15-16 December 2021.

More details

Other network members

Department of Linguistics and Phonetics at Kiel University:

Division of Speech, Music and Hearing at KTH Royal Institute of Technology:

MIDI group at KU Leuven:

Centre for IMS at Linnaeus University:

Centre for Languages and Literature and Lund University Humanities Lab at Lund University:

Computational Linguistics Group at Trinity College Dublin:

GrEP at Universitat Pompeu Fabra:

University of Malta, Institute of Linguistics and Language Technology: