RWTH Aachen University | Aachen, Germany
Augmented iconicity – How kinetic data analyses may inform gesture theory
Adopting a cognitive-semiotic perspective, this talk aims to highlight how empirical research into multimodal interaction may inform gesture theory and vice-versa (Mittelberg 2019; Wolf et al. 2017). I will present a set of studies carried out in the Natural Media Lab (RWTH Aachen University), which – while cutting across modalities and disciplinary boundaries – have two things in common: They a) where motivated by questions central to semiotic and cognitive linguistic theory and b) involve motion-capture technology (MoCap). In multimodal interaction research, MoCap has proven to be a powerful tool to investigate furtive gestures and head movements in minute detail. Compared to 2-D video recordings, 3-D numerical MoCap data represent quickly emerging and vanishing hand shapes and movements on a millimeter and millisecond scale. For example, visualizing otherwise invisible motion traces allows for new insights into the dynamic and often rather schematic gestalt properties of communicative movements (Mittelberg 2019). First, the focus will be on qualitative analyses of gestures metonymically evoking actions and objects, by showing how a gesture’s iconicity may be augmented through visualizing and freezing its motion trajectory. MoCap plots of gestures employed to describe paintings, for instance, have revealed formal and aesthetic qualities that facilitate deriving the underlying embodied image schemas and force dynamics (Mittelberg 2013, 2018). Second, I will discuss quantitative approaches that involve pattern analysis within and across study participants, e.g., to visualize the speakers’ individual use of gesture space via heat maps (Priesters & Mittelberg 2013). In another study, gesture signatures of selected movement types served as input for an algorithm searching the MoCap data set for similar trajectories, which afforded automated retrieval of all the tokens of a given gesture type (Schüller et al. 2017). Finally, I will address some of the technical and methodological challenges that come with exploiting MoCap technology in the context of interdisciplinary gesture research.
Mittelberg, I. (2013). Balancing acts: Image schemas and force dynamics as experiential essence in pictures by Paul Klee and their gestural enactments. In: M. Borkent et al. (eds.), Language and the Creative Mind. Stanford: CSLI. 325–346.
Mittelberg, I. (2018). Gestures as image schemas and force gestalts: A dynamic systems approach augmented with motion-capture analyses. Cognitive Semiotics 11 (1). https://doi.org/10.1515/cogsem-2018-0002
Mittelberg, I. (2019). Peirce’s universal categories: On their potential for gesture theory and multimodal analysis. Semiotica 228. 193-222. DOI 10.1515/sem-2018-0090
Priesters, M., Mittelberg, I. (2013). Individual differences in speakers’ gesture spaces: Multi-angle views from a motion-capture study. Proceedings of the Tilburg Gesture Research Meeting (TiGeR), June 19-21.
Schüller, D., Beecks, C., Hassani, M., Hinnell, J., Brenger, B., Seidl, T., Mittelberg, I. (2017). Automated Pattern Analysis in Gesture Research: Similarity Measuring in 3D Motion Capture Models of Communicative Action. Digital Humanities Quarterly 11, 2.
Wolf, D., Rekittke L.-M., Mittelberg I., Klasen M., Mathiak K. (2017). Perceived conventionality in co-speech gestures involves the fronto-temporal language network. Frontiers in Human Neuroscience, doi: 10.3389/fnhum.2017.00573
Irene Mittelberg is Professor of Linguistics and Cognitive Semiotics at the Institute for English, American, and Romance Studies of RWTH Aachen University, where she directs the Centre for Sign Language and Gesture (SignGes) and founded the Natural Media Lab, a gesture research lab equipped with motion-capture technology. She holds an M.A. in French linguistics and art history from Hamburg University and a Ph.D. in linguistics and cognitive studies from Cornell University. Her work combines the semiotic theories of C. S. Peirce and Roman Jakobson with contemporary embodied approaches to language, cognition, and multimodal interaction, notably to examine how image schemas, iconicity, metonymy, metaphor, viewpoint, and frames motivate coverbal gestures. Another focus is comparing processes of sign formation and the use of space in gesture, architectural design, and the visual arts. Recent interdisciplinary work includes pattern analysis in kinetic gesture data and the adoption of Peirce’s universal categories for neuroscientific research into gesture. She published a monograph on Metaphor and Metonymy in Language and Gesture (2006), co-edited Methods in Cognitive Linguistics (2007), edited a special issue on gesture for Sprache und Literatur (2010), and has (co-)published over 50 journal articles/books chapters. Currently, she is preparing, with Todd Oakley, a special issue on Cognitive Semiotics for Zeitschrift für Semiotik.