BreMM19 | Cassens & Wegener

Jörg Cassens and Rebekah Wegener
University of Hildesheim | Hildesheim, Germany
Paris Lodron University of Salzburg | Salzburg, Austria

Combining the lab with the crowd to understand multimodal responses to fantasy text reading

The observation of individuals reading texts reveals a complex interplay of individual, textual and contextual features. Different aspects of the text, the setting in which it is read and the current mood of the reader all influence the way a reader reacts to a text emotionally or in terms of interest. The reaction of the reader can be observed on many levels, ranging from spoken expressions to physiological measures, facial micro-gestures, reader posture and behaviour (audio-visual and eye-tracking). These responses are brought together with aspects of the stimulus texts to reveal the textual triggers for reader expressed affect.

Hsu et al (2015) argue that a reader’s reaction to a text is not just a reaction to the lexical items, but to their organization and the context in which they are embedded. To capture this complexity, Wegener et al (2018), Wegener (2011) and Wegener et al (2017) model the reading process as a series of layers of contexts. But capturing this level of data complexity limits the scale of any study to the relatively small number of individuals that can be studied in the lab, reducing our ability to generalise our findings or verify results statistically.

In this paper we report on a method for extending small scale lab based experimental data that is very rich with large scale but relatively poor crowdsourced data. In a small pilot study, we combine lab based data that includes reader reactions (audio-visual and eye-tracking) with reader annotations (textual) and reader interviews (audio-visual and eye-tracking) with large scale crowdsourced data. This process involves aligning time-aligned data with text-aligned data. Once aligned, we can use this to identify and extract data from the GoodReads database to find portions of the texts that systematically trigger identifiable responses in the readers. We present both an overview of the project as a whole and tools developed to align the different data types and visualize the findings.


Hsu, C.-T. Jacobs, A. M. Citron, F. M. and Conrad, M. “The emotion potential of words and passages in reading harry potter–an fmri study,” Brain and language, vol. 142, pp. 96–114, 2015.

Wegener, R. and Lothmann, T. “That is not normal rabbit behaviour: on the trail of the grammar of fictional worlds,” in On Verbal Art: essays in honour of Ruqaiya Hasan (in press), R. Wegener, A. Oesterle, and S. Neumann, Eds. Sheffield: Equinox, 2018.

Wegener, R. “Parameters of context: From theory to model and application” Ph.D. dissertation, Macquarie University, Faculty of Human Sciences, Department of Linguistics, 2011.

Wegener, R.; Kohlschein, C.; Jeschke, S.; and Neumann, S. “EmoLiTe – A Database for Emotion Detection During Literary Text Reading.” In Proceedings of the 5th International workshop on Context Based Affect Recognition ACII 2017, 2017.


Jörg Cassens: Jörg Cassens is a lecturer and senior researcher in media informatics at the University of Hildesheim, Germany. His main research interests are the applicability of socio-technical, psychological and semiotic theories for design, implementation and deployment of intelligent systems. Working at the intersection of Human-Computer Interaction and Arti cial Intelligence, he is particularly interested in the usability of and user experience with computational systems. He has worked on the development of psychologically sound context models, interfaces based on meaning bearing behaviour and on requirements engineering methodologies for intelligent systems.

Rebekah Wegener: Rebekah Wegener is a lecturer and senior researcher in
linguistics and semiotics at the Paris Lodron University Salzburg, Austria, and co-founder of learning technology startup Audaxi in Sydney, Australia. Her research interests include context modelling, theoretical and applied linguistics as well as intelligent learning and teaching technologies. She is currently working on models of context for text understanding and multimodal environments as well as behavioural interfaces for arti cial intelligence.

  • © 2020 University of Bremen || Faculty of Linguistics and Literary Science