Daniel Echeverri and Huaxin Wei
The Hong Kong Polytechnic University | Hong Kong, China
Hear, See, Do: Empirical Insights of Interactive Storytelling and Metamodality
This paper presents a qualitative study that sought to approach multimodality, not as a way to understand how meaning is made but to understand how different semiotic forms can drive the comprehension of interactive storytelling. This consideration was the ground to observe the unique experiences of interacting with Letters to José – a physical/digital hybrid interactive story. This interactive story represents a series of narrative events through paper-based mechanisms that trigger different visual and auditory media as a transmedial experience.
In this study, distinctive attention was paid to metamodality; or the nesting of modes within other modes, according to Burn’s analytical framework on the relation between orchestrating and contributory modes. Based on the empirical data gathered, the paper intends to examine from the perspective of design the contentions between auditive, visual and performative modes. It also discusses the challenges these contentions bring to the design of interactive stories. In particular, the paper focus on the modes’ material qualities as cognitive cues to involve audiences in the storyworld, studying how they lead the exploratory and ontological interactions – sound as the orchestrator of the experience and performance as a bridge between the narrative and the participant; moreover, how the domination of a single mode over others impact narrative comprehension. Lastly, the paper will identify, from the making and study of Letters to José, the benefit and challenge of introducing design-led empirical research to the multimodal community.
Burn, A. (2014). The Kineikonic Mode: Towards A Multimodal Approach to Moving Image Media. In C. Jewitt, The Routledge Handbook of Multimodal Analysis (Second Edition, p. 26). Abingdon, Oxon: Routledge.
Dena, C. (2009). Beyond Multimedia, Narrative, and Game. In R. Page (Ed.), New Perspectives on Narrative and Multimodality (First, pp. 184–201). New York, NY, USA: Rutledge.
Green, M. C., & Brock, T. C. (2000). The role of transportation in the persuasiveness of public narratives. Journal of personality and social psychology, 79(5), 701.
Kress, G., & Van Leeuwen, T. V. (2001). Multimodal discourse: The modes and media of contemporary communication.
Mitchell, A. (2008). Narrative production and interactive storytelling. Refractory: A Journal of Entertainment Media, 13, 18.
Daniel Echeverri is a Graphic Designer, PhD student and design educator based in Hong Kong. He holds a Master of Fine Arts degree in Visual Communication Design from Kent State University in. He is currently in the second year of his PhD studies in the School of Design at the Hong Kong Polytechnic University where he also tutors in the BA in Interactive Media in the same school. His doctoral research is currently on Narratives, Memory and Interaction and seeks to explore ways in which tangible interaction can support interactive narratives.
Huaxin Wei is an assistant professor in the School of Design at the Hong Kong Polytechnic University. Her research interests include interactive narrative, game design and analysis, and interaction design. Her primary focus is in the area of game narrative, where she established a descriptive framework for systematic analysis of video games using a variety of structural perspectives. Huaxin earned her PhD from the School of Interactive Arts and Technology at Simon Fraser University, Canada.