Belinda Crawford Camiciottoli
University of Pisa | Pisa, Italy
Analysing intersemiotic complementarity in OpenCourseWare lectures: An experimental mixed-method approach
As the primary instructional genre in higher education, the lecture is a medium through which experts communicate knowledge to novices by means of multiple semiotic resources. Indeed, lectures have been described as “multimodal formations” (Brabazon, 2006) that may integrate various resources (e.g., language, images, sounds, gestures) within the interactional setting. As such, to fully benefit from lectures, students need to be able to construct meanings through intersemiotic complementarity, that is, how different modes “complement each other in the ways that they project meaning” (Royce, 2007, p. 63). During a lecture, students are challenged not only absorb a high concentration of complex academic content, but also to interpret interpersonal meanings driven by lecturer-audience interaction. Non-verbal cues (e.g., prosodic features, facial expressions, gestures) may play a key role in helping students to accurately interpret lecturers’ interpersonal meanings, specifically their attitudes towards topics, thus responding to students’ interest in knowing the viewpoint of an expert (Northcott, 2001). The potential clarifying and/or reinforcing function of non-verbal resources is particularly useful when lecturers and students do not share the same language background, reflecting a frequent situation in today’s globalized academic world.
In this short paper, I describe an experimental methodology for analysing the co-occurrence of verbal and non-verbal resources in the expression of attitude in university lectures that combines techniques from corpus linguistics and multimodal discourse analysis. The analysis is based on a small dataset of 31 video clips from five OpenCourseWare lectures that was drawn from a larger multimodal corpus of video clips across a range of multimedia genres. In the first phase, the linguistic expression of attitude was teased out from the speech transcripts of the video clips by means of part-of-speech tagging in order to identify all adjectives. Because adjectives constitute a highly frequent and open-class grammatical category, such automated processing is critical for a thorough analysis, even with relatively limited amounts of textual data. The adjectives were then extracted and filtered by means of text analysis software to distinguish only those that encoded the speaker’s attitude according to Martin and White’s (2005) appraisal model. In the second phase, the corresponding segments of the video clips that contained the previously identified attitudinal adjectives were then carefully viewed to pinpoint co-occurring non-verbal resources, such as prosodic stress, gesturing, gaze direction, facial expressions, body positioning. This analysis was facilitated by multimodal annotation software that allows users to code and mark particular verbal and/or nonverbal features. The annotations are then synchronized with streaming videos, thus allowing researchers to capture the contemporaneous use of multiple semiotic modes. The mixed-method approach that incorporated corpus tools enabled a systematic and reasonably exhaustive analysis the expression of attitude in the lecture clips, which would have been much less feasible through video clip observation alone, and could also be applied to larger scale analyses of multimodal data. Findings from such analyses can inform multimodal approaches to improve comprehension among L2 students in higher education settings by raising their awareness of how “meanings are made (as well as distributed, interpreted, and remade) through many forms and resources of which language is but one—image, gesture, gaze, body posture, sound, writing, music, speech, and so on” (Jewitt, 2013, p. 4109-4110).
Brabazon, T. (2006). Socrates in earpods? The ipodification of education. Fast Capitalism, 2(1). Retrieved from http://www.uta.edu/huma/agger/fastcapitalism/2_1/brabazon.html.
Jewitt, C. (2013). Multimodal teaching and learning. In C. A. Chapelle (Ed.) The encyclopedia of applied linguistics. (pp 4109-4114). Oxford: Blackwell Publishing Ltd.
Martin, J. R., & White, P. R. R. (2005). The language of evaluation. Appraisal in English. Basingstoke: Palgrave Macmillan.
Northcott, J. (2001). Towards an ethnography for the MBA classroom: A consideration of the role of interactive lecturing styles within the context of one MBA programme. English for Specific Purposes 20(1), 15-37.
Royce, T. D. (2007). Intersemiotic complementarity: A framework for multimodal discourse analysis. In T. D. Royce & W. L. Bowcher (Eds.), New directions in the analysis of multimodal discourse (pp. 63–109). Mahwah, NJ: Lawrence Erlbaum.
Belinda Crawford Camiciottoli is Associate Professor of English Language and Linguistics at the University of Pisa. Her research interests include interpersonal, pragmatic, and multimodal features of discourse found in both academic and professional settings. She has published in leading international journals, including Intercultural Pragmatics, Journal of Pragmatics, Discourse & Communication, Discourse, Context and Media, English for Specific Purposes, and Text & Talk. She is co-editor of Multimodal analysis in academic settings: From research to teaching (2015, Routledge) and a Special Issue of System entitled Multimodal Perspectives on English Language Teaching in Higher Education (2018).