Frederico Belcavello, Alexandre Diniz da Costa, Vânia Gomes de Almeida, Marcelo Viridiano and Tiago Timponi Torrent
Federal University of Juiz de Fora | Juiz de Fora, Brazil
Multimodal Analysis for Building Semantic Representations in The Tourism Domain Using Frames and Qualia
When people decide to travel, they need information to plan their tourism activities. There is a lot of information available online in several types of semioses, but it is humanly impossible to access and categorize all this information. Based on that, we aim to build a common semantic representation for both textual and imagetic information that can contribute to grouping and categorizing tourist attractions as means of mapping the contributions of these different types of semioses to meet the needs of the traveler in a friendly and simple query interface. We follow Kress’ and Van Leeuwen’s (2006) proposition that the sign representation is always multiple and that the different uses of visual elements affect meaning, as far as the choice between expressing something verbally or visually makes great difference. The theoretical approaches that guide us through our linguistic analysis are Frame Semantics (Fillmore, 1982) and Qualia Relations (Pustejovsky, 1995). FrameNet is a lexicographic database that describes the words in a language against a computational representation of linguistic cognition based on frames, their frame elements (FEs) and the relations between them. The analysis is attested through the annotation of sentences representing how lexical units (LUs) instantiate the frames they evoke. Qualia relations are proposed between words and model the various dimensions of the lexicon, including constitutive (the entity and its parts or material), formal (characteristics of the entity that distinguishes it in a larger domain, such as shape, dimensions, color, position, size), telic (the inherent purpose or function of the entity) and agentive (characteristics that make the entity come into existence) relations between lexical units. One example of a LU related to Tourism is map.n. It evokes the frame Text. Its Core Frame Element is Text itself, defined as the symbolic and linguistic information a topic. This LU map.n is linked via constitutive role to the substance paper.n, since it is made of paper. It has a prototypical function of guiding people. Thus, guide.v is telic of map.n. The agent that produces a map is a cartographer. We model cartographer.n as agentive of map.n. And finally, as a map is a type of text, the LU text.n is formal of map.n, since it distinguishes the map in a larger domain. This Tourism domain modeling based on frames and qualia can collaborate to establish semantic relations between words and contribute to better ways of representing information to the tourist, even in multimodal analysis. For this pilot study, we analyse a Brazilian television travel show called “Pedro pelo Mundo”, in which each episode, the presenter, Pedro Andrade, explores a city, region or country, highlighting its cultural and socioeconomic aspects. For the analysis, we annotate both the video and the audio superimposed in the episodes for the frames and qualia relations instantiated by both the text and the images. After the annotation is carried out, the resulting data for both semioses is compared for the (a)synchronization of the audiovisual combination. The preliminary results suggest that adding a multimodal domain to the linguistic layer of annotation and analysis contributes to enrich the kind of information that can be offered to tourists about attractions, although close attention must be paid to distinct patterns in which semantic models are instantiated by each semiosis.
FILLMORE, C. J. Frame semantics. In: Linguistics in the Morning Calm. Seoul, South Korea: Hanshin Publishing Co. 1982. pp. 111-137.
KRESS, G. R., & VAN LEEUWEN, T. (2006). Reading images: the grammar of visual design. London, Routledge.
PUSTEJOVSKY, J. The Generative Lexicon. Cambridge, USA: MIT Press. 1995.
Frederico Belcavello is a Ph.D. student in Linguistics at the Federal University of Juiz de Fora. Professor at the Centro de Ensino Superior of Juiz de Fora and a journalist at the Federal University of Juiz de Fora. He has experience in the area of Communication, with emphasis on Video Difusion, working mainly in the following subjects: television, TV genres, infotainment, documentary. Research Fields: Multimodal Corpora and Analysis. Frame Semantics. Computational Linguistics.
Alexandre Diniz da Costa is a Ph.D. student in Linguistics at the Federal University of Juiz de Fora. English Teacher at the Teacher Training School of the Federal University of Juiz de Fora. Research fields: Machine Translation. Qualia Relations. Frame Semantics. Computational Linguistics.
Vânia Gomes de Almeida is a Ph.D. student in Linguistics at the Federal University of Juiz de Fora. Research fields: Construction Grammar. Frame Semantics. Computational Linguistics.
Marcelo Viridiano is a Master’s student in Linguistics at the Federal University of Juiz de Fora. Research fields: Multimodal Corpora and Analysis. Frame Semantics. Computational Linguistics.
Prof. Dr. Tiago Timponi Torrent is a Professor of Linguistics at the Federal University of Juiz de Fora. He leads the research group FrameNet Brasil and coordinates, in the homonymous laboratory, the development of lexical and syntactic resources for Brazilian Portuguese. Research fields: Multimodal Corpora and Analysis. Construction Grammar. Frame Semantics. Computational Linguistics.