rooted in an artistic, art-based, multimodal approach (see Heilä-Ylikallio, Østern, Kaihovirta-Rosvik & Rantala, 2004; 2005; 2007;
Kaihovirta-Rosvik, Østern, Rantala & Heilä-Ylikallio, 2011;
Kaihovirta-Rosvik, Østern & Ylikallio, 2011; Østern & Heilä-Ylikallio, 2008; Østern, Heilä-Heilä-Ylikallio, Kaihovirta-Rosvik & Rantala, 2010). This approach can therefore be considered to some extent established in literature education for younger students, but is rarely explored for adolescents in secondary education (see also Lewis &
Dockter, 2011; Lundström & Olin-Scheller, 2014).
Albers (2009) explains this by arguing that students “create symbolic, metaphoric and literal messages that point to their interpretation of texts, their connection to a text, and what they want the viewer to know about their reading of this text” (Albers, 2009, p. 8).
Transmediation offers possibilities for negotiations of multiple interpretations and representation, and thus “has the potential to capture the postmodern reality of multiple texts, multiple meanings, and multiple interpretations” (Semali & Fueyo, 2001, n.p.). In my understanding, this echoes a performative approach to literary interpretation, reading, and text.
Research on transmediation (Carey, 2012; McCormick, 2011;
Oldakowski, 2011; Siegel, 1995; Smagorinsky & O’Donnell, 1998;
Suhor, 1984; Whitin, 2005) has demonstrated how students who recast meanings from one mode or sign system into another, expand the interpretive potential of the text under examination.
Transmediation’s ability to enable students to create new meaning in another sign system is what scholars have referred to as its non-redundant potential; each form or representation we engage in uses its own features on the meaning we make or interpret (Zoss, 2009).
Thus transmediation promotes new ideas potentially unavailable in other semiotic systems. Smagorinsky and O´Donnell-Allen (1998) show how students who transmediate new understandings of literary texts take ownership of their own learning by semiotically mediating their understandings of texts, interpreting difficult texts using contemporary stances toward literature, and connecting their own lives to the contexts of the characters represented in literary texts.
Mills (2011a, p. 6) describes three key principles of transmediation regarding children’s multimodal and digital meaning-making; the process of knowledge transformation; continual revision of intents for representing knowledge in response to the possibilities and constraints of sign-making systems; and the centrality of digital text production with its potential to convert semiotic content via the discrete sign-making system inherent in software interfaces.
Suhor (1984) warns that, like any pedagogical process, transmediation could be approached superficially, and distinguishes
between “literal” and “imaginative” forms of transmediation, the latter of which results in generative meaning-making. Literal transmediation, where concepts are merely reproduced rather than explored in another sign system, does not encourage students to engage in a reflective process. Transmediation, as an act of translating meaning from one sign system to another, is considered to increases opportunities for generative and reflective thinking (Siegel, 1995).
Semali (2002) also uses transmediation to describe the process whereby one’s negotiation with texts is represented in new text forms through other sign systems, and discusses how that process supports students in more complex thinking.
Given this potential, transmediation appears to enable critical examinations of literary texts since mediating across sign systems can illuminate such textual elements as underlying values of the text creator(s), hidden biases, and cultural symbols, as well as the values, biases, and cultural lenses of the readers themselves (Hadjioannou &
Hutchinson, 2014). This way the approach of transmediation can bridge the analyses of the digital videomaking process and literature interpretation, and offer an interpretive approach in relation to literary reading and literature education.
The concept of transmediation has common features with several concepts developed within theoretical approaches to multimodality;
there are several concepts that try to explain the shifting of sign systems. The concept of transmediation has common features with the term transduction developed within the social semiotic theory of multimodality (Kress, 2003; 2010), which refers to remaking meaning across modes. The process of transmediation is also described using terms like transformation (Kress & Selander, 2012), resemiotisation (Iedema, 2001b) and re-design (Cope & Kalantzis, 2000), where every transformation and transduction is viewed as a creative act of designing and re-designing. Transformation, transduction, resemiotisation, and re-design all produce changes in meaning (Kress, 2008; Lindstrand & Selander, 2009). The representation and communication through a series of different modes is also referred to as semiotic chain (Stein, 2008); Semali and Fueyo (2001) use the term transmedial experiences, by which they refer to the abilities to engage
in multiple ways of mediating knowing between sign systems. The different sign systems become alternative ways of seeing, knowing and expressing ideas.
The choice of transmediation rather any other closely related concept is grounded in its use in relation to literature reading, and specifically poetry reading, in several previous studies (see e.g. Albers, 2009;
Carey, 2012; Hadjioannou & Hutchinson, 2014; McCormick, 2011;
McVee, Bailey & Shanahan, 2008; Mills, 2011a; Siegel, 1995;
Oldakowski, 2011; Whitin, 2002; 2005). In this study the concept of transmediation is used to interpret the findings of the analyses of the students’ digital video and the process of digital video designing, with the intent of considering how the literary text is explored, reviewed, and negotiated throughout the process. In order to analytically explore the transmediation process from poem to digital video, I theoretically need an approach that takes into consideration multiple modes of meaning-making. The next chapter therefore deals with the theoretical underpinnings of a multimodal approach.
3 MULTIMODALITY:
A SOCIAL SEMIOTIC APPROACH
This chapter is the second of two establishing the theoretical framework of the study, and deals with social semiotic theory of multimodality. Initially, I discuss some central theoretical concepts and how they are to be understood in the study, including a discussion of multimodal designing as a meaning-making process (3.1). Following this, I present and discuss the kineikonic mode, the mode of the moving image (3.2). The approach of social semiotic theory of multimodality offers an opportunity to examine analytically both how semiotic resources are used in representing multimodally with the kineikonic mode, and students’ negotiations of the affordances and constraints of the different modes and mediums during the process of videomaking. Accordingly, I present the metafunctions of text (3.3) and the strata of text production (3.4) to explain how these theoretical approaches inform such an analytical angle.
Multimodality takes the approach that meaning is made out of a multiplicity of modes. The starting point of multimodality is that representation, communication, and interaction consist of a multiple of modes, all with the potential to make meaning. Additionally, all modes are formed by the cultural, historical, and social context they are part of (Jewitt, 2009a; Kress, 2003; 2010), meaning that they will change over time and with use. The idea of meaning-making with several modes simultaneously is not new; disciplines such as
linguistics and semiotics have studied different forms of meaning-making long before the term multimodality came to being. However, the use of the term and the theorising around the notion of multimodality have grown remarkably during the past two decades.
What multimodality specifically wants to turn attention to is how different resources for making meaning are not separated but combined into an integrated multimodal ensemble (Jewitt, Bezemer
& O’Halloran, 2016). This fact becomes even more noticeable with the development of digital technologies, which enable people to combine resources more easily and affordably than before. A relevant example is the moving image (Burn, 2013; Burn & Parker, 2001;
Jewitt, Bezemer & O’Halloran, 2016). Pointing at a constellation of technological, economic, and social changes, the literature around the concept of multimodality traces the start to 1990s (Jewitt, 2008;
Jewitt, Bezemer & O’Halloran, 2016), with the New London Group´s (1996) manifesto on a pedagogy for multiliteracies and Kress and van Leeuwen´s (1996) grammar of visual design. Design was an essential concept to both publications and was used to signify the process of designing meaning, as well as the product (Siegel & Panofsky, 2009, p.
100).
Multimodality is by now a commonly used concept, and is rather a field of research than a theory or discipline (Jewitt, 2009a; Kress, 2010). As such, it can be approached using different theoretical perspectives, which has contributed to an actively evolving area of research (see e.g. Jewitt, 2009c). Jewitt (2009a) compares the approach of multimodality to the definition of ethnography by Green and Bloom, who make a distinction between doing ethnography, taking an ethnographic perspective, and using ethnographic research tools. Jewitt, Bezemer, and O’Halloran (2016, pp. 5–6) distinguish between doing multimodality and adopting multimodal concepts. This study I categorise as doing multimodality since multimodality is significant both theoretically and with regard to research design.
Multimodality is applied by using social semiotic theory of multimodality to adopt a lens for exploring students’ multimodal designing in videomaking practices using several semiotic resources
to represent and negotiate their interpretation of a poem.
Multimodality is also applied in the methodological approach in the research design because of the deliberate choice to enable and acknowledge students’ possibilities to represent and explore their interpretive work of literature using a variety of semiotic resources.
Despite the range of perspectives and theories within the field of multimodality, there are some common essential principles. These are: (a) representation and communication always draw on a multiplicity of semiotic resources, all with equal potential to contribute to meaning; (b) all resources have been shaped through their cultural, historical, and social use and are not fixed but articulated and situated; (c) and people orchestrate meaning though their choice and configuration of semiotic resources (Jewitt, 2009b, pp. 14–15). Nevertheless, the variation of theories and perspectives within the field of multimodality requires a clear positioning. In this study, social semiotic theory of multimodality is used because of its emphasis on the agency of sign makers and its focus on modes and their affordances, as well as the social uses and needs they fulfil (Jewitt, Bezemer & O’Halloran, 2016).
Social semiotic theory of multimodality is strongly associated with the works of Kress and van Leeuwen (Kress & van Leeuwen, 2001;
1996/2006; see also Jewitt, Bezemer & O’Halloran, 2016). It takes Halliday’s theories of social semiotics and system functional grammar as a starting point, which Kress and van Leeuwen developed and extended to a range of modes (Jewitt, 2009b, p. 29). Hence, multimodality indicates what is attended, namely all the modes that appear in multimodal ensembles, whereas social semiotics provides the theoretical and analytical tools (Kress, 2008, p. 92). However, Machin (2009) is sceptical about the application of a theory from linguistics in trying to explain visual communication, or any other communicative mode. He acknowledges several well-made attempts to highlight multimodality’s potential to allow us to think more of the communicative function of images, but calls for a need to be aware of work on visual communication if we are to develop a more robust multimodality.
Social semiotic theory of multimodality focuses the person and the process of meaning-making, the social agency (Jewitt, Bezemer &
O’Halloran, 2016; Kress & Jewitt, 2003; Jewitt, 2009b), where the emphasis is on the sign-makers and their situated use of semiotic resources. There is an interest in understanding how people communicate and make meaning with a wide range of semiotic resources (Kress & Van Leeuwen, 2001). This focuses the question of what choices people make from the resources available in process of meaning-making. Krogh and Piekut (2015) investigate processes of
“voicing” viewed as agentive endeavours in writing, and stress that agency emphasises the writer’s subjective and transformative engagement with knowledge. Even though Krogh and Piekut approach agency from a different theoretical perspective, and in relation to students’ writing, they present an understanding of how agency is linked to the overall education aim of linking personal experience with a collective reality (Krogh & Piekut, 2015).