Social semiotic theory of multimodality focuses the person and the process of meaning-making, the social agency (Jewitt, Bezemer &
O’Halloran, 2016; Kress & Jewitt, 2003; Jewitt, 2009b), where the emphasis is on the sign-makers and their situated use of semiotic resources. There is an interest in understanding how people communicate and make meaning with a wide range of semiotic resources (Kress & Van Leeuwen, 2001). This focuses the question of what choices people make from the resources available in process of meaning-making. Krogh and Piekut (2015) investigate processes of
“voicing” viewed as agentive endeavours in writing, and stress that agency emphasises the writer’s subjective and transformative engagement with knowledge. Even though Krogh and Piekut approach agency from a different theoretical perspective, and in relation to students’ writing, they present an understanding of how agency is linked to the overall education aim of linking personal experience with a collective reality (Krogh & Piekut, 2015).
pre-given and fixed and not affected by its use. Jewitt, Bezemer, and O’Halloran exemplifies the semiotic resource of “length of shot”, or frame shots, in the following way:
For instance, the (material) resource of physical distance has been shaped by photographers and film-makers over time into the semiotic resource of ‘length of shot’. That resource is used to instantiate levels of social intimacy: a close-up is often used to suggest a ‘close’ social relationship – ‘intimacy’ or ‘intensity’ among other things, while a long shot tends to be used to suggest a more ’formal’,
‘absent’ or ‘distant’ social relationship. (Jewitt, Bezemer &
O’Halloran, 2016, p. 71)
A mode refers to a socially organised set of semiotic resources (Jewitt, Bezemer & O’Halloran, 2016). Bezemer and Kress (2008, p. 171) define a mode as a “socially and culturally shaped resource for meaning making”. Thus, modes are socially shaped and culturally given resources for meaning-making, encompassing a variety of things, including but not limited to print. Image, speech, gesture, and written language are examples of modes; and in order for something to count as a mode, a set of resources and organising principles that are recognised within a community is required (Jewitt, Bezemer &
O’Halloran, 2016). Throughout the transmediation process in this study, the students are involved in utilising different semiotic resources during the different phases of the videomaking process; for example the use of the mode written language and mediums of paper and pencil during the phase of writing the synopsis, or the use of the semiotic resources of sound and transition in the medium of digital video. Multimodal digital designing, then, extends beyond alphabetic print to include still and moving images, colour, and sounds.
A fundamental concept within the social semiotic theory of multimodality is design. Design refers to the focus of the sign maker in the meaning-making process; “[d]esign refers to how people make use of the resources that are available at a given moment in a specific communicational environment to realise their interests as makers of a message/text.” (Kress & Jewitt, 2003, p. 17). The notion of design recognises that meaning-making is about choosing and assembling resources according to individual interest and ideological position –
and positioning – as well as perception of audience and context. The terms aptness (Kress & Jewitt, 2003, p. 11) and agency also refer to the underlying principal of the relation between what is to be expressed and how. What establishes this relationship is the interest of the sign makers. Bezemer and Kress (2008, p. 174) propose design instead of composition to reflect a social shift, conceptually, from competence in a specific practice considered in terms of understanding to a focus on the interest and agency of the designer in the making of texts (see also Kress & Selander, 2012, p. 267). In social semiotic theory of multimodality the connection between meaning-making and the social interest of the sign makers is emphasised, which acknowledges the sign makers’ social agency (Kress & Jewitt, 2003; Jewitt, 2009b). Fulwiler and Middleton (2012) use compositing, referring to the process where novice filmmakers are able to combine different modes and semiotic resources into a seamless, complex, and rich whole; a process of compounding and mixing. With the terms compositing and recursivity, Fulwiler and Middleton question the sequential and linear process often connected with digital video composing as going from script to film to edit, rejecting a one-to-one relationship where still images are matched to a corresponding idea or word. Compositing refers to “layering”, which requires “a careful assessment of the multiple competing modes of meaning-making, and the ways they can be synthesized to create a specific, synchronous effect” (Fulwiler & Middleton, 2012, p. 43). It is in line with these definitions that I use the term multimodal designing; a process of active meaning-making utilising and layering a multiplicity of semiotic resources according to interest and ideological positioning as well as perception of audience and context.
Considering design as a forming of ideas in the shaping of new products, the understanding of design is first and foremost engaged with the idea that form precedes function (Kress & Selander, 2012).
To use this understanding of design in relation to a social phenomenon such as learning is, according to Kress and Selander, to make a categorical mistake. Instead, Kress and Selander emphasise what they refer to as interaction design processes where one not only focuses on products but also on the making of products together with
users. “This further emphasises an understanding of designs in learning as a central aspect to understand learning as meaning-making activities and engagement. It is about what takes place when human beings learn and how possible learning paths, including all kinds of choices and decisions are constructed” (Kress & Selander, 2012, p. 266, italics in original).
Design is the practice in which semiotic resources, the interests of the sign-maker(s) and the social context are brought together. From the perspective of the designer, it is a process of giving shape to the interests, purposes, and intentions in relation to the semiotic resources available for realising these purposes in a specific situation (Bezemer & Kress, 2008; Kress, 2010). Consequently, meaning-making is a process of design. This implies that there are not only individual aspects of design, but also cultural, social, and historical aspects of what is possible to express (Selander, 2011). In this understanding, design does not primarily point to aesthetic or artistic aspects, but to meaning-making and collaborative processes as well as performative and transformative activities (Selander, 2017).
Bazalgette and Buckingham (2013) are doubtful about the term design, which in their opinion implies a view of communication as a wholly rational, controlled process. They argue that to describe the meaning-making processes involved in film production as design limits our understanding of the plentiful creative, and indeed accidental and unexpected, discoveries involved in processes during the filmmaking. Additionally, they argue that multimodality cuts off consideration of the institutional, technical, economic and historical dimensions of these choices. Much of what Bazalgette and Buckingham call attention to is worth taking into consideration, and anyone who has taken part in activities inside a classroom will understand how many different factors, such as social hierarchies, group dynamics, economics, and pedagogical and personal values, come into play. But, that does not turn aside the possibilities of students’ agency and interest as a valuable and interesting analytical approach, just as it is valuable and important to study the intentions of teachers and the economical and structural conditions for teaching and learning.
Design draws attention to the affordances of the modes and media, which in the social semiotic theory of multimodality is referred to as modal and material affordance (Kress, 2003; 2010; Kress & Jewitt, 2003). Affordance refers to what is possible to express and represent with a mode, given its materiality and cultural and social history of that mode. Different modes of expression hold particular potentials and limitations for meaning-making. The modal affordance refers to the material features of the mode, whereas modes also have social, cultural, and historical aspects that affect how they are and can be used. The written text has possibilities and limitations, different affordances than the image. The written text and the image appear differently; written text uses temporal dimensions where something is expressed before something else, while the image uses spatial dimensions where everything appears simultaneously. Writing is structured by the logic of time; in writing some words precede others.
Meaning is therefore attached to the organisation of first, second, third – and last. The image, on the other hand, is organised by the logic of space and of the visual elements in spatially organised arrangements. Placing something centrally or above means that something else will be placed on the side or under, and all this can be used in the meaning-making of the text as a whole. That indicates the differences in the affordances of different media, the medial affordances. (Kress, 2003; 2010; Kress & Jewitt, 2003; Kress & van Leeuwen, 2006.)
The affordances of modes also bring forward what in social semiotic theory of multimodality is referred to as semiotic principle. The term refers to principles that apply across modes. Jewitt, Bezemer and O’Halloran (2016) exemplify the semiotic principle by explaining how different modes produce intensity.
For instance, all modes have resources for producing intensity. In the mode of speech, that is realized by the intensity of sound – ‘loudness’, it is also realized lexically, e.g. as ‘very’. In the mode of gesture, intensity might be realized by the speed of movement of the hand or by the extent of the movement. In the mode of colour, it might be done through degrees of saturation, and so on. (Jewitt, Bezemer &
O’Halloran, 2016, p. 62, italics in original)
In my understanding, the semiotic principle may serve as a reply to some of the critique aimed at the social semiotic theory of multimodality. Some researchers point out that the theorisation developed by Kress and van Leeuwen mostly deals with still images, photographs, or advertisements, and call for a development in relation to the special features of moving images (see e.g. Burn, 2013;
Burn & Parker, 2003; Halverson, 2010; Halverson, Bass & Woods, 2012). This is by all means a valid remark, and is further elaborated in the next section (3.2). But by applying the semiotic principle, the analytical concepts and theory developed by Kress and van Leeuwen (2006) can be related to the moving image just as well. Issues of contact, distance, and point of view prominent in the interactive level of the metafunctions, might be established by other modes than visual, such as sound, written text, or other resources of the kineikonic mode. Similarly, the analysis on the compositional level certainly applies to the kineikonic mode with the focus on sequencing for semiotic rhythm by the organisations of clips and transitions, as well as how salience might be created by the use of sound, visual effects, or gestures.
Representation, in the approach of multimodality, is not to be considered as a direct mirroring of the world, but a reflection of how we make meaning of and interpret the world (Selander, 2009; 2011).
Similarly, Hall (1997) considers representation as a process through which individuals in a cultural context use language to create meaning. Here language refers not only to linguistic language but also to modes such as image, body language, and music. Representation is therefore a central part of how meaning is created and mediated.
Meaning is something that is made, rather than something already fixed to be interpreted (Hall, 1997). Meaning is from this view always constructed and reconstructed according to time and space, and therefore changing. From this perspective, representation is understood as ways individuals choose to express their understanding of specific aspects of the world and transform them to their own representation (Selander, 2011). Halverson (2013) points out that whether one considers digital production as art or as literacy, representation lies at the core of its practices.
An essential aspect from a multimodal approach is not just the multiplicity of modes, but also how the modes interact and what this interaction creates. Multimodal ensemble refers to representations or communications that consist of more than one mode, brought together not randomly, but deliberately, to make meaning. The meaning maker “orchestrates” an ensemble that includes modes have been chosen with rhetorical intent for their affordances and in the interest of the meaning maker (Kress, 2010, p. 161). In a study on multimodal composing, Hull and Nelson (2005) discover how a unique synergy is created when multiple modes are combined in digital compositions. They argue that is in the “semiotic relationships between and among different, co-presented modes […] that the expressive power of multimodality resides (Hull & Nelson, 2005, p.
224). In their study they argue that multimodal composing is not simply an add-on art where images, words, and music are combined to increase the meaning-making potential of the text; rather they wish to highlight how multimodality affords not just a new way of making meaning, but a different kind of meaning (see also Fulwiler &
Middleton, 2012).
As the students in this study decide how to combine modes for a specific purpose, analysis of the moment-by-moment processes of multimodal designing enables the analytical focus of “unpacking”
how meanings are brought together, as well as the possibilities and constraints of the semiotic resources available. When the text shift from print-based to digital or screen-based, the semiotic resources used expand to include e.g. sound effects and moving images, and the complexity of multimodal ensembles clearly expands in the digital environments (see also Burn & Parker, 2001; Serafini, 2013). With a focus on students’ use of semiotic resources during a digital videomaking process, the next section elaborates the mode of the moving image: the kineikonic mode.