Bringing forth a world

In this first part, an array of loosely coupled theories is presented, which go by the acronym 4EA (e.g., Protevi, 2010; Reinhardt & Loke, 2013; Ward & Stapleton, 2012) and can all be counted as part of the third paradigm. The acronym covers the terms “embodied, embedded,

69 extended, enactive, and affective.” To these I also count “distributed” and “situated” (for an overview of these theories, see for instance Menary, 2010a; Pasquinelli, 2004a, 2004b;

Ward & Stapleton, 2012). The theories have one common feature that should be singled out in relation to interfaces. All of them, each in its own way, promote the idea that meaning is created not found. In terms of cognition, this is the question of how a meaningful world arises or is “brought forth.” This particular phrase is an allusion to Maturana (2002, p. 31) who states that we “continuously bring forth the world and worlds that we live” (see also Varela & Dupuy, 1992). The everyday world we experience may seem given, but it is in fact the end point of our perception and cognition, not the starting point. Fritjof Capra (1996) expresses it in this way:

There are no objectively existing structures; there is no pregiven territory of which we can make a map – the map making itself brings forth the features of the territory. We know that cats or birds will see trees, for example, very differently from the way we do, because they perceive light in different frequency ranges. Thus the shapes and textures of the ‘trees’ they bring forth will be different from ours. When we see a tree, we are not inventing reality. But the ways in which we delineate objects and identify patterns out of the multitude of sensory inputs we receive depends on our physical constitution. As Maturana and Varela (1980) would say, the ways in which we can couple structurally to our environment, and thus the world we bring forth, depend on our own structure. (p. 271)

But it is essential to understand that theories are not answers to a general question of how things come to be. Rather, their claim is that the world (meaning) as we know it is shaped, in a non-trivial way, by particular constraints, dependencies, and conditions because these are part and parcel of its (the world’s) enactment. To paraphrase Capra (1996), we see the world as we do because we are human (and the world is the world). The world emerges through the consummation of actions in the face of constraints particular to us. As Capra writes, “we are not inventing reality.” (p. 271)

It is difficult to provide a strict demarcation of what to include in the second wave, and it is even more difficult to find commonalities in what one includes. The theories overlap to a great extent on various points, but it is far easier to characterize what they are opposed to than that in which they are united. For instance, Bødker (2006) counts situated action, distributed cognition, and activity theory as part of the motley crew of theoretical ideas that challenge cognitivism. Several of the authors referred to could easily fit under more than one header. Most of the authors writing under any of the headings include parts or all of the other approaches in their review of the relevant literature (e.g., Sutton, 2006). Harrison et al.

(2007) recount a telling story of their problems in finding an appropriate name for the third paradigm. They tried “embodied and situated,” “meaning-making,” “post-cognitive,” etc., before settling on “third paradigm.” The trouble is that, as with any other antithetical endeavor, it is a lot easier to identify the elements in cognitivism that these various theories oppose than it is to identify commonalities across the alternative answers and approaches they offer.

70 In very broad terms, all of the theories touched upon can be said to lend support to the rejection of two elements of cognitivism. The first is the placement of cognition and perception inside an isolated mind or a singular subject. The second is the rejection of “the idea of a world or environment with extrinsic, pregiven features that are recovered through a process of representation” (Varela et al., 1991, p. 138). Rejecting one of the above automatically puts the other into question, but there is no set way in which the rejection of either (or both) plays out. The only certainty is that the interface is not to be considered a border between two separate entities. In Bateson’s (1972) words:

The mental world – the mind – the world of information processing – is not limited by the skin. (p. 460)⁵⁶

While Bateson uses the word “mind,” for this purpose, “meaning” is more suitable. His statement is the rejection of the proposition that meaning originates from the mind of a subject (social construction) or from the objective world (objectivism). In fact, it is a misunderstanding to enclose meaning “inside” entities at all. Meaning is what is produced, created, or what takes place when borders are crossed. Thus, interaction in relation to HCI means the production of meaning. In addition to the two aforementioned points, other themes can be found that vary in emphasis: the opposition to the representational aspects of cognitivism (e.g., Chemero, 2009; Varela et al., 1991); the view that man’s body is an essential part of cognition (e.g., Lakoff & Johnson, 1999; Shapiro, 2011); the idea that meaning is always bound to a specific situation (Clancey, 1997; Suchman, 2007); and the premise that cognition is always enacted (Noë, 2004; Varela et al., 1991). Throughout, there is a strong philosophical influence from phenomenology and hermeneutics (Heidegger, 1962; Merleau-Ponty, 1962), sometimes indirectly, but often directly by philosophically trained thinkers (e.g., Dreyfus, 2002; Mingers, 2001).

Enaction

The first constraint to consider is the performative aspect itself. This constraint is crucial because it turns our assumptions on the head. The world is something we enact or perform rather than “the something” we do something with or to. Barad (2007, p. 49) states that knowing is the result of a direct material engagement with the world. Tweaking this statement, we should say that the world is the result of a direct (or active) material engagement.⁵⁷ The crucial thing to point out is that perceiving, meaning, and knowing are not about the world. They are not separate phenomena that take place in a realm different from the result. Rather, the phenomena resulting from them are ongoing and only exist qua their enactment. “The meaningful is not in our mind or brain, but is instead essentially worldly...we do not store the meaningful inside ourselves, but rather live and are at home in

56 The quote can be read in a variety of ways. I tend to read it along the same lines as Ingold (2000) where the statement brings the entire question of “placing” the mind anywhere into doubt: “If mind is anywhere, then, it is not ‘inside the head’ rather than ‘out there’ in the world….The mind, Bateson had always insisted, is not limited by the skin” (p. 3; see also note 24).

57 There is a growing contingent of research on the significance of matter and materiality (see Carlile, Nicolini, Langley, & Tsoukas, 2013; Leonardi et al., 2012; Malafouris, 2013).

71 it...the meaningful is the world itself,” as Haugeland (1998, p. 231) puts it. This is important for something as trivial as our assumptions about the relationship between a process and its result. In articles 1 and 2, I argue the case that phenomena cannot represent one another and that such pairs as process/result and map/territory are in fact commutable versions or transforms of each other. The point is not that there are two phenomena that can be exchanged for one another; it is that there is only a single phenomenon that is captured (described) in two different ways (process and result) and then attempted and compared. We can capture the phenomenon in both ways, but that does not mean that we end up with two phenomena. When we consider an enactive or performative process, we are therefore not considering something (a process) different from the result. Rather, we consider the enaction because it matters what we do in relation to the world we encounter. The world is enacted.

This is what is meant by “performative” and “enactive” (Stewart et al., 2010; Varela et al., 1991). The small prefix “en-” allows us to discern the reference to “something” that is acted out. We might say that just as there are no “neutral,” pregiven features or objects in the world, there are no neutral processes that simply carry information. “Organisms do not passively receive information from their environments, which they then translate into internal representations. Natural cognitive systems...enact a world” (Di Paolo, Rohde, & De Jaegher, 2014, p. 39). There are differences in how you perceive the world when you are bicycling as opposed to walking or flying – in the dual sense of being different processes – and seeing different things (e.g., a bump on the road has different affordances relative to your mode of transportation).

Enactivism is obviously indebted to pragmatism, as developed by William James, Charles Sanders Peirce, and John Dewey (see Hookway, 2015 for an introduction).⁵⁸ It has been proclaimed that there has been both a “Practice Turn” in sociology (Nicolini, 2013;

Schatzki, Knorr Cetina, & von Savigny, 2001) and a “Pragmatic Turn” in cognitive science (Engel, Maye, Kurthen, & König, 2013). This glove has been picked up in HCI (Kuutti &

Bannon, 2014), and Peirce (1878) is credited with coining the maxim that makes the connection with enactivism:

Consider what effects, which might conceivably have practical bearings, we conceive the object of our conception to have. Then, our conception of those effects is the whole of our conception of the object. (p. 293)

Like Fontana’s cut, we recognize processes in their consequences. Seeing a thing is not simply the observation of, say, a physical object. A thing is itself a consequence of an act, and we recognize the act in it. The same idea can be found in Leonardi et al. (2012) who explicitly attempt to understand technologies in terms of their consequences. Just as the world is not simply there prior to action, our actions are not “there” to be applied prior to our meeting with the world. The material informs us, the actors, of our capabilities.

58 There are also obvious affinities with the late Wittgenstein’s (1997) parole “meaning is use” and speech act theory (Austin, 1962; Searle, 1999) that are not considered here.

72 Rather than being simply constrained by structure, as the typical conventional interpretive understanding wants us to believe, human choice and agency are made originally possible through the very resources that objects and structures dispose. (p.

10)

Pondering the idea that what you see is a product of a process allows the reflexive realization that the process that allows you to see what you see now must have already taken place. The enactive approach is therefore the most direct route to the reflexive problems discussed above as well as the idea of “zero distance.” Being able to see what is in front of you as the result of a process that has already passed or is still in progress is a way of reverse engineering the processes of your thought. It is a way of recognizing what is in front of you as the endpoint of those processes.

Relative to interfaces, this idea translates into the conception that interfaces are not located anywhere in particular but are instead “anywhere” meaning is created. Drucker (2011b) creates the link to phenomenology and hermeneutics when she defines interface as “a space that supports interpretative events and acts of meaning production” (p. 3).

The Perceptual Robotics Laboratory is a bit more heavy-handed in scoping out the aim of the ENACTIVE Network that structures the research on enactive interfaces:

Enactive Interfaces are related to a fundamental ‘interaction’ concept which is not exploited by most of the existing human-computer interface technologies....enactive knowledge is a form of knowledge based on the active use of the hand for apprehension tasks. (Percro, 2015, para 1)

Interaction here becomes synonymous with the “active use of the hand.” In this framing, the

“enactive approach” is about a particular type of knowledge that is set apart from traditional symbolic or iconic knowledge. The case can be made that the hand “possesses its own know-how” (Radman, 2013, p. 384) and that there are particularities to our world that are entirely dependent on humans having hands (Wilson, 1998). However, with the above quote, we have drifted away from considerations of enaction toward examining another type of constraint. In a generalized way, “the hand” becomes a metonymic term for the idea that our world is always constrained by the fact that we all have bodies, something most of us can agree on.

Embodied and embedded

Wilson and Foglia (2011, para. 23) succinctly sum up the connection between enactment and embodiment:

[T]he world that is given and experienced is not only conditioned by the neural activity of the subject, but is essentially enacted in that it emerges through the bodily activities of the organism.

73 Embodied cognition (Chemero, 2009; Lakoff & Johnson, 1999; Mingers, 2001; Shapiro, 2011), sometimes referred to as embodied embedded cognition (EEC) (e.g., Haugeland, 1998), posits that cognition depends on having a body (e.g., sensorimotor capacities) and, furthermore, that the body (these capacities) is embedded in a larger context (social and natural environment) such that consciousness and intelligent behavior emerge in the interaction between the brain, body, and world. The pivotal idea developed by Merleau-Ponty (1962) is that perceptions are inherently meaningful to us because we have bodies (Mingers, 2001, p. 112). In terms of process and pattern, this is a way of saying that the body puts restraints on the cognitive processes in a manner that is discernible in the resulting patterns of perception. An example is Gibson’s (1986) idea of affordances, examined below, whereby the scale of the body, say the length of the legs, puts constraints on how high a step one can take and, consequently, the perception of what is “step-up-able”

(Warren, 1984).

Embedded or environmental constraints are relatively straightforward to understand. We can see that different environments look, feel, and impact us differently (put different strains on us). We are able to trace the body in the meaningful world we engage in. A puddle of water is “jump-over-able” relative to our physique. Most people will also be able to follow a more complex version of the Darwinian idea of environmental pressure that drives the development of different body types.⁵⁹ It is a bit more difficult to comprehend the idea, let alone translate it into design ideas, that if you had a different body, the world would literally appear different. To make matters worse, there are many different conceptions of embodiment, embeddedness, and what a body is (Ziemke, 2003). As the different ideas find their way into the HCI environment, the reception of the embodied-embedded perspective takes equally disparate forms. For example, Dourish (1999, 2001) traces the philosophical roots of embodiment and argues for a better understanding of human interaction with a system as a fundamentally embodied phenomenon. In a bachelor’s thesis from the University of Gothenburg (Weschke & Börman, 2009), Dourish’s ideas are given a very literal interpretation as a transition from “traditional interaction” with a desktop environment, to a broadening of scope, to “interaction with everyday objects such as interactive whiteboards, furniture, spoons, cups, or even digital jewellery” (p. 1). This interpretation is not “wrong.” Dourish’s philosophical ideas are just somewhat removed from a “design mode” that speaks of creating “more natural interfaces by incorporating social and/or physical skills in order to create a more meaningful interaction space, allowing participants to engage in systems in a more natural way” (p. 2). A different example is Hurtienne (2009), who employs concepts such as “image schemas” and “mental models.”

This is surprising since such concepts go against the non-representationalist aspects of the 4EA literature. The fundamental notion at stake here, that bodily and environmental

59 There is a more complex meaning of embeddedness at stake. When we engage in the world, the material

“kicks back” informs us of our own capabilities. “[W]ork materials are more than mere stimuli for a disembodied cognitive system. Work materials from time to time become elements of the cognitive system itself. Just as a blind person’s cane or a cell biologist’s microscope is a central part of the way they perceive the world, so well-designed work materials become integrated into the way people think, see, and control activities, part of the distributed system of cognitive control” (Hollan, Hutchins, & Kirsch, 2000, p. 178). This sense is investigated in article 3.

74 constraints have an effect on the meaningful world we experience, becomes somewhat obfuscated by the fact that different constraints can be accepted in different ways and without buying into the entire 4EA package. There are very innovative ideas in circulation in the HCI community that start from embodied interaction (e.g., Williams, Kabisch, &

Dourish, 2005), but no clear direction from applying the approach has emerged.

Extended

One reason that the different theories have not been united under an acronym and do not cluster under a single headline is that their sometimes overlapping ambitions have very different foci. The embodied-embedded position points to cognitive processes that are constrained by intrinsic bodily dynamics as well as external structures that constrain and mold the cognitive process at play. The extended cognition thesis (Clark, 1998; Menary, 2010b; Noë, 2004) says basically the same thing, albeit with respect to breaking down the strict separation of the mind and world. The mind is not limited by the skin, as Bateson (1972) maintains. It extends beyond the limits of the skull, the skin, and the body.

Conversely, the environment is not just an “outside” we meet from inside. It is implicated in our cognitive processes in an indispensable way. Clark (1998) borrows the term

“scaffolding” from the Soviet psychologist Lev Vygotsky (1986). Together with Chalmers (2011), he gives the example of the game Tetris wherein the physical rotation of a block with a button can be executed more than three times as fast as the “same” action performed in the mind as a “mental rotation.” In other words, the mind is designed to work with an environment, not to make representations of it. This makes sense from an “economical”

standpoint. It is very inefficient to create a system with the responsibility of creating authentic copies or representations of the world. A much better idea is to create a system that takes advantage of the world’s features and allows it to carry the burden of processing.

As Brooks (1990, p. 5) famously said, “the world is its own best representation.” Clark (1989) calls this the “007 principle”:

In general, evolved creatures will neither store nor process information in costly ways when they can use the structure of the environment and their operations upon it as a convenient stand-in for the information-processing operations concerned. That is, know only as much as you need to know to get the job done. (p. 64)

This is an important transgression of the traditional border between the subjective and the objective. Instead of having two separate “areas” or things, each responsible for different effects on the meaningful world that appears to us, the two are inextricably co-conditioned.

This idea is indirectly what article 2 is about. Clark and Chalmers (2011) provide an analogy:

The extraordinary efficiency of the fish as a swimming device is partly due, it now seems, to an evolved capacity to couple its swimming behaviors to the pools of external kinetic energy found as swirls, eddies and vortices in its watery environment.

These vortices include both naturally occurring ones (e.g. where water hits a rock) and

These vortices include both naturally occurring ones (e.g. where water hits a rock) and