Methods for data analysis

The methods applied in the analyses pertaining to each of the seven overall research questions are explained in detail in the introduction to each of the analyses in Chapter 5.

Therefore, in the following, only some general reflections as to the data analysis are

provided. These include an elaboration on my definitions of the editing and checking phases and some overall considerations as to the mixing of the qualitative and quantitative parts of the study in terms of the data analysis. In Section 4.3.2.1, the so-called “observer effect” is discussed, and in Section 4.3.2.2, the procedure applied for transcribing both the semi-structured and the retrospective interviews is presented.

In Chapter 5, RQ1-RQ6, which explore TCI processes, are answered in an order which follows a workflow logic (cf. Figure 9). Thus, first, research questions RQ1, RQ1a, RQ2 and RQ3 which cast light on the editing phase of the translation processes are addressed. Then, RQ4 which addresses the checking phase is answered. Next, RQ5 which draws on data from both the editing and checking phases is addressed. As I established already in Section 1.2, I consider the editing phase to be the part of the MT-assisted TM translation process when the translators first evaluate the matches and, when necessary, modify them. Inspired by the drafting phase in Jakobsen’s (2002) tripartite model of the translation process as explained in Section 3.2.1, more specifically, I define the editing phase as the phase running from when the translator starts working with the first segment of the text until he or she finishes working with the last segment in the text. Inspired by Jakobsen’s end revision phase, in this thesis, by checking phase I mean the translator’s potential final examination of whether the target text is adequate, and more specifically, I consider this phase to run from the translator’s return to the first segment after the editing phase until he or she decides that the translation is finished. After having dealt with the research questions concerning the editing and checking phases, RQ6 is answered, which deals with the review part of the experimental study. As also mentioned in Section 1.2, review is defined as the examination of the translation conducted by a person other than the original translator which I regard to cover both bilingual (comparison of source and target text) and monolingual (review of target text) examination of the translation. Finally, the last part of Chapter 5 attends to RQ7, which explores the translators’ attitudes to TCI.

Figure 9. Structure of analyses in Chapter 5

Workflow

In mixed methods terms, the qualitative and quantitative parts of the study are not only combined at the design and data collection stages as explained earlier in this chapter, but also at the data analysis and interpretation stages. At the data analysis stage, the two forms of data are integrated by, for example, cross-checking a qualitative analysis of the screen recordings with quantitative data from keystroke logging, and through data conversion. In terms of data conversion, when addressing RQ1-RQ6, some of the qualitative data are quantitized. When relevant, this will be addressed further in the analyses pertaining to the individual research questions. Following the process of data conversion, these data are analyzed by means of descriptive statistics. This is in line with the purposive sampling technique as well as the research questions which are largely explorative and descriptive in nature (Onwuegbuzie & Leech 2005, p.287; Teddlie & Tashakkori 2009, pp.23–24). The integration of qualitative and quantitative parts at the interpretation stage is accomplished in part in the syntheses following the individual analyses in Chapter 5, and in part in the discussion section in Chapter 6.

As indicated above, the analyses mainly draw on data generated as part of the experimental study. However, the contextualization which precedes the individual analyses in Chapter 5 draws on the data collected as part of the contextual study, as does the analysis addressing RQ7. In the remaining analyses, the data from the contextual study are brought into play when I find them to inform the analyses. Since, as mentioned in Section 4.3.1.1.3.3,

Translator F accidentally deleted his screen recording file after the experiment, data from his translation processes are only included in analyses that do not draw on screen recording data.

4.3.2.1 The observer effect

A threat to the validity of the collected data is the possibility that those I observed at

TextMinded reacted atypically because of my presence. Daymon and Holloway (2011, p.275) refer to this as the “observer effect” and O’Brien (2009, p.258) calls it the “white coat effect”. Saldanha and O’Brien (2013, pp.29–32) also point to the risk that people change their normal behaviour, usually by improving it, when they know that they are being studied, referring to this as the “Hawthorne effect”. Specifically with regard to TPR, Hansen identifies this awareness of being studied as a weakness. She states that “[k]nowing about the experiment and feeling observed may have an impact on the results because

nervousness or stress changes the participants’ mental processes and the intensity of this impact cannot be measured precisely” (Hansen 2013b, p.97). The best we can do is to reflect on the impact this may have on our findings. Daymon and Holloway note that the observer effect “tends to disappear the longer you are able to spend in the research setting” (2011, p.275). In the current thesis, the hope was that my regular visits at TextMinded as part of the preparations for the study and my presence at TextMinded for several weeks would help the employees to go about their tasks in their accustomed ways. A risk that is particularly relevant for this study was that the fact that the study was conducted around the time when MT was first implemented at TextMinded, and that this could impact on the translators’

behaviour. I never specifically stated that I was particularly interested in MT, but at least some of the translators expected me to be so. This might have led them to tone down their scepticism out of politeness, or it might have reinforced negative attitudes to MT if the

translators took the experiment to be an opportunity to voice their scepticism and potentially influence the decision of implementing MT. In relation to the experiment, the translators’ and reviewers’ work may also have been influenced by the fact that the translation was not to be delivered to the client afterwards (cf. Mossop 2007a, p.17).

O’Brien (2009, pp.258–259) states that participants will sometimes voluntarily inform the researcher that they have behaved differently than normal. However, in the current study, considerations as to the observer effect led to the specific formulation of a question in the post-experimental questionnaire which asked the translators whether or not they thought their processes had been influenced by the fact that they were participating in an

experiment and asked them to explain why this was or was not so. As shown in Figure 10, four answered “yes”, three translators answered “no”, and one answered “no” and added

“maybe a little”. Translators E and H answered “yes” and explained that they did not let the process be interrupted in different ways. Translator F also answered “yes” and explained that he might have put a little more thought into his solutions than usual. Thus, this indicates that he attempted to improve his performance, as suggested by Saldanha and O’Brien. Translator A also answered “yes”, but her explanation was somewhat ambiguous.

On the one hand, she stated that she spent less time checking terms and polishing expressions than usual and said that she would usually also have consulted one of her colleagues. This suggests that participation actually led to a drop in her performance. On the other hand, she stated that in terms of her “approach”, she worked as she normally did. It is unclear what she meant by this. Translator C answered “no” and added “maybe a little”. She stated that she worked as she usually would, but interestingly, she also stated that she might have been affected by the time factor, but that she “managed to finish in time”. The translators were at no point informed that they had a restricted amount of time to finish the translations, but because of their professional schedules, we had to book time in their calendars for participating in the experiment. As this was estimated to take three hours in total, she must have had an impression that she had to finish within a certain time. The remaining three translators (B, D and G) answered “no”, and Translator G added that during the retrospective interview, he could see that he had worked as he normally would. None of the translators mentioned that the presence of MT influenced their way of working.

Further, it is evident from my field notes (and my memory) that the employees at

TextMinded were very open and welcoming towards me. Several times in my field notes I have noted that I felt at home and that, towards the end of my stay in particular, the employees asked me whether I was coming back and that I was always very welcome. I take that as an expression that they felt at ease with me being there. I expect this to have impacted positively on them behaving naturally in my presence as opposed to if I had only come in during the week when the experiment was conducted.

Figure 10. The observer effect - answers to question 12 in the post-experimental questionnaire

4.3.2.2 Transcription of interviews

The semi-structured interviews and retrospective interviews from the experiment were transcribed on the basis of the audio files. According to Kvale and Brinkmann (2009a, p.199 ff.), transcription is a process of interpretation and is thus the first part of the analytical process. In fact, Kvale and Brinkmann (2009a, p.200) state that transcription is a translation from spoken to written language which results in an impoverished and decontextualized rendition of a conversation. In line with this, Halkier (2008, p.70) states that transcription is a reduction of data, but nevertheless a necessary one.

The choice of who should do the transcription has been discussed (cf. e.g. Gibbs 2007, pp.15–17; Kvale & Brinkmann 2009a, p.202). Gibbs states that it is an advantage for the researcher if s/he does the transcription him/herself, because it is an opportunity to start the data analysis. However, he also states that it also is a good option to employ someone else to transcribe if the audio files are easily understandable, and that it is best if the transcriber knows something about the subject matter and the context of the interviews.

Post-experimental questionnaire, question 12 (translated from Danish):

Do you think the way you worked today was influenced by the fact that you knew that you participated in an experiment (tick the appropriate box)?

Yes No

Explain here why you think that it did/did not influence your way of working:

Translator answers (translated from Danish):

Translator A: Yes – I spent less time on checking terms and polishing expressions than I would have done if the translation had to be delivered to the client. Normally, I would also have asked one of my colleagues for advice on some of the expressions which caused problems. But in terms of my approach there was no difference between a “real” translation situation and the experiment.

Translator B: No

Translator C: No (maybe a little) – I have worked in the same way as I normally do, made the same considerations, but the time factor might have played a role. However, I managed to finish in time.

Translator D: No

Translator E: Yes – If it had not been an experiment, I would probably have taken a small break.

Translator F: Yes – Maybe a little more attention to solutions. A little more thought was given to alternatives.

Translator G: No – I deliberately tried to work as I normally would and in the following talk I could see that I succeeded in that.

Translator H: Yes – I did not let myself be interrupted by colleagues/e-mails.

Kvale and Brinkmann state that in most interview studies, a secretary would do the

transcriptions, and that researchers who are particularly interested in communication form and language style can choose to transcribe themselves in order to capture the many details relevant to their specific analyses. In the current thesis, the interviews were first transcribed by a Masters student in Translation and Interpreting at the Department of Business

Communication, Aarhus University. I then double-checked all of the transcriptions while listening to the audio files, making a limited number of amendments.

It is generally agreed that the level of detail in transcriptions should be determined by the purpose of the study in question (Gibbs 2007, p.10ff.; Koskinen 2008, p.88; Kvale &

Brinkmann 2009a, p.202ff.). Like Koskinen (2008) and Jensen (2013) in their analyses of focus group interviews, I tried to balance analytic needs and readability by keeping

transcribing conventions to a minimum. Thus, the interviews were transcribed in full and, for example, “pauses and overlaps are roughly marked, but their durations are not calculated, and phonetic features such as pitch and intonation are unmarked” (Koskinen 2008, p.88).

This also means that I have chosen not to follow more complex systems such as the Text Encoding Initiative (TEI) and the Gesprächsanalytisches Transkriptionssystem (GAT)

described by Göpferich (2008; 2010). The TEI system comprises guidelines on transcribing in a machine-readable format so that transcripts can be analyzed automatically using

electronic support, which is not relevant for me in this context. Further, I found the GAT system too detailed in terms of my analytic needs; however, there are overlaps between the features marked in the transcriptions in the current study and the features marked when following the GAT system.

Following Kvale and Brinkmann’s recommendation (2009a, p.203), the transcriber was provided with explicit guidelines asking her to transcribe all words as they were said by the respondent (i.e. that she should not “tidy up” their speech if it did not follow the

grammatical rules used in writing), and to mark any spot that she found difficult to hear instead of guessing so that I could pay particular attention to those spots when checking the transcripts. She was also provided with a list of transcription symbols (cf. Figure 11) and signed a non-disclosure agreement.

Figure 11. Transcription symbols