The experimental study

In order to explore translators’ interaction with an MT-assisted TM system and their attitudes to this interaction, an experiment was conducted at TextMinded. This experiment consisted of an MT-assisted TM translation part conducted in May 2013, where eight translators were asked to translate the same two source texts, and a review part in August 2013, where each translator reviewed one of their colleagues’ texts. The MT-assisted TM

translation part is described in Section 4.3.1.1.3.1, and the review part is described in Section 4.3.1.1.3.2.

Exploring translators’ interaction with MT-assisted TM in a way that allows for comparisons across match types, translators and texts and at the same time acknowledges that

translation is a context-dependent activity and should thus be investigated in a workplace setting is – admittedly – something of a balancing act. As pointed out by Ehrensberger-Dow and Massey, departing from the viewpoint that workplace studies focus on authentic translation assignments, “[o]ne of the broader challenges of workplace TPR is how comparisons can be made when so many factors differ (e.g. source texts, language combinations, settings, use of translation memory)” (2015, pp.11–12), concluding that workplace researchers cannot count on being able to make comparisons. However, the question is whether opting for comparability by means of an experimental design is incompatible with a workplace study and the view of translation as a context-dependent activity? Or, as stated by Risku et al., “[w]here, exactly, do we draw the line between the need to reduce the research object’s complexity for operationalization and empirical investigation and the loss of ecological validity and relevance?” (2013, p.167). In the present thesis, I have sought to strike a balance between ecological validity and allowing the findings to be comparable. I have done this by conducting an experiment, at the same time as insisting on the translators being permitted to work in a way that resembles their typical work situations as much as possible. Thus, I follow the suggestions by Christensen (2011, p.156) and Göpferich (2008, pp.14–16) regarding conducting an experimental field study with a high degree of ecological validity.

It should be noted that the experiment might be more accurately described as a quasi-experiment, a field experiment or an experimental field study (Oates 2006, pp.133–134;

Christensen 2011, p.156; Gile 2016, pp.225–226; Mellinger & Hanson 2017, pp.7–8), since it was conducted in a natural, workplace setting (and not in a laboratory) in which all variables cannot be controlled. Recognizing that the distinction between an experiment and a quasi-experiment is not clear-cut, according to Mellinger and Hanson (2017, pp.7–8) and Gile (2016, p.225), many studies in TS fall into this category. I shall continue to refer to the conducted study as an experiment, keeping in mind that it is not an experiment in the strictest quantitative terms.

4.3.1.1.3.1 The Machine Translation-assisted Translation Memory translation part The MT-assisted TM translation part of the experimental study took place during a week in May 2013. In the following, the translators participating in the experiment, the MT-assisted TM tool they used and the two source texts they were asked to translate are introduced.

Then, the steps in the execution of the experiment are described, and methods for data collection used in the individual steps are also included.

4.3.1.1.3.1.1 The translators

During the preparatory meetings and discussions between the management at TextMinded and me, it was agreed that, from a research, practicality and cost point of view, it was appropriate to engage eight translators in the experiment. The eight translators were selected in cooperation with a key account manager at TextMinded with special

responsibility for the distribution of internal resources. Three of TextMinded’s 11 in-house translators were not included in the experiment: since it was a prerequisite for inclusion in the experiment that the translators had experience of the CAT tool to be used and two of the 11 translators had not, these were not included. According to the key account manager, the last translator was left out by coincidence. This can be characterised as a purposive sampling technique as subjects who could provide particularly valuable information about the research questions were selected. Purposive sampling is common in mixed methods studies, and studies employing purposive sampling focus on an in-depth exploration of a limited number of cases rather than including a large sample (Kemper et al. 2003, p.279;

Teddlie & Tashakkori 2009, p.25). This is aligned with the thesis’ purpose of exploring TCI in depth and from several perspectives.

Prior to the experiment, I sent an e-mail to the eight translators, asking them whether they would like to participate. In this e-mail, they were also asked whether they had specific preferences concerning eight time slots suggested for the experiment. Admittedly, this combination of asking them whether they would participate and the planning aspect might have made it a bit difficult for them to refuse to participate. However, at no point did I sense any resistance towards participating, rather the contrary. I received positive responses from all translators.

The eight translators were all experienced translators. They all had Danish as their native language, and to varying degrees, all translators were used to translating from English into Danish. As is visible from Table 1, five of the translators were women and three were men (randomly named with a letter from A to H). Their experience with professional translation ranged from 6.5 to 23 years (the exact number of years is not given in Table 1 for reasons of anonymity). The translators did not receive compensation for their participation since they were employees at TextMinded; TextMinded bore the costs of the experiment in terms of the time spent. The translators will be further introduced in Section 5.1.3.

Translator Gender Professional translation experience (years)

A F 20-25

B M 11-20

C F 5-10

D F 20-25

E F 5-10

F M 11-20

G M 11-20

H F 11-20

Table 1. The translators

4.3.1.1.3.1.2 The Machine Translation-assisted Translation Memory tool

The MT-assisted TM tool used in the experiment was the CAT tool SDL Trados Studio 2011 which at the time was the CAT tool primarily used at TextMinded. The TM applied was TextMinded’s client-specific TM for Bang & Olufsen for the language pair English-Danish. The TM was used to pretranslate the source texts with matches with match values down to 70%.

Source segments with matches below 70% were pretranslated using an MT engine. The MT system was SDL’s baseline MT engine, SDL BeGlobal Enterprise. TextMinded had trained the baseline engine with all of their TM data on the language pair English-Danish and with their client-specific termbase for Bang & Olufsen. After translating the two source texts by means of the TM and the MT engine, we had two pretranslated source texts which were to be edited by the translators, producing two target texts. The process is illustrated in Figure 7.

Figure 7. Pretranslation process (inspired by Mesa-Lao 2015, p.4)

The two pretranslated source texts were included in a SDL Trados Studio project package together with the Bang & Olufsen TM and termbase. The project package also contained a reference text for one of the source texts (the FAQ text, cf. Section 4.3.1.1.3.1.3): a PDF file with the fully formatted source text (cf. Appendix 5). Eight identical project packages were created and stored in a folder on one of TextMinded’s drives which was accessible to all translators. During the translation process, the MT engine was active so that for each TM match, an MT match was also provided. The MT match was visible to the translators in the Translation Results window in the upper part of the SDL Trados Studio interface (cf. Figure 8), which meant that the translators had the option of replacing a pretranslated TM match with an MT match if they wished to do so. For each TM match, translators could see the

MT: SDL BeGlobal baseline + TextMinded

ENG-DA TM + Bang &

Olufsen termbase

Pretranslated source text Source text

(0% translated)

Untranslated segments

(0-70%) Source text

pretranslated with TM matches

(> 70%) TM: TextMinded

ENG-DA Bang &

Olufsen

Target text Human translator

match value, and textual differences between the new source segment and the source segment retrieved from the TM were highlighted. MT matches were clearly marked by the abbreviation “AT” for “Automated Translation”; however, no confidence scores indicating the quality of the provided match were provided. AutoSuggest was enabled so that translators received translation suggestions during typing. CM and 100% matches were marked as confirmed translations which meant that if the translators used the shortcut Ctrl+Enter after editing a segment, SDL Trados Studio would skip these segments and place the cursor in the next unconfirmed segment (cf. Section 2.3). It was ensured that the TM was not updated during the course of the experiment so that all translators were presented with the same matches and not matches produced by the translator(s) before them.

Figure 8. SDL Trados Studio 2011 interface 4.3.1.1.3.1.3 The source texts

The translators were asked to translate two source texts from the Danish company Bang &

Olufsen; the company had confirmed that their texts could be used as data in the study.

Bang & Olufsen sells high-end audio, video and multi-media products and is a regular client of TextMinded. The source texts were provided by TextMinded and were authentic

translation tasks in the sense that the source texts were assignments that TextMinded had previously undertaken for Bang & Olufsen during the two months prior to the experiment.

However, the target texts which were sold to Bang & Olufsen had been translated and reviewed by external translators, and thus not by the translators who participated in the experiment. Furthermore, the translations were not available online during the time of the experiment.

The source texts were 1) a Frequently Asked Questions (FAQ) text that related to Bang &

Olufsen’s surround-sound speaker system BeoLab 14, a technical text, and 2) a Newsletter about the music system BeoSound 5, a more creative/marketing-oriented text. The source texts are included in Appendices 1 and 2 together with the pretranslated matches. Both source texts were in English and were to be translated into Danish, the native language of the translators. As mentioned above, for the FAQ text, the translators also received a reference text with the fully formatted source text (cf. Appendix 5).

Translation Results window

Source text Target text

Termbase entries

M

Match type

The FAQ text comprised 625 words and the Newsletter 368 words. Thus, in total, the

translators were asked to translate 993 words each. In Table 2, the distribution of words and segments between match types is included for each text.¹⁴

FAQ text Newsletter

Table 2. Distribution of words and segments between match types in the two source texts

The decision to let the translators translate these two texts was made for a number of reasons. Firstly, for reasons of ecological validity, I wanted the translators to translate authentic, whole texts, i.e. not excerpts of texts, as well as texts of a certain length. As pointed out by O’Brien (2009, pp.261–262) and Muñoz Martín (2010b, pp.181–182; 2012, pp.17–18), working with short texts (in O’Brien’s terms between 200 and 300 words and in Muñoz Martín’s between 200 and 250) is problematic because translators usually work with longer texts and because, as argued by Muñoz Martín, if we let translators work with short, incomplete texts, “we run the risk of taking somewhat special behaviours – those related to starting to translate a text and also those associated to translating the beginning of a text – as the reference for normal behaviour” (2012, p.18). Further, based on TextMinded’s experience with the productivity of the translators, it was estimated that the translation of the texts was manageable within approximately one hour. This was suitable from a practical perspective, since the management at TextMinded was willing to invest approximately this amount of time in this part of the experiment, and from a research perspective, since I wanted it to be possible for the translators to complete the translations in one single sitting and avoid fatigue. A technical text, i.e. the FAQ text, was chosen as one of the texts since technical translation is said to constitute the majority of produced translations (Kingscott 2002). Further, technical translation is said to be the “genre of text (…) most likely to continue driving the use of MT for translation” (Specia 2012, p.2). A more marketing-oriented text, i.e. the Newsletter, was also included in the experiment since MT technology is generally assumed to perform more poorly on appellative texts (Schmitt 2015). Thus, it would be interesting to compare the two. Finally, many thoughts went into the choice of

14 It is a known problem that the word count in SDL Trados Studio and the word count in, for example, Microsoft Excel differ. As argued by Tatsumi (2010, p.66), it is thus preferable to choose one single way of counting the words in the different segments in order to ensure consistent measurement. In this study, also in line with Tatsumi, Excel is used to count the word number of each source segment using the following formula: =IF(LEN(TRIM(A1))=0;0;LEN(TRIM(A1))-LEN(SUBSTITUTE(A1;" ";""))+1), where ”A1” is the cell containing the segment whose number of words we want to count.

client and the choice of translation direction. Since the translators had different areas of expertise concerning text types and translation direction (where some translators translated more often from Danish into English and some more often the other way around), and since certain translators often translated texts from certain clients, it was impossible to find a client, a text type and a translation direction where all translators had the same

prerequisites. Thus, it was not possible to control for these aspects. However, in dialogue with TextMinded, it was found that the selected text types, the selected client and the selected translation direction was the best compromise we could make. I shall comment further on this in the discussion section (Section 6.3.1).

In both source texts, TM matches and MT matches were provided for approximately half of the source text words, i.e. TM matches down to a match value of 70% were available for approximately half of the source text words, and the MT engine was used to translate the remaining half of each of the texts. This also meant that the amount of data in each of the TM match types (CM, 100%, 95-99%, 85-94%, 75-84% and 70-74%) was considerably lower than the total number of words translated by means of MT. This has to be taken into

account in the interpretation of the findings since the findings pertaining to the different TM match categories necessarily build on less data than the findings pertaining to MT matches.

At least two differences between the FAQ text and the Newsletter are worth noting and should be taken into account in the interpretation of the findings. First, the FAQ text contained tags indicating formatting and the presence of visual elements, whereas the Newsletter did not. Since the translators are typically expected to ensure that a target text contains the same tags as the source text and since tags are not included in the word count above, we would expect the translators to spend relatively more time on the FAQ text.

Second, in the case of the FAQ text, the target text that was produced before the

experiment had not been included in the TM that was used to train the MT engine; however, this was the case for the Newsletter. This might have resulted in matches of a higher quality in the Newsletter since the MT engine had “seen” the translation before. Both of these issues will be addressed, when relevant.

Finally, since authentic source texts were used, no measures were taken to ensure that the segments were a specific length. The texts thus contained segments consisting of between 1 and 29 words in the FAQ text, and between 5 and 32 words in the Newsletter. This might be problematic, since MT has e.g. been found to perform better on longer segments than on shorter ones (Plitt & Masselot 2010; Federico et al. 2012). However, in balancing

comparability and ecological validity, I assigned the latter higher importance (cf. Teixeira 2014b, p.179).

4.3.1.1.3.1.4 Conducting the Machine Translation-assisted Translation Memory translation part & methods for data collection

The MT-assisted TM translation part of study consisted of four steps which are described in the following. Thus, in Section 4.3.1.1.3.1.4.1, the preparations for the experiment are first described (step 1). Next, in Section 4.3.1.1.3.1.4.2, the part of the experiment where the translators translated the two texts is described, including the methods for data collection used in this step (step 2). In Section 4.3.1.1.3.1.4.3, the preparations for and execution of

the retrospective interviews are described (step 3), and in Section 4.3.1.1.3.1.4.4, the post-experimental questionnaire which constitutes the last step in the MT-assisted TM

translation part is described (step 4).

The eight translators were asked to participate in the experiment at different times during a week in May 2013. One translator participated in each of the time slots in Table 3.

Monday 27 May 2013, 9 am – 12 pm Monday 27 May 2013, 13 pm – 16 pm Tuesday 28 May 2013, 13 pm – 16 pm Wednesday 29 May 2013, 9 am – 12 pm Wednesday 29 May 2013, 13 pm – 16 pm Thursday 30 May 2013, 9 am – 12 pm Friday 31 May 2013, 9 am – 12 pm Friday 31 May 2013, 13 pm – 16 pm

Table 3. Time slots for the MT-assisted TM translation part of the experimental study

The translators translated at different times during the week because this made it possible to make participation in the experiment fit into each translator’s individual schedule, reducing the disturbance to their usual work. Also, it made it possible for me to observe the translators during the translation process and to carry out the retrospective interview with each of the translators within a few hours of the translation process. The drawback of letting the translators translate the texts at different times during the week was that they might, for example, discuss the texts and their solutions. However, I asked them not to do so (cf.

Section 4.3.1.1.3.1.4.1.2).

4.3.1.1.3.1.4.1 Step 1: Preparations

Approximately two weeks prior to the experiment, I tested the experimental setup with one of my colleagues. In the week before the experiment, I installed the keystroke logging and screen recording tools used during the experiment on the translators’ computers (cf.

Sections 4.3.1.1.3.1.4.2.1 and 4.3.1.1.3.1.4.2.2). Just before each translator’s participation in the experiment, he or she received simple instructions. I connected the MT engine in SDL Trados Studio so that the MT engine would be active during the experiment and thus provide the translators with MT matches in TM matches (as described in Section

4.3.1.1.3.1.2). Also, I checked whether the time displayed on the translator’s computer was consistent with the time on my computer. This was relevant for the observational protocol I produced during the experiment (cf. Section 4.3.1.1.3.1.4.2.3). The test of the experimental setup and the instructions provided to the translators are described in more detail in the following two sections.

4.3.1.1.3.1.4.1.1 Testing the experimental setup

No actual pilot study was conducted prior to the experiment. I considered the possibility of carrying out a pilot study at TextMinded with at least two translators, but since I did not want to create too much attention about the experiment and since it would disqualify the participating translators from being part of the experiment, I decided to conduct a test of the experimental setup instead. This test was carried out with a colleague of mine, a translator and translation scholar. In the test, she translated the FAQ text in SDL Trados Studio on a laptop I had set up for her at her usual desk. She received the instructions

No actual pilot study was conducted prior to the experiment. I considered the possibility of carrying out a pilot study at TextMinded with at least two translators, but since I did not want to create too much attention about the experiment and since it would disqualify the participating translators from being part of the experiment, I decided to conduct a test of the experimental setup instead. This test was carried out with a colleague of mine, a translator and translation scholar. In the test, she translated the FAQ text in SDL Trados Studio on a laptop I had set up for her at her usual desk. She received the instructions

In document Aalborg Universitet (Post-)Editing - A Workplace Study of Translator-Computer Interaction at TextMinded Danmark A/S Bundgaard, Kristine (Sider 84-99)