Inter-observer reliability

Reliability in research is concerned with the question of measurement. Kazdin (1982) states that “when direct observations of behavior are obtained by human observers, the possibility exists that observers will not record behavior consistently” (p. 48). A way to strengthen the reliability of the results is to obtain inter-observer reliability, as recommended by Kazdin (1982). Establishing inter-observer reliability helps to ensure that the process has been fair, ethical, and rigorous (Richards et al., 1998).

According to Kazdin (1982), inter-observer reliability is desirable since it improves the chances of obtaining consistency, of minimizing biases which the individual assessor may have, and of ascertaining whether the target behaviors were clearly defined. The use of permanent products such as DVD’s enables reviewing them several times and thus, helps to achieve these goals.

“Unless a measure is reliable, it cannot be valid” (Robson, 2002, p. 101).

Therefore, the data collection and analyses reporting observer reliability will be

presented first, as prerequisites to the other analyses.

There were four independent observers in this study, in addition to the researcher:

• One observer to establish reliability of target behaviors;

• One observer to establish reliability of the variable session part;

• Two observers to establish reliability of the Parent-Child ERA.

One of the two observers recruited to establish reliability of the Parent-Child ERA was also the observer for the Parent-Child ERA analysis.

3.9.1.1 Inter-observer reliability of target behaviors (dependent variables) As mentioned in section 3.9.2, all sessions were videotaped. The video material was downloaded to the computer and burned to DVD’s for analysis. Prior to the actual video analysis, the researcher first trained herself and then intensively trained an independent observer³⁸ to identify the target behaviors and measure their frequency and/or duration data (see Appendix P, for coding guidelines) by using DVD’s of a child who the researcher used as a trial subject, and whose data was not ultimately included in this study. The coding was done on a form designed specifically for the video observation (Appendix E).

For the actual video analysis, in order to determine inter-observer reliability, 25%

of each child’s video material (a total of 19 sessions of 20 minutes each, for all subjects) was randomly selected, observed and scored separately by the researcher and the independent observer, and Intraclass Correlation Coefficients (ICC) were then computed on the two sets of scores. The observer was blind to any information regarding research questions as well as to the session number. The results are presented in section 4.1.1.

3.9.1.2 Inter-observer reliability related to the independent variable session part (directed vs. undirected)

The researcher, who was also the therapist, strictly followed the protocol guidelines in the directed and undirected parts of the session. In order to determine whether she had implemented the protocol guidelines in all four conditions of this study (DM and UM

38 The independent observer was an experienced kindergarten teacher, MA in Education.

as well as DP and UP), an independent observer was first provided with the definitions of the directed and undirected parts of a session, then watched the first two minutes from each session part (directed and undirected) of all the sessions (presented in random order), and subsequently documented as accurately as possible whether that part was directed or undirected. The observer was blind to any information regarding research questions as well as to the session number. Cohen’s Kappa was then used to compute inter-observer reliability. The results are presented in section 4.1.2.

Computing inter-observer reliability was carried out in order to strengthen procedural reliability, which is defined as “a measure of the extent to which the actual application of the independent variable over the course of an experiment matches its description in the method section of the research report” (Cooper et al., 1987, p. 239).

It also refers to the issue “treatment integrity”, which is “the extent to which the independent variable is implemented or carried out as planned” (Cooper et al., 1987, p. 237), or what is also known as "treatment fidelity"; both terms refer to the uniformity, accuracy, consistency, and replicability of a treatment delivered in a particular research setting, as pointed out by Hennessey and Rumrill (2003). The authors indicate that assurance of treatment fidelity is the first step toward accurately ascribing changes in the dependent variable to systematic changes in the treatment or independent variable.

3.9.1.3 Analysis related to therapist's behavior 3.9.1.3.1 Parent-Child ERA inter-observer reliability

To increase internal validity, a procedure was administered to find out whether the therapist in her dual role consistently demonstrated similar affective and behavioral characteristics in UM and DM as well as in UP and DP. Before answering this question, a reliability analysis was carried out. Two independent observers³⁹ rated the therapist’s interactions with each child by watching the middle five minutes from each half (directed and undirected) of one session randomly chosen for each child (see Figure 3.1), and rating it on 14 variables taken from the Parent-Child ERA (see section 3.4.3).

39 The two observers, who were chosen for this procedure, were psychology students who had extensive training using this assessment prior to serving as observers for this study.

Session x

0---2.5---7.5---10---12.5---17.5---20 time (minutes) The observed time intervals are marked in grey.

Figure 3.1. Time sampling by observers using Parent-Child ERA

The middle five minutes were selected as the optimum section to watch since the initial section can be influenced by the events that happened to the child before the child entered the room, and the last section can be influenced by any decay in attention. The two observers were blind to any information regarding research questions as well as to the session number, but they knew whether they were watching a directed part of a session or an undirected part, since this information had to be taken into consideration while rating the therapist’s interactions. They were given the definitions of the directed and undirected parts of the sessions as detailed in section 3.7.

As the sample in this research was small, percentage agreement reliability was computed for their ratings. (Appendix U) The results are presented in section 4.1.3.1.

3.9.1.3.2 Parent-Child ERA analysis

To further address the above mentioned issue of potential bias, one of the independent observers watched the middle five minutes from each half (directed and undirected) of three additional sessions randomly chosen for each of the four children (A, As, O, and C). The observer followed the same procedure for two additional sessions for Z (since she completed only 11 sessions out of 16). Then, she rated the therapist’s interactions with each child on 14 variables taken from the Parent-Child ERA. In order to evaluate the presence or absence of a difference between the ratings in the play condition and the music therapy condition, means and SD’s were computed.

As in the previous section, the observer was blind to the research questions as well as to the session number, but she knew whether she was watching a directed part of a session or an undirected part since this information had to be taken into consideration while rating the therapist’s interactions (Appendix V). The results are presented in section 4.1.3.2.