Results: Reliability

Results related to observer reliability will be presented as follows:

4.1.1 Inter-observer reliability of target behaviors (dependent variables) 4.1.2 Inter-observer reliability related to the independent variable session part (directed vs. undirected)

4.1.3 Analysis related to therapist's behavior

4.1.3.1 Parent-Child ERA inter-observer reliability 4.1.3.2 Parent-Child ERA analysis

4.1.1 Inter-observer reliability of target behaviors (dependent variables)

As explained in section 3.9.1, a valid measure must be reliable. Therefore, this must be tested through observational studies, before any other analyses can take place.

Inter-observer reliability actually gives a score between 0 to 1 of how much consensus there is in the ratings or scores given by the observers.

In order to evaluate inter-observer reliability, 25% of each child’s video material (a total of 19 sessions of 20 minutes each, for all subjects) was randomly selected, observed and scored by an independent observer (see section 3.9.1.1). The researcher and the independent observer, separately, scored each target behavior as a frequency or duration sum score, and Intraclass Correlation Coefficients (ICC) were then computed. Table 4.1 presents the inter-observer reliability of all dependent variables.

Table 4.1. Inter-observer reliability of dependent variables in session analysis

Variable Agreement ICC

Spontaneous imitation 0.99

Spontaneous initiation 0.94

Spontaneous turn-taking 0.96

Spontaneous synchronization 0.93

Duration of spontaneous synchronization 0.76 Duration of spontaneous turn-taking 0.92 Number of events of spontaneous turn-taking 0.91

The correlations reported in this table show that with one exception (0.76), all the ICC’s were above 0.90. The very high inter-observer reliability on six variables out of seven indicates a high level of agreement between the researcher and the independent observer for these variables. For duration of spontaneous synchronization, the ICC of 0.76 still represents a relatively high level of agreement between the two observers.

In order to assure that a behavior could be described and identified as spontaneous synchronization, it was deemed necessary during the process of analyzing the results to establish a minimum time period of two seconds (or more) of this behavior taking place, so that the observers can be sure that it could be described and identified as spontaneous synchronization. The majority of events actually did last for more than two seconds. However, seven events of spontaneous synchronizations of two seconds were counted because it was noted that even in these short events, the child clearly attempted to simultaneously match the therapist’s action, e.g., the child tried to match either the rhythm of the music in the music therapy sessions or stretched out his/her hands to hold a toy/game together with the therapist.

4.1.2 Inter-observer reliability related to independent variable session part (directed vs. undirected)

In order to find out whether the therapist had implemented the protocol guidelines in relation to session part (directed or undirected), in all four conditions of this study (DM and UM as well as DP and UP), an independent observer watched the first two minutes from each session part, for all sessions randomly presented, and documented as accurately as possible whether that part was directed or undirected (see section 3.9.1.2). Cohen’s Kappa was used to compute inter-observer reliability for these nominal variables and was found to be 0.89. Landis and Koch (1977) consider Kappa

of 0.40 to 0.59 as moderate inter-rater reliability, 0.60 to 0.79 as substantial and 0.80 or above as outstanding.

4.1.3 Analysis related to therapist's behavior 4.1.3.1 Parent-Child ERA inter-observer reliability

As explained in section 3.1, the music therapist in this study was also the researcher.

In order to increase internal validity, a procedure was administered to find out whether the therapist consistently demonstrated similar affective and behavioral characteristics in UM and DM as well as in UP and DP. Before answering this question, a reliability analysis was undertaken. Two independent observers rated the therapist’s interactions with each child, by watching the middle five minutes from each half (directed and undirected) of one session randomly chosen for each child, and rating it on 14 variables taken from the Parent-Child Early Relational Assessment (Parent-Child ERA) (Clark, 1999; Clark et al, 1985) (The procedure is detailed in section 3.9.1.3.1). As the sample in this research was small, percentage agreement reliability was computed for their ratings, and found to be 72.14 % or 94.25% (see Appendix U), obtained by dividing the number of agreements by the overall number of observations and multiplying this ration by 100. These two different percentage agreements were calculated and presented to demonstrate two different computing methods found in the literature in relation to the Parent-Child ERA⁴¹.

4.1.3.2 Parent-Child ERA analysis

To further address the above mentioned issue of bias due to the therapist’s intentions in the dual role of therapist and researcher, one of the independent observers watched three additional sessions for each child, and rated the therapist’s interactions with the child as described in section 3.9.1.3.2 (see also Appendix V). In order to evaluate the presence or absence of a difference between the ratings in the play condition and the music therapy condition, means and SD’s were computed. The results are presented

41 According to one method, which resulted in the lower percentage agreement of 72.14%, an agreement exists if the observers use the same rating point on the 5 point scale for rating a specific item. The rationale for the other method, which resulted in the higher percentage agreement of 94.25%, is that it may be difficult at times to get an exact agreement as it is not a frequency count, time sampling approach. Therefore, 1’s & 2’s rating points, which are considered as ‘areas of concern’ on the PCERA, are accepted as an agreement. Same for points 4’s & 5’s, which are considered as ‘areas of strengths’. Ratings of 2’s and 3’s or 3’s and 4’s are not considered as an agreement (Clark, 1999; Harel, 1995).

in Table 4.2.

Table 4.2. Means and standard deviations for Parent-Child ERA analysis

Mean Standard deviation

Play condition 4.75 0.55

Music therapy condition 4.68 0.69

The results comparing the play condition and the music therapy condition are very close indicating almost no difference in the interactional behavior of the therapist in play and in music therapy.

In summary, the results for observer reliability showed very high inter-observer reliability for six variables and high for a seventh variable, attesting to the high reliability of the data.