The quality and quantity of the studies available for the synthesis

The robustness of the synthesis further depends on the quality as well as on the quantity of the studies available for the synthesis (i.e. the evidence base). If studies of poor quality are uncritically included in the synthesis, the trustworthiness of the synthesis will necessarily be affected nega-

tively/reduced. Also, the fewer the studies available for the synthesis (i.e. the smaller the evidence base), the more insecure the conclusions of the synthesis will be, other things being equal.

As a point of departure, the quality of the 44 studies available for the synthesis is, generally speak- ing, of a fairly high quality, since the studies that were assigned an overall ‘low’ weight of evidence in the research mapping have been excluded from the synthesis. Hence, only the studies that were assigned an overall ‘high’ or ‘medium’ weight of evidence in the research mapping have been made available for the synthesis. The overall weight of evidence assigned to each study was based on an overall assessment/appraisal of each study on the following three aspects, cf. Section 4.5: 1) the credibility of the research findings for answering the study question(s) when taking account of all quality assessment issues, 2) the appropriateness of the chosen study design including the data and methods of analysis for addressing the specific review questions and 3) the relevance of the particular focus of the study (including conceptual focus, context, sample and measures) for ad- dressing the specific review questions. It should be mentioned here that the three individual as- pects contained within the overall weight of evidence assignment of each study may have been assessed to be either higher or lower than the overall weight of evidence assigned to each study.

That being said, an overall assessment of the research designs applied in the studies available for the synthesis gives rise to a more critical appraisal. Rieper & Foss Hansen (2007: 79) have devel- oped an evidence typology concerning the relation between the type of review question posed and the research design to be used for addressing that type of question. From there it follows that in so far as the review question is about cause-and-effect or effects of interventions (i.e. ‘Effec- tiveness. Does this work? Does doing this work better than doing that?’), as is primarily the case of this systematic review, studies applying a randomized controlled trial (RCT design) should be given the greatest weight of evidence, other things being equal, followed by cohort studies and quasi- experimental studies.

Overall study design Number of studies

Cross-sectional study 28

Secondary data analysis 8

Experiment with non-random allocation to groups 4

Cohort study 3

Random experiment with random allocation to groups 1

Systematic review 1

Views study 1

Table 5.4.3.1 Overall study design

N=44. There are 46 answers, as two studies have been coded as having applied more than one study design

Table 5.4.3.1⁵³ shows that, together, the 44 studies available for the synthesis far from live up to such standards. RCT studies and quasi-experimental studies are almost non-existent (N=1 and N=4, respectively). According to Figure 5.4.3.1, even cohort studies are rare (N=3) and other types of longitudinal research designs have not been applied. Both cohort studies and other types of longitudinal research designs are preferable to, for instance, cross-sectional designs because they analyse a development in the data/the population under investigation by involving repeated ob- servations of the same variables over longer periods of time, which are precluded when applying a cross-sectional design. In line with this, as was observed in Section 5.3.2.2, only a few of the stud- ies available for the synthesis actually present their analysis results in such a way that evidence on the sub-question concerning ‘The timing of dropout’ (i.e. the possible time-varying effect on drop- out of the different factors investigated) can be obtained. This also affects the sub-question ‘What happens to dropouts after dropout?’. As mentioned in Section 5.3.2.3, the lack of post-university longitudinal surveys that compare occupational whereabouts and related occupational aspects between graduates and dropouts for a longer period of time after exmatriculation has the implica- tion that evidence on the possible time-varying effect of these aspects and the longer term impli- cations of dropout is lacking. Hence, notwithstanding the evidence typology, the study design most frequently used among the studies available for the synthesis is, as seen in Table 5.4.3.1, a cross-sectional design on the basis of either university administrative data or a questionnaire sur- vey.

53 A similar figure can be found in Appendix 3.

Other aspects of quality on the level of the individual studies available for the synthesis concern the data and the specific methods of analysis applied across the studies.⁵⁴ Ideally, as pointed to in Section 5.2.1, data should not contain a skewness in focus in favour of non-malleable and pre- university malleable factors or hard facts at the expense of many university malleable factors and softer/attitudinally based aspects, including issues such as study approach, student effort, motiva- tion and satisfaction with studying. This skewness is partly considered to be a consequence of the extensive use of university administrative data containing only hard facts about the students. Fur- thermore, it would have been preferable if more studies had used as their principal method of analysis multivariate (multinomial) logistic regression analysis including the use of instrumental variables as an alternative to estimating relations of cause-and-effect in the absence of controlled experiments/experimental identification. The use of instrumental variables has been found to be very limited among the included studies. Alternatively, as pleaded for by Tinto (1975: 94, footnote 3), evidence on why dropout occurs would have benefitted from more studies applying a path analytic approach in relation to viewing university dropout as a process. Again, the use of path analysis has been found to be very limited.

O’Neill et al. (2011) reach a somewhat similar conclusion concerning study designs and methods of analysis in their literature review on factors associated with dropout in medical education: “No relevant experimental studies of dropout were found. Experimental designs might yield valuable information that complements the findings of cohort studies, for example on educational interven- tions and their effect on dropout. Although cohort studies of dropout provide useful knowledge, they must account for confounding in order to be of real value. Dropout is most likely multi- factorial in nature *…+; therefore it is probably wise to continue to use designs which allow for sub- sequent multivariate analyses.” (ITT2770688: 449)

Notwithstanding the above characterisation of the studies available for the synthesis, they are still of a fairly high quality and, in any event, the best available for the synthesis. The next section (cf.

Section 5.4.4) will evaluate upon the issue of generalisability of the synthesis findings.

A second important aspect on the level of the individual studies for assessing the robustness of the synthesis has to do with the quantity of the studies available for the synthesis (the evidence base).

The overall point to make concerning this aspect is rather disappointing. As has been mentioned several times during the narrative synthesis in Section 5.3, the evidence base for investigating most of the aspects underlying the three review questions is small.

The evidence for each of the three aspects underlying the first review question ‘What is dropout from university studies?’ rests upon, at most, a handful of studies. The evidence on the sub- question ‘Different processes leading to different types of dropout behaviour?’ rests upon the four core studies. Apart from the comprehensive evidence that first-year dropout is both more exten-

54 For a characterisation of the 44 studies available for the synthesis on these issues, consult Appendix 3.

sive and often different from later dropout, the evidence base for the sub-question concerning

‘The timing of dropout’, including the evidence base on the possible time-varying effects on drop- out of the different investigated factors, was found to be rather small. Evidence on the sub- question ‘What happens to dropouts after dropout?’ also primarily rests upon a few studies.

Since the evidence on the second review question ‘Why do such dropout phenomena occur at uni- versities?’ comprises 42 of the 44 studies available for the synthesis, the potential for robust syn- theses of the findings seems very promising at first sight. However, because many different factors have been investigated for a possible effect on university dropout, the evidence base of quite a few of the specific aspects investigated is, nevertheless, rather small. The aspects containing the greatest evidence base (in most cases coinciding with the aspects found to contain the most une- quivocal evidence of an effect on university dropout) mostly pertain to hard facts about the stu- dents, such as their age, gender and socio-demographic background, as well as to pre-university malleable factors such as prior academic achievement. The evidence base is, to the contrary, pre- dominantly small concerning the possible effect on dropout of the university malleable factors, which have been the focus of this systematic review.

Lastly, as pointed to in Section 5.3.1.1, the evidence on the third review question ‘What can be done by the universities to prevent or reduce dropout?’ merely rests upon the findings of three studies.

The evidence base for addressing most of the aspects underlying the three review questions is, therefore, not very robust. What is more, some of the aspects investigated in relation to the three review questions are close to being investigated country-specifically (cf. Section 5.4.4 below). Con- cerning the first review question ‘What is dropout from university studies?’, the evidence on the aspect ‘Different processes leading to different types of dropout behaviour?’ is solely derived with- in a British, Norwegian and Spanish context, whereas the evidence on post-university student tracking in relation to ‘What happens to dropouts after dropout?’ is derived mostly within a Danish and especially a German context. The evidence on the second review question ‘Why do such drop- out phenomena occur at universities?’ also suffers from such country-specific investigation. While hard facts such as parents’ occupational status (social class) and institutional resources as poten- tial determinants of university dropout are primarily investigated within a British context, other more attitudinally based aspects (e.g. issues of a student’s study approach, effort, motivation and study satisfaction) are non-existent within the British research and have been obtained elsewhere.

This mostly has to do with the different types of data available across the European countries which, among other things, is an effect of different data policies working at the national (or re- gional) levels across the countries (cf. Section 5.2.1). Concerning the third review question ‘What can be done by the universities to prevent or reduce such dropout phenomena?’, the evidence is also country-specific as the three intervention studies are quite unique and narrow in context.

5.4.4 The generalisability of the studies available for the synthesis within the geographically set