E VALUATION

41

3 C

The overall theoretical inspiration is found in organisational institutionalism as described in the previous chapter. But the thesis also draws on conceptual frameworks from evaluation use theory. In the following subsections, I will elaborate on the conceptual background of evaluation and evaluation use theory.

42

some other kind of information processing with a chronology or causality and with the objective of impacting decision-making. It is important here to note that evaluation according to Vedung is intended to play this role in decision-making.

Hereby, Vedung implies that evaluations do not always do so, which is a relevant distinction for studies on evaluation use and learning from evaluation.

The role of evaluation in relation to decision-making is particularly relevant to this thesis. To illustrate this importance, other scholarly definitions also refer to the subsequent use of evaluation. For example, Cousins et al. (2004) also define evaluation as a tool to improve future decision-making. They define evaluation as a

‘systematic inquiry leading to judgements about program (or organisation) merit, worth, and significance, and support for program (or organisational) decision making’.

Cousins et al.’s definition takes for granted that evaluations support decision-making, which we will see later is often not the case. Both definitions have much in common with Scriven’s early, somewhat minimalist, definition: ‘Evaluation is the process of determining the merit, worth, and value of things’ (Scriven, 1991:

1). However, Scriven does not define the purpose of evaluation (intended support for decision-making) and his definition is indeed very broad.

In this thesis, Vedung’s definition is used due to its reservations regarding the usefulness of evaluation to decision-making. Another important element of this definition is its uncompromising stance on the retrospective nature of evaluation.

Though it seems counter-intuitive that evaluation can be anything but retrospective, common planning tools such as impact assessment and ex ante appraisals are often labelled ‘evaluation’. The distinction was useful for this thesis in order to frame the cases to be mainly retrospective programme evaluation in the European Commission and not ex ante evaluation nor impact assessment, which are also important but very different tools in the Commission.¹ Therefore, the

1 The Commission defines evaluation as “a judgment of interventions according to their results, impacts and needs they aim to satisfy.” http://ec.europa.eu/dgs/internal_market/evaluation/evaluation/index_en.htm (1st November, 2014).

43

definition allows for more generalisability of the findings than would be the case if completely different tools such as impact assessment and ex ante appraisals were also contained in the definition.

3.1.2 PURPOSE OF EVALUATION

Evaluation has two main functions in democratic governance; namely, enhancing accountability and facilitating learning (van der Meer and Edelenbos, 2006;

Scriven, 1991). Whereas the former emphasises retrospective assessment in order to secure accountability and oversight of the programme in question, the latter emphasises improved policy-making. The former type is commonly referred to in the evaluation literature as ‘summative evaluation’, while the latter is called

‘formative evaluation’. This thesis starts from this distinction and dichotomy, as it focuses on the implications of evaluation systems for the formative role of evaluations.

Scriven’s dichotomy is often taken as a starting point for discussion about evaluation use and learning. However, in this case it should be noted that the literature on use is far more developed and extensive than the evaluation literature on learning. For the same reason, this thesis uses the term ‘evaluation use’ to cover both evaluation use and learning.

Scriven introduced the distinction between summative and formative evaluation (Scriven, 1967), but most evaluation literature takes it as a base for the elaboration of standpoints. Alkin and Christie (2004) have made several attempts to map the evaluation literature. The result is that one important part of the literature

investigates the formative aspects of evaluation under the heading ‘evaluation use’

(Shulha and Cousins, 1997: 199) and essentially focuses on participatory evaluation and the delicate relationship and participation of stakeholder ‒ and particularly intended users ‒ in the evaluation process. This type of evaluation can also focus on personal and organisational learning, as well as ownership,

engagement, self-determination and emancipation. Hence, organisation theory on learning and processes was also integrated in the theory building (Torres and Preskill, 2001; Preskill and Torres, 1999) as well as in cognitive theories on learning and knowledge sharing (Dolowitz and Marsh, 1996; Owen and Lambert, 1995). In collaborative modes of evaluation, the evaluands or stakeholders are

44

involved in the evaluation practices, and the evaluator assumes the role of teacher while the idea of the objective and independent evaluator is down-played.

Collaboration has been promoted as a vehicle for capacity building and organisational learning (Cousins and Earl, 1995b; Cousins and Earl, 1995a) as well as stakeholder empowerment (Fetterman, 1994).

The proponents of ‘evaluation use’ include Daniel Stufflebeam, Marvin Alkin, Michael Q. Patton, Hallie Preskill, Jean King, David Fetterman and Brad Cousins.

Michael Q Patton is probably the most notorious and the reason ‘evaluation use’

or ‘utilization’ plays such an important role in evaluation literature. His key contribution, ‘Utilization-focused evaluation’ (Patton, 1997), is a well-written textbook that is mainstream in evaluation curricula, despite Patton’s extreme views that make him the most prominent representative for the formative branch of the evaluation literature. According to Patton, the evaluator should cast away any methodological considerations of reliability and validity if they stand in the way of evaluation use. In other words, the use by intended users of the evaluation takes priority over any other consideration. This view is also sometimes referred to as ‘pragmatic’ and is criticised for being too close to management consulting and too far from objective independent evaluation.

In contrast to the views of Michael Q Patton, a number of authors emphasise the summative role of evaluation by focusing on methodology, objectivity and reliability. This branch in the evaluation literature is represented by Thomas Cook, Gary Henry and Melvin Mark, Huey Chen, Carol Weiss, Ray Pawson and Rick Tilley, who are sometimes referred to as the ‘positivists’. Here we also find the adherents to the so-called ‘realist’ type of evaluation. According to Alkin and Christie (2004), proponents of the positivist evaluation are also supporters of a value-free type of evaluation. They focus particularly on high reliability, and emphasise that evaluation should be as close as possible to objectivist social science and that truth about a social phenomenon can be value-free and studied out of its political context.

Very dear to the ‘positivist’ views is the discussion of evaluation’s place in relation to science and research. Research and evaluation are both related to the systematic collection of data and production of knowledge. According to Owen

45

(2002), traditional academic social research, as well as policy analysis and program evaluation, is based on systematic social inquiry. They all use the same scientific methods and can analyse the same study objects. However, they have different objectives. Whereas science aims for universality and generalisability, evaluation aims to study a particular phenomenon and does not apply methods and research designs in order to generalise the findings to the population or to seek a universal truth. In other words, the purposes and uses of the knowledge produced are different (Alkin and Taut, 2003). In the case of research, the goal is

generalisable knowledge that holds true across all settings, times, and for all individuals represented by the produced knowledge. Science aims to produce general theories through rigorous testing of hypotheses. The collection of systematic knowledge is based on information gathered by systematic social scientific procedures. The systematic creation of knowledge is based on linking empirical observations and conclusions in a coherent, shareable, and persuasive way with the aim of ensuring objectivity, reliability and validity of findings.

It is important to focus on the distinctions between knowledge produced through research and knowledge produced through evaluation. In most cases, the focus of an evaluation is more specific. Evaluation can thus be compared to a case study:

its purpose is context-specific and applicable only within a particular setting at a particular point in time, and it is intended for use by a particular group of people (Cronbach and Suppes, 1969). According to Scriven (1967) and the definition used earlier by Vedung, evaluation is different from research in terms of its unique purpose of judging the worth, merit, or quality of the evaluated object, as well as in terms of the context-specific uses of the knowledge produced to make these judgments.