A RTICLES

72

sui generis organisation, from the outset it is difficult to talk about a genuine

‘population’ of complex international organisations such as the EU Commission.

73 Table 5-1 Overview of article methodologies

No Title Research

design Methodology Methods Data sources 1 Evaluation use in the

organisational context – changing focus to improve theory

NA Literature

review

Bibliographic search

Scientific articles

2 Evaluation in the European Commission – for learning or accountability?

Longitudinal case study (The European Commission’s evaluation system)

Qualitative content analysis Historic analysis

Unstructured expert interviewing Document review

Transscribed interviews Documents Homepages Literature 3 Evaluation use in

evaluation systems – the case of the European

Commission and its LIFE programme

Single case study (LIFE programme)

Qualitative content analysis

Semi-structured interviewing

Transscribed interviews Documents

4 Evaluation and Policy Learning –the learner’s perspective

Multiple case studies(LIFE programme, Intelligent-Energy Europe II programme and FP7)

Qualitative content analysis

Structured interviewing

Transscribed interviews

5.2.1 FIRST ARTICLE

The first article is purely theoretical, and uses organisational institutionalism to create a model for different types of evaluation use. It is based on a literature review that focuses on the application of organisational institutionalism in the evaluation literature. Therefore, this article is the theoretical starting point for this thesis, as it discovers that organisational institutionalism has not been used to

74

investigate evaluation use in the evaluation literature. The article thus discovers a gap that is filled theoretically by itself and the other articles in this thesis.

5.2.2 SECOND ARTICLE

The second article is based on a case study of the Commission’s implementation of evaluation practices over time. Thereby, the article investigates how an evaluation system is implemented and how the issues of accountability and learning are reflected in the practices implemented.

The article is based on data generated from 58 recorded interviews, two group interviews and one conference on evaluation in the EU, along with numerous informal talks with experts and Commission desk officers and personal

observations including evaluation steering committee meetings. Interviewees were sampled purposefully and according to availability, and included Commission employees working in evaluation units and policy units as well as external evaluators, evaluation trainers and consultants working with the Commission in the setting up of the evaluation system. Several of the interviewees were senior staff who had performed key roles in the early implementation of the evaluation system and thus had a good historical overview of evaluation in the Commission.

Interview data were validated with document data comprising more than a hundred public and non-public documents, such as internal evaluation policy papers, guidelines, and minutes of meetings in the evaluation network (on document analysis see Stewart and Kamins, 1993).

Data were analysed according to the principles of qualitative content analysis (Schreier, 2012; Mayring, 2002; Mayring, 2000; Mayring, 2004; Kohlbacher, 2006) and by using the NVIVO software package (Bazeley, 2013). Qualitative content analysis focuses on both latent and manifest content through stressing the visualisation of patterns in the data. The data were coded according to the interviewees’ perceptions of accountability and learning, as well as their rationalisation of the Commission’s evaluation practices. Findings were triangulated and validated with document data, and interviewees had the

opportunity to comment on the findings. Finally, the reliability of the findings was strengthened by the author’s prolonged engagement in the field over a period of

75

two years as well as five years of professional experience with evaluation of EU policies.

5.2.3 THIRD ARTICLE

The third article analyses the use of four evaluations of the Commission’s Programme for the Environment and Climate Action (LIFE) over a ten-year period. The LIFE programme was chosen because it is a ‘classic’ centrally managed EU expenditure programme (parts of the programme was externalised to EACI in 2014). The programme has experienced a full Commission evaluation cycle (ex ante, midterm, final and ex post), and therefore represents a complete picture of evaluation use over an entire policy cycle as well as an entire evaluation cycle. Moreover, by looking at four evaluations over time, we get a more

diversified idea of the effects of the evaluation system on the use of evaluations.

The analysis is based on 16 semi-structured in-depth interviews and eight follow-up interviews. The informants were sampled purposefully according to relevance and availability and consisted primarily of staff from the Directorate General for the Environment (DG ENV), consultants who performed the evaluations,

representatives of Members of the EP’s Committee for the Environment (member assistants) and Council members. The data collection continued until ‘saturation’

was achieved. Document data were also coded and used primarily to validate interview data through triangulation (Bryman, 2012). The document analysis comprised four retrospective evaluations of the LIFE programme (midterm, 2003;

ex post, 2009; midterm, 2010; final, 2013) and several other documents, including DG ENV presentations to the Committee of Regions and the EP, internal

Commission documents, the combined ex ante and impact assessment along with fiches and Commission position papers for the new LIFE programme 2014‒2020.

The methodology applied in the paper is based on the principles of qualitative content analysis, and the coding and analysis of data was carried out using the NVIVO software package. The first 16 semi-structured interviews were analysed with a view to existing conceptual frameworks developed in the evaluation literature and described earlier, and the eight follow-up interviews were conducted to check for saturation. Coder reliability was sought by using the existing

conceptual frameworks and subsequently running three rounds of coding on the

76

interview data (Kohlbacher, 2006). The credibility of the findings was further strengthened by a prolonged engagement in the field, with the researcher conducting interviews in four consecutive waves over a period of one year. In addition, the researcher has several years of experience with evaluation of EU programmes, including working as evaluator on the ex post evaluation of LIFE from 2009. Findings were triangulated and validated with document data, and follow-up interviews and interpretations were checked against interview data.

Interviewees were debriefed and were given the opportunity to comment on the findings of the article, and peers with comprehensive knowledge of the subject gave important comments to the draft article before submission.

5.2.4 FOURTH ARTICLE

The research design of the fourth article is an example of a collective of three cases. The case evaluations are midterm or interim evaluations commissioned by the European Commission and produced by external parties using a methodology comparable to that of all European Commission midterm evaluations. The cases are of similar budget sizes for the programme cycle 2007‒2013. Moreover, the programmes work in similar ways through project support and are all managed by the Commission or by a Commission executive agency. The first case is the midterm evaluation of the Programme for the Environment and Climate Action (LIFE). The second case is the midterm evaluation of the environmental research programme within the Framework Program 7 for Research and Development that was conducted in 2010. The third case is the interim Evaluation (2009) of Competitiveness and Innovation Framework Programme (CIP) ‒ Intelligent Energy ‒ Europe (IEE).

The fourth article was based on data generated from 25 structured in-depth interviews from the three case evaluations. The interviews were structured around 20 questions that related general features of the evaluation system to learning from evaluation. The questions probed the design and implementation phases of the evaluation, as well as the learning from the evaluation. Interviewees were sampled purposefully and according to availability, and included Commission officers in the programming unit (the principal learners) as well as Commission officers in the steering committee that managed the evaluation implementation. External experts and consultants directly involved in the evaluation implementation were

77

also interviewed. Interview data were validated with document data comprising the three case evaluations and other relevant documents, such as the impact assessments and ex ante evaluations of the new programme cycle 2014‒2020.

As with the two previous articles, the methodology applied in this article was qualitative content analysis, and the coding and analysis of data was carried out using NVIVO. As described by Mayring, the data was classified (coded) along the lines of the existing conceptual framework from Dunlop and Radaelli (Dunlop and Radaelli, 2013), basically following the interview guide.