This chapter describes the research design and data used in the three articles. First, I describe the general dataset and the data collection methods. Following this, I will provide an overview of the methodologies used in each article: qualitative content analysis, social networks analysis and regression analysis.
5.1 Data
The thesis is based on both primary and secondary data. The primary data were collected through interviews with policy makers and experts, the secondary data are based on policy documents. In substance the dataset consists of two parts: information on evaluation practices and data on informal networks. The dataset is focused on the 28 member states of the European Union that constitute the
31
population of this study. I focus on the EU countries, given the substantial initiatives that have been undertaken over the years in both academic literature (Borrás, 2004) as well as by international organisations (such as the European Commission and OECD) to encourage mutual learning and the spreading of good practices among the policy makers. Accordingly, the population of the EU member states provides a good opportunity to study the possible differences among countries belonging to a broadly similar policy space.
The primary data were collected through interviews. In total, 62 interviews were conducted1, 52 with policy makers and 10 with academic or independent experts. The aim was to carry out at least two interviews per country, one with a senior innovation policy manager for a strategic perspective and one with a senior policy evaluation manager or expert with in-depth knowledge about the evaluation practices for innovation policy. Most of the interviewees were from national ministries or agencies responsible for innovation policy. In some cases where it was perceived that additional information was required, academic or independent policy experts were also interviewed.
The interviews combined semi-structured and structured designs. The former was required to provide space for improvisation when deemed necessary by the interviewer, and the latter to provide uniformity in the way the interviewees were presented with the questions. The interview guide consisted of two main blocks, the first using a semi-structured approach and the latter a structured one. The first block focused on the national evaluation practices, for example on the types of evaluation carried out and their frequency. The second block looked at the informal networks of policy makers. The questions about the informal networks were not asked from the academic or independent experts, as, for the purpose of this thesis, I was interested in the networks of policy makers only.
In order to verify and complement the data collected through interviews, I acquired secondary data from policy documents. The types of documents included, for example, evaluation reports and national evaluation strategies. The documents were received in part from the interviewees and in part from public sources, such as the RIO database (Research and Innovation Observatory) and the SIPER database (Science and Innovation Policy Evaluation Repository). We triangulated the secondary data with the primary data to check for possible mismatches.
5.2. Methodologies
Each of the three articles of this dissertation is guided by a specific research question and therefore all of them required a different methodological approach. Article 1 uses qualitative content analysis, Article 2 employs social network analysis and Article 3 utilises regression analysis. I will proceed by describing the methodological approaches of each article separately.
1 Susana Borras kindly conducted some of the interviews.
32
5.2.1. Article 1 – quantitative content analysis
The empirical aim of Article 1 was to explore the extent to which national evaluation practices match our concept of system-oriented innovation policy evaluation. The data used for this article included the primary data from interviews and the secondary data from policy documents. Combining these data sources and checking them against each other enabled me to build a solid dataset on national evaluation practices that would be used for subsequent comparative analysis.
In order to analyse the qualitative data on each country and to be able to compare countries against each other, we used qualitative content analysis (Kohlbacher, 2006; Schreier, 2012). Based on our analytical framework (see Table 1 in Article 1) we assigned values on a three-point scale (from ‘0’ to ‘2’, according to their intensity) to each of the conceptual attributes. Arguably, the quantitative value assignment loses some of the depth of information as compared to qualitative value assignment. However, given the relatively large number of cases under observation, the quantitative value assignment was necessary to enable a comparative perspective across the 28 countries and the conceptual attributes.
5.2.2. Article 2 – social network analysis
The second article aimed at exploring the structures of the informal networks of policy makers in innovation policy and analysing their consequences for policy learning. To fulfil this empirical aim, I made use of the interview data on informal networks. During the interviews with policy makers the interviewees were presented a list of EU member states and asked to rate each country on a four-point scale (“often”, “sometimes”, “rarely”, “never”), according to the intensity of the perceived interaction2. In order to establish a coherent and comparable subset of data, I included only the interviews conducted with the national head of innovation policy or equivalent3. This was necessary in order to ensure that the perspective from each country would come from a roughly similar level and thus ensure comparability across countries. In addition, one can assume that the heads of innovation policy are in a good position to adopt a general overview of the informal networks in their area of responsibility.
2 The exact wording of the question was: “Please mark how often do you exchange views on innovation policy with the following countries”.
3 In a small number of countries where the competence of innovation policy is equally divided between two government offices, I merged the answers of the two respective heads of innovation policy.
33
Table 3. Methodologies and methods used
No. Title Methodology Methods Data sources
1 Towards System-Oriented
Innovation Policy Evaluation?
Evidence from EU28 Member States (co-authored with Susana Borrás)
Quantitative content analysis
Semi-structured interviewing
Document review
Transcribed interviews
Documents and websites
2 Policy learning in the EU:
The informal networks of innovation policy directors
Social network analysis
Structured interviewing
Transcribed interviews
3 The Rules of Attraction:
Informal Networks of Innovation Policy Makers in the EU28
Regression analysis
Structured interviewing
Transcribed interviews for dependent variables
Public databases for independent variables
I analysed the data using social network analysis (Scott, 2017; Wassermann & Faust, 1994). This method is specifically designed to analyse the structures of social relations in a particular setting. A social network is perceived as a combination of nodes and ties, where the nodes are actors and ties are the interactions between them. In my dataset I assigned values to the ties between countries based on the intensity of data. In order to mitigate the possible discrepancies in interpreting the four categories of intensity by the interviewees, I treated ‘often’ and ‘sometimes’ as ‘1’ and ‘rarely’ and ‘never’ as ‘0’. This binary dataset allowed me to establish a robust dataset of actors and the ties between them.
34
5.2.3. Article 3 – regression analysis
In the third article I set out to analyse the determinants of the network structures, in other words, what are the variables that are most likely to determine whether there is a tie between two countries or not. In order to answer the question I first distinguished between dependent and independent variables. The two dependent variables were ‘the existence of a symmetric tie’ and the ‘the existence of an asymmetric tie’, both based on the network data collected through interviews. The independent variables included a range of indicators under the categories of geographic, policy and cultural proximity (see Table 1 in Article 3 for a detailed overview). The independent variables were based on publically available data sources, such as the Doing Business Index or the World Borders Dataset. Furthermore, two control variables – ‘GDP per capita’ and ‘population’ – were used, both based on data from Eurostat.
The effects of the various factors on the existence of ties were estimated through regression analysis.
More specifically, I used logit regressions (Menard, 2002), which are commonly used for analysing binary data and therefore well suited for analysing dyadic network data. I build six models by adding independent variables one at a time and testing them. This allows for the successive assessment of the effects of each individual variable.