Research Limitations and Validity of Results

contribute to a thorough understanding of how evaluation methods influence the willingness to invest in open data, both in the public sector and the private sector.

Table 4: Open Data Research Agenda Category Research question Governance

mechanisms

What types of governance mechanisms are ODIs using and how do these types relate to the success/failure of the ODI?

Engagement mechanisms

How are users engaging with open data?

Value generating mechanisms

How is open data used for value generation?

Evaluation mechanisms

How do evaluation methods influence the willingness to invest in open data?

described traits and abilities within a structural framework. Such stories are instrumental in explaining complex results (Yin, 2009). Afterwards, I applied a coding scheme from the theoretical lenses to highlight events in the storyline. A challenge of this approach is that the researcher must use creative insight and careful interpretation to make sense of, and explain observed findings (explanation building), as well as identify events and mechanisms that have not been observed. Reliability can be difficult to establish in such a study. In an attempt to improve conformability, I used secondary sources of data to corroborate my primary interview and observation data.

For the Basic Data Program case study, I had access to a number of program documents as well as to an external review of the program. For the Opower case, I used a Harvard Teaching Case and a published economics paper that reviewed Opower´s methodology for evaluating energy efficiency gains. Additionally, I made use of online material and internal documents.

To create credibility is always an issue in case study research and is certainly a limitation of the qualitative part of this research. To improve credibility I had research participants review my narrative and correct factual errors. Of course, all interpretations and conclusions remain my own (and my co-author´s in the Opower case). To introduce traceability, I created an Outlook meeting request, which the interviewee approved. While most interviews were recorded, technical difficulties prevented this in two cases, and descriptive notes were relied on. Both case studies are single cases and thus it is difficult to generalize outside of the specific context in which the events happened. In order to make the context more explicit and introduce a level of transferability (Venkatesh et al., 2013), I used secondary data to create a thick context description so that the readers could themselves evaluate how the context influenced the events.

There are also a number of limitations related to the quantitative work.

Firstly, the sample size used in in both PLS studies (Paper III and Paper VII) was small as there is limited secondary data to be found that are consistent across multiple countries and are fit for the purpose of reflecting different degrees of openness of government data. I was essentially limited by the number of observations in the Open Data Barometer. However, I have reasonable evidence to believe that the model contains enough power to draw preliminary conclusions from results.

Secondly, while the majority of the secondary data providers are reliable and have a long history of collecting and disseminating data, the Open Data Barometer is relatively new as it was constructed for the first time in 2013, collecting data for 2012, which was used in the pilot PLS study. However, I participated in the data collection

myself and as such can verify that scientific methods were used in the process. The data in the Open Data Barometer are based on an expert survey, using similar methodology as the Web Index and the World Economic Forum´s global technology reports and global competitiveness reports.

Thirdly, comparing impacts from open data between countries, where many ODIs are still in their infancy, might be premature. That being said, there are types of open government data that have been available for many years although worldwide interest in open data is relatively recent. For instance, Obama´s open government initiative was born six years ago, and even at the time, important types of data such as geographic data and weather data were already available free of charge. In Denmark, address and property data were made available free of charge in 2003.

The use of mixed methods introduces its own limitations. While I believe my conceptualizing efforts benefitted from this approach, there are limits concerning triangulation of different types of data and research methods. For instance, some attributes of open data are more difficult to measure than others. It is relatively easy to estimate whether data are published under open licenses, or if they are disseminated in machine-readable formats. However, it is much more difficult to evaluate qualitative attributes like data quality as they depend on the nature of use, the needs of the users and other less generalizable attributes. The quality of basic data was certainly one of the main areas of emphasis in the BDP. However, in the econometric modeling, I could not find appropriate societal level measures that reflected data quality. While quality aspects or dimensions do appear in the final liquid open data construct, I could not emphasize the importance of data quality in the quantitative work to the same degree as was done in the case studies. Accordingly, not all of the insights from the qualitative studies could be carried forward to the quantitative studies.

There are also limitations and biases related to the researcher herself. I needed to familiarize myself with PLS and a case study research methods when I started working on the PhD study, and as such much time was spent learning the two approaches. I am relatively confident of the robustness of the qualitative analysis due to fact that the quality indicators in quantitative studies are well documented and easily comparable between studies, and I had relevant statistical experience from previous education and work. However, when reflecting on this PhD study, I conclude that it is difficult to learn the skills required for qualitative data analysis from a book or a single PhD course. I believe these skills are best acquired through experience over time, and conclude that this is an area with growth opportunities for myself as a researcher.

While it is hard to evaluate one´s own biases, I suspect I have leaned towards

positivism in spite of my focus on the CR-based mechanism approach. I also must make explicit that I truly believe open data offer the opportunity to generate value, which does make me biased in my evaluation (which is incidentally not a very positivistic trait).