Data analysis

In the middle of this continuum there is the prior-research-driven theme development approach in which a review of the literature typically “provides insight into the possible development of a thematic code” (Boyatzis 1998, p. 37).

The thematic analysis method was used to describe, organize and interpret aspects of the phenomenon under investigation. In this analysis, themes were generated inductively from the raw data and identified at the ‘manifest level’ i.e. directly observable in the data (Boyatzis 1998).

The approach that I employed to analyze the data was a hybrid approach of the data-driven approach and the prior-research approach. I was reading relevant literature during data collection and data analysis processes and in some cases concepts from the literature were used as sensitizing concepts to make sense of data. However, in other cases, they were used rather unconsciously, since the literature informed me as a researcher regarding the phenomenon under study. As such, in the continuum, my data analysis approach is situated between the data-driven approach and the prior-research approach as illustrated in Figure 11.

Figure 11: Theme development continuum diagram

I began by analyzing the data from the first round of 22 semi-structured interviews.

When reviewing the interview transcripts and background material, together with the field notes recording my impressions at the time of each interview, I was looking, specifically, for indicators of how the BI output was used in the prioritization process as it unfolded in practice. This was an iterative process during which I considered

different codes and reflected upon them until I felt confident about the code. In the following paragraphs I make an attempt to open the box of the coding and theme development process.

My first reading of the transcripts did not yield many codes or themes. As Boyatzis explains, this is a rather common phenomenon for researchers new to thematic analysis. As a result, I decided to approach the transcript in another way. I started to develop a timeline of the IT project prioritization process and its phases (described in detail in chapter 5). This timeline facilitated my understanding of the complex prioritization process.

Next, I went back to read the transcripts again. This time I performed three different activities in order to categorize the data in different ways. While reading in MS Word I applied open coding using interviewees’ words or phrases by making a comment in in the margin (see Figure 12). The codes captured components of raw data across the different sources which varied from a single word to entire passages that captured ideas and themes such as “hard data”, “BI fosters dialogue”, “cost-benefit is free lunch”,

“benefits”, “feelings” and prioritization criteria or irrational behaviors (i.e., outside the official guidelines).

Each interview was coded separately on the basis of in vivo terms or phrases used by the interviewees (i.e., first-order categories (Van Maanen, 1988)), based on the categorization and theme analysis techniques suggested by Miles and Huberman (1984) and Boyatzis (1998). I also highlighted passages that I found interesting but had no immediate instinct for how to code in yellow. Last, passages or information that were related to the process of IT project prioritization were entered in a separate spreadsheet.

Figure 12: An example of coding during the second iteration

In the third iteration, I read the transcripts again and this time I edited different codes, giving them different or better labels. At the same time, I spend more time in the yellow passages to figure out a good code. Once I had settled on a code label, I entered all the codes into a spreadsheet. After entering all the codes that I had identified in the 22 first transcripts I started to do the same for the meeting transcripts and the follow-up interview transcripts. At the end of this process, after many months of work, I had a long spreadsheet that contained all the codes from all the transcripts of the interviews and meetings.

During the next phase of the analysis, codes that were recognized as similar were collated into the same first-order categories, using informants’ language whenever possible. Many researchers agree and recommend staying as close to the data as possible” (Wolcott 1994 as cited in Boyatzis 1998, p. 35) or naming phenomena by

“close examination of the data” (Strauss and Corbin 1990 as cited in Boyatzis 1998, p.

35). At this stage iterative cycles were performed between data analysis and consultation with relevant literature as guides to theme development.

As a result, the first order codes were organized into data-tables that supported a single theme or topic across data sources (Vendelø and Rerup 2011). The interview coding activity continued until it was not possible to ascertain any more distinct, shared patterns among the data. In this manner theoretical saturation was accomplished (Glaser and Strauss 1967). In parallel with the development of the first-order categories, linkages among the categories started to surface and become evident. These linkages were the seeds that initiated the development of second-order themes.

In the next step, I developed the second-order themes by using four key questions to sort through the raw data. The questions were developed based on the understanding of the IT project prioritization process that I had gained to that point. Those questions were:

1. What are the characteristics of the IT project prioritization process?

2. What are the characteristics of the BI output and how is it perceived by the decision-makers?

3. How do decision makers use the BI output in this prioritization process?

4. What kind of information is sought by decision-makers and how is it used in the process of IT project prioritization?

As IT project prioritization consists of two levels, the SSG level and the IT committee level, the questions were asked for each level respectively.

Through an iterative analysis second order themes in relation to the characteristics and use of the BI output at each level respectively emerged as “transparently observable”

(Eisenhardt, 1989, p. 537) phenomena in the data, which I use to present and discuss the findings in chapter 6, inspired by Pratt (2009). Below, I present the data structure (see Table 9) that emerged at the end of the data analysis process for the SSG level as an example. A similar structure emerged for the IT committee level. It should be noted that here only one example of raw data is offered while in reality there are many

examples of each second order theme. The themes consist of theoretically distinct concepts that emerged from the data when analyzed at a more abstract level.

In the final step, the second-order themes were assembled into aggregate dimensions, which led to an understanding of the role of BI in the organizational decision making process of IT project prioritization presented in chapter 6. During the analysis the focus was on capturing phenomena and processes that spanned multiple levels. The emergent dimensions thus depict the use of the BI output as performed by decision-makers in the IT project prioritization process that ultimately grant an organization-level character to the unit of analysis.

The emergent data structures from the SSG level and the IT committee level will serve as frameworks to present my findings in chapter 6.

Based on the above analysis of the IT project prioritization process at Group IT and the use of BI in this process, I performed two workshops. The first workshop included a three hour presentation of the results to the PPMO team members who were in charge of the information specifications, acquisition, standardization and distribution in the IT prioritization and portfolio management process in Danske Bank. This was the team that introduced the BI output as a mandatory input in the IT project prioritization process.

The purpose of the workshop was twofold. Apart from the obvious purpose of presenting the results to the team and increasing their awareness of the problems/issues that the process and the people involved were currently facing, the workshop was also used as a validation tool for the results. That is, the workshop was meant to discuss and validate the results of the analysis I had performed on the 22 semi-structured interviews. The main findings were summarized in a paper that was sent to participants before the workshop. In the workshop I briefly presented the findings and then most of the time was used to elaborate on the different views of the participants.

First order codes (raw data) Second order themes Aggregate Dimensions

“It’s completely in the dark for me how that [project] could find its way to the project plan. Everything, that’s before that [the plan], is an absolute black box and it depends on the manager [development manager]”

(Business Analyst)

Lack of transparency of the IT prioritization process

Process and project characteristics

“There are so many interdependencies between areas and simultaneous effects, it is highly uncertain [the impact of a project in other projects]” (Development Director 2)

Lack of overview of projects’ interdependences

“We really don’t know at this stage, how complex it is. With the complexity of a project of that kind, I actually risk spending double as much money.” (Business Representative 3)

A projects’ technical complexity

“This number [cost calculation] could go up and down as a year expands.” (Head of PPMO)

Cost calculation inaccuracy, and unreliability of benefits calculations

“In 3 or 4 years time who takes care of the money we spent now when we are in 2014?

Nobody!” (Development Director 2)

Lack of benefit follow-up process on the implemented projects

“There is some practical issues involved in actually executing a plan and the limiting factor is really resources and competences.”

(Development Director 4)

Resource capacity and competency constraints

“It's not just benefit-cost ratio, it is much more and if we combined those things then we think that we get a nice view on this [the project proposal]” (SSG Meeting 1.2)

Supplementing the BI output

Tactics of using the BI output

"I don’t think it’s a benefit discussion it’s much more whether actually it’s worthwhile doing it." (Business Representative 3)

Substituting the BI output

“I interpret the reports and I make statements on what I see as the outcome from the analysis of the report.” (IT Finance Manager)

Interpreting the BI output

“So I will not tell them… since I am not being rewarded for being honest about the total costs of ownership, I’ll simply cut down on the information.” (Business Analyst)

Re-framing the BI output

Table 9: Datastructure that emerged as a result of the data analysis process: from first order codes to final dimensions

Many questions were posed to the participants of the workshop about the findings and their opinions on the reasons for the results. They were asked to elaborate on the findings and to provide their own interpretations on the findings. In this way, the workshop led to fruitful discussions both in terms of how the organization could address the problems identified and identifying important aspects of the use of the BI output by different decision-makers. In the workshop the discussions revolved around the validity of the findings with the participants confirming most of the findings.

Specifically, the Head of IT Governance stated:

The picture that you are painting feels right. We have discussed many times that they see benefits as a free lunch and I guess it is right that we should give them incentives in order to emphasize the importance of the benefit follow-up process. (Notes from the first workshop)

Overall, the members of the workshop validated my first results and we agreed that another workshop should be held in the future to follow-up on new results. At the time, many organizational changes were taking place and some of the main points of the workshop were incorporated into these changes.

A follow-up workshop was duly held with some of the participants from the first workshop. In this workshop I reported the newest findings based on an analysis of all data collected. We focused on which problems were addressed by the organizational changes that had taken place as well as what other problems remained unaddressed and how they might be addressed. The participants discussed the final themes and aggregate dimensions that emerged during the final analysis of the empirical data. In general, they agreed with the findings although they were surprised by how common the substitution tactic (described in chapter 6) was used in the decision-making process and thus realised that they had to make more room for other kinds of inputs into the prioritization process through formal information flows. They were also impressed

with the openness that interviewees had shown during the interviews. This workshop thus served as a validation of my findings and conclusions.

Apart from the validation workshops, I also presented the analysis and findings of my research at several conferences and workshops where valuable input was provided throughout this research project.