PART 3: ANALYSIS
10: R ELATIONSHIP BETWEEN THE ATTRIBUTES AND THE METHODS
10.4 The relationship between the attributes and the methods in IT evaluation
The relationship between the parameters’ attribute(s) and the four IT evaluation methods in the group called, IT evaluation, is described in the following.
10.4.1 Purpose of IT evaluation
The relationship, as shown in Table 32, between the purpose for completing the IT evaluation and the four methods, is primarily based on the methods’ need for inputs and the delivered output. For example, considering the purpose, Manage an IT investment. It is necessary to input empirical data (data representing an actual progress), and the output need to present a detailed picture of how the IT investment’s progress is, and this is done best if the output is multiple. All the methods have some strengths and weaknesses in relation to the four purposes defined in chapter 9. Some of them have even been developed specifically for fulfilling one of the four IT evaluation purposes.
Table 32. Relationship between Purpose of IT evaluation and the four IT evaluation methods
NPV MBITI IE CSF
Purpose of IT Evaluation
Identify the best IT investment X (X) X (X)
Evaluate impact from IT investment X X X
Manage IT investment X (X) X
Increase knowledge, skill and experience of IT
evaluation X X X X
The influence on choice of IT evaluation method is, for each of the four purposes, described below.
Identify the best IT investment
The first purpose indicates that more than one IT investment has to be evaluated and that they then will be compared with regard to the evaluation output. The last step is the
prioritisation of the evaluated IT investments so that the highest scoring is ranked first. The input to the IT evaluation is based on estimates rather than actual measurements. The choice of IT evaluation method would therefore have to focus on the ability to compare the IT evaluation output (based on estimates) for different IT investments, and this is done best if the output of the method is single in nature.
Two of the cases (NIRAS and H&S) used this purpose in their IT evaluation. Both of the IT evaluations were completed on the basis of estimates as input, but they also had some experience in using the technology from earlier work which occasionally provided input to the IT evaluation. Comparing the IT investments was done most easily using methods that provided a single output.
Evaluate the impact of the IT investment
Having this purpose for the IT evaluation indicates that only one IT investment is evaluated. The main activity is evaluating the IT investment for the company, in most cases, based on estimates. The value is very dependent on the IT evaluation criteria used and should therefore be chosen so it supports this activity. To do this will require an ability to analyse the resources spent, and compare them with the impact on the company using the IT investment.
Two cases (Rambøll and TJAS) were pursuing this purpose for their IT evaluations. At the time of the IT evaluations (implementation stage for both case studies), the impact of the applications was evaluated by estimating the costs and benefits given by the existing and expected conditions. The need for detailed output that enable an overview of the impact from the IT investment, was found to be beneficial.
Managing the IT investment
If a company evaluates an IT investment for this purpose it will focus on one IT investment in which the current status is identified. The IT evaluation’s output will provide the IT evaluation champion(s) with an assessment of the IT investment’s status/progress at the evaluation time with regard to the criteria used. The choice of IT evaluation method is almost similar to the previous purpose, but the usage of the output is different and may in some cases mean a difference in choice of method.
The case study with Cowi evaluated the progress of their IT investment: DWS. Like the case studies with Rambøll and TJAS (described above) detailed output were found as beneficial for this purpose.
Increasing the knowledge, skill and experience in IT evaluation
The last purpose is, in several senses, different to the first three. The important aim is not the IT evaluation output but the process of completing the IT evaluation. The choice of IT evaluation method will therefore depend on different reasons than were stated in the previous purposes. The method should be chosen on the basis of the company’s current level of IT evaluation practice, and the two possible sub-attributes either use a method corresponding to their current level (so they can improve the knowledge, skill and experience by using such a method), or to increase the level by using a more advanced method (this will enable the company to get an insight into IT evaluation by using a more advanced method).
Three out of five case studies (Rambøll, Cowi and H&S) were completing the IT evaluation for this purpose. The reason for this was that the companies used the case studies as an opportunity for improving their knowledge and experience. All four methods were seen as a tool to fulfil this purpose which supports the fact that they can be used to increase the knowledge, skill and experience in conducting IT evaluations.
10.4.2 IT evaluation criteria
The crosses in Table 33 are derived from the characteristics of the four methods that can be found in chapter 3. Most notable is that two criteria, Quality improvements and External satisfaction, are not supported in any of the four methods. The method, CSF, supports the criterion, requirements, whereas the other criterion included, end-user satisfaction, is not directly supported but may indirectly be assessed using the method.
Table 33. Relationship between IT evaluation criteria and the four IT evaluation methods
NPV MBITI IE CSF
IT Evaluation Criteria
Financial X X X
Strategic X X
Competitive X
Effectiveness of use X
Quality improvements
Requirements X
End-user satisfaction (X)
External satisfaction
Technology X
Risk X X
The most used criterion in the case studies was Financial (all five case studies). Many of the available methods are focused on this criterion (this is also reflected in Table 33). Three of the case studies were also focusing on other criteria like strategic value, requirements and risk.
10.4.3 Format of output
The first part of Table 34 is focused on the type of output from the methods and the second part on the multiplicity of the methods’ output. Both parts are derived from the characteristics of the four methods (for a more detailed description of the methods see chapter 3).
Table 34. Relationship between Format of Output and the four IT evaluation methods
NPV MBITI IE CSF
Format of Output
Financial X X X
Ratio X X
Qualitative X X X
Requirements X
Single X (X)
Multiple X X X
Only a few of the case study companies had decided more precisely which format of output they wished as a result of the IT evaluation. Most of the case study companies were focused, however, on evaluating the financial value. Rambøll was focused on the more qualitative output, as they expected that the financial value would not show a positive output.
The main experience from the case studies regarding the multiplicity of the output was related to the detail level. Some of the case study companies were interested in the overall value of the IT
investment, where others were interested in the details. H&S was comparing two alternative IT investments, which is best done by using methods that provide a single output. The IT evaluation champion in this case study did, however, say that the methods with the detailed output were the most revealing. Cowi were most interested in methods that provided detailed output, primarily because they wanted to be able to develop a monitoring plan to achieve the financial benefits.
10.4.4 IT evaluation champions
Table 35’s content is derived on the basis of the organisational levels’ data accessibility and the methods’ data requirements. In general the levels’ data accessibility can be described as going from data representing the company’s overall status (strategic), to data about managerial issues (tactical), to detailed data representing the actual working procedures (operational). The methods’ data
requirements are analysed and from that the relationship between the parameter’s attributes and the methods is determined.
Table 35. Relationship between IT evaluation champion(s) and the four IT evaluation methods
NPV MBITI IE CSF
IT Evaluation Champion(s)
Operational (X) (X)
Tactical X X X X
Strategic X X X
Depending on which of the three company levels the IT evaluation champion(s) are placed might have some influence on the choice of method. The influences are described in the following.
Operational
In IT evaluations where the champion(s) is/are placed on the operational level, it implies that the requirements for the IT investment need to be identified. The complexity of the IT evaluation method will also have to be small, because the employees from this level are not familiar with IT evaluations, and the data available is focused on operational characteristics.
The choice of method will therefore have to fulfil these aspects.
None of the IT evaluation champion(s) in the case studies were primarily from the operational level, although this was a part of their job for a few of them.
Tactical
If the IT evaluation champion(s) is/are from the tactical level of the company, it generally implies that the IT evaluation is either making an IT investment proposal for approval by the strategic level, or to manage the IT investment. This level does most of the IT evaluation.
The choice of method needs therefore to be a method that is suitable for one of these purposes. In order to enable a more detailed picture of the tactical level’s managerial tasks, two sub-attributes are defined.
o IT investment proposal o Managing IT investment
All five case studies were completed with IT evaluation champions, who were categorised as belonging to the tactical level. These are, as defined, working daily with management tasks either related to building projects or IT. The IT evaluation champions had, in general, a high level of knowledge of the company’s business activities and usage of IT (some on a user level and others on an expert level). The case studies were affected by a bias towards methods that provide detailed output.
Strategic
When the IT evaluation champion(s) is/are from the strategic level, then the IT evaluation in most cases would be focused on the strategic aspects of the IT investment. Furthermore the IT evaluation method should not be troubled with a high detail level because this is too time