Approaches to analysis
Having access to a broad range of reliable data sources, such as those indicated in the previous section is clearly essential but not, alone, sufficient to ensure effective monitoring of progress is achieved. If the data generated and gathered represents the 'ingredients' then what are required are the equivalent of 'recipes': different approaches to combining them in order to achieve a desired result.
The activities and initiatives being carried out to make progress towards achieving your agreed strategic objectives in pursuit of your stated aims will be occurring at different levels across the institution. The monitoring of KPIs may provide evidence of the 'big picture' across the institution but may not alone present the complete picture - especially when it comes to tracking progress at a programme or project level which may in themselves be considered 'operational' in nature, but upon whose successful completion the realisation of strategic aims depends.
The JISC-funded Benefits of ICT Investment - Landscape Study (BILS) conducted by the University of Strathclyde identified a range of evaluation methodologies, each with their own strengths and weaknesses.
These include:
Financial Evaluation
Financial Evaluation involves identifying increased revenue streams or cost reduction that can be attributed, in whole or in part, to the investment. While increases in income streams or reductions in cost are often claimed for ICT investment projects, there are often difficulties in using Financial Evaluation as an evaluation methodology.
Target Setting
Evaluation of an ICT investment may be made against specific, measurable targets, where a desired benefit is identified, a measurable value placed on it and measurement of that target is built into the ICT implementation. SMART (Specific, Measurable, Assignable, Realistic, and Time-Constrained) Targets can be used. More information on setting SMART targets.
Benchmarking
The strength of benchmarking is that it provides a methodology from which institutions can exchange ideas and best practice, whilst also evaluating progress. However, the value of benchmarking is variable depending on the context of the activity being benchmarked. For example, a technical benchmark (i.e. processing capacity) may be well-defined and easy to replicate. However more caution is required in drawing comparisons in more context-specific and complex activities, such as e-learning.
Information Review
This can be defined as evaluation through review by management committee, comparing perception of outcomes against expectation. Often this may be based on no more than anecdotal evidence, narrative reports and the judgement of the staff involved.
Investigative
Some investment is speculative in nature i.e. where an investment is made to allow a technology or approach to be piloted, or where a pilot service is offered to build capacity or test demand. In these circumstances, it would be inappropriate to set targets, as the investment is exploratory in nature. However, it is important that sufficient information is gathered during the investigation to allow an evaluation to be conducted at its conclusion and to ensure that future decision-making is properly informed.
Customer-focused
In FE and HE, customer-focussed evaluation focuses primarily on canvassing student and/or staff opinion of ICT investments and resultant services through the use of tools such as questionnaires, surveys and focus groups. The strength of this approach is that it focuses on the view so the service recipients, the most important stake-holder group. It also allows a range of views to be gathered. However, the results are inevitably subjective and, if the evaluation is poorly designed, they can be misleading.
Compliance
Investments can be evaluated in the context of compliance with legislation and/or internal policy, through techniques such as sample testing (e.g. website compliance with disability legislation) or exception reporting (e.g. security breaches).
External Accreditation
Evaluation can be conducted in the context of external quality control procedures such as External Audit review of information systems, quality assurance of teaching, compliance with ISO standards of other quality Kitemarks (e.g. COBIT, etc.)
It is clear that no institution will adopt all of the above and that there is a need to adopt the most appropriate framework for the task in hand.
The Benefits of ICT Investment Landscape Study (BILS) categorises the evaluation models listed above based on two differentiating factors:
- Evaluation Type. This describes the spectrum of methodologies that ranges from entirely quantitative in nature, to those that are entirely qualitative in nature
- Context. This describes the type of comparison that is made as part of the evaluation, and this ranges from an entirely internal comparison (where benefits are measured against internally-developed comparators such as targets or key performance indicators) to external comparisons, where results are measured against externally derived benchmarks
According to the BILS study, if these two differentiating factors are used as axes in a quadrant diagram, the individual evaluation methodologies can then be mapped as individual points (see Figure 1 below). While the list of Evaluation Methodologies included in this diagram is not exhaustive, this framework will accommodate any relevant methodology. It should be noted that this mapping is not an exact science, but is instead intended to show a relative positioning of the methodologies in relation to the two differentiating factors.
Figure 1 : Evaluation Framework
Taken from Benefits of ICT Investment - Landscape Study (BILS) conducted by the University of Strathclyde, 2008
The BILS study also suggests consideration of the following factors when choosing the most appropriate approach in a given context:
- Type of Investment
- Scale of Investment
- Type of Benefit
- Scope of Benefit
- Complexity
- Alignment
- Availability of Comparison Data
- Type of Decision
- Institutional Culture
