evalKit
About the Project
Background
evalKit was a 6-month Joint Information Systems Committee (JISC) funded project, the main research period being October 2002 - March 2003. Updates to the materials were made in August 2004. The research was conducted by the Information Management Research Institute based in the School of Informatics, Engineering and Technology at Northumbria University.
The main deliverable of the evalKit Project was to provide the education community with a web based directory hosting relevant resources and a searchable database of currently available toolkits and tools for the evaluation of Information and Communication Technologies.
The Research
Project methods comprised of:
- Desk based research, to define the area of evaluation and toolkits, to develop the analysis framework for the website, consisting of inclusion criteria and the meta analysis of the toolkits thereby chosen;
- A series of short key informant interviews was carried out at the start of the project, using snowball sampling to ensure representation across the education community;
Input of the project's Editorial Board to the development of toolkit inclusion criteria and the selection of toolkits;
- An Expert Day held February 20th 2003, at the Netskills Training Suite, University of Newcastle-Upon-Tyne with invited participants, representing the breadth of the education perspective in the target domains, tested the prototype website and contributed to the modification of its design; ,
The design of the website is based on the principles of usability and human computer interaction, as developed by Nielsen (2000). This approach is underpinned by the provisions of ISO13407: user-centred design process. This aims to achieve quality by incorporating user centred design activities throughout the website development lifecycle, taking into account the needs of all sectors of the stakeholder community, including the disabled.
Defining Evaluation
Evaluation is an activity central to the improvement and management of an organisation or process. The Merriam-Webster Online Dictionary defines evaluation as a process 'to determine the significance, worth, or condition of an entity usually by careful appraisal and study'. 'The practise of evaluation does not exist in isolation. It is associated with a range of other concepts: performance measurement and benchmarking, quality, validity, effectiveness, value for money, best, value, audit.' (Banwell, 2000). As a result of the wide-ranging nature of evaluation it was necessary that the evalKit Directory took an equally broad view of the concept of evaluation to include toolkits covering the various permutations of evaluation outlined above.
Toolkit Selection Criteria
The selection criteria for the inclusion of toolkits had to be deliberately broad. This was due to the nature of the definition of toolkits. Many of the tools found could not be described as a toolkit as many were stand-alone tools. Stand-alone tools were included in the evalKit database to give the user the widest possible collection of tools to choose from. Toolkits and tools were included if they could satisfy the following criteria:
- Does the toolkit meet the project definition of evaluation?
- Does the toolkit cover the topic of ICT?
- Is the toolkit/tool appropriate for use by the education community?
- Is it clear how the tool can be used?
- What is the authority behind the toolkit?
- Is the tool current enough to be applicable to present ICTs?
- Is the tool written in English?
The Research Team
Project Director: Dr. Linda Banwell
Research Assistant: Gayle Haswell
The Editorial Board Members were:
Ian Winship, Electronic Services Manager, Learning Resources, Northumbria University
Catherine Hare, Senior Lecturer, Division of Information and Communication Studies, School of Informatics, Northumbria University
Sandra Parker, Research Fellow, IMRI, School of Informatics, Northumbria University
Gillian Rutherford, Learning Resource Centre Manager, North Tyneside College
