Policy Considerations: What is your institution's position?
You will need institutional support to determine the optimum system for your institution, because otherwise you will have issues related to technical and adminstrative support and upgrading, and staff and students may have to learn to use a number of different assessment engines and user interfaces. You will need to check on what your institution allows you to do in terms of policy implications and institutional impact.
Sim et al (2004: 222) 'The perceived benefits of CAA of freeing lecturers' time can be illusive if no institutional strategy or support is offered, successful implementation may be left to chance and CAA may be developed in an anarchic fashion. In order to utilize the features within software packages staff training and development is necessary and this may not be feasible without institutional support'.
If you are utilising, or proposing to utilise, e-assessment across major sectors your institution will need to complete an end-to-end analysis of the systems you will use and the experience that the students will receive. E-assessment is now a proven technology, and therefore staff and student users expect secure, reliable and flexible systems that are tailored to their learning and assessment requirements. For those managing these systems, these expectations have to be delivered with the added constraints that systems should be cost effective and sustainable.
Components of e-Assessment
e-Assessment requires the provision of assessment software, server & network infrastructure, student PCs, administrative staff support and questions. To create questions and assessments across disciplines a staff education programme is normally required to ensure the staff have an accurate knowledge of the capabilities and limitations of online assessment. Publishing written policy and procedures ensure that all the assessment stakeholders - students, teaching staff, administrative staff, specialist support staff, external examiners and academic & quality assurance managers, can be clear about the process and can have confidence in the operation of e-assessment.
Policy and Procedures, Roles and Responsibilities
Designing, building and operating an institutional-wide e-assessment system is not a small undertaking. It is likely to involve specialists in e-Learning, educational systems, information systems, academic departments, student IT support, quality assurance specialists and academic management. For people from these groups to work together, agreeing roles and responsibilities within policy and procedure documentation, ensures they can each apply their professional strengths and contribute to a successful institutional e-assessment system.
The University of Dundee Policy and Procedures for Computer-Aided Assessment have been in use since October 2002, although they are regularly revised. These procedures are available for others to utilise as appropriate (with due acknowledgement), but it is recognised that they are also limited in that they are written for the specific software, hardware and staffing configurations at Dundee. Another key resource is the Bristish Standard BS7988:2000 - Code of practice for the use of information technology (IT) in the delivery of assessments. Specific areas of importance are highlighted below.
IT suites and the presentation of invigilated assessments
The physical environment for taking assessments needs to be familiar to students and spacious. Screens should not overlook each other, and frequently a back-to-back arrangement is space efficient and secure. Questions may be presented to students in a random order and also even in a structured assessment, students will move at a different pace through the assessment. Some institutions favour individual screened corrals, but these carry the additional risk of participants being shielded from invigilators observation. The University of Dundee, found the use of screens unnecessary. Colchester Institute report in Case Studies of MLEs in Further Education that 'plans for the introduction of e-assessment are posing real challenges to the institution in estate terms. The computer suites are designed in 'daisy-pod' style to facilitate interaction, not as the face-on single units required by the examination bodies'. This example demonstrates that flexibility of learning space is an important consideration.
Invigilation of examinations should be carried out following the institutions conventional examination procedures.
Open Assessments and Personal Authentication
A common topic for debate with e-assessment is how a tutor can be sure that the student completing the assessment is the correct student on their own. The simple truth is that unless the assessment is delivered under invigilated conditions using secure systems, then you cannot be sure. However, many conventional assessments are taken in an open environment - in course essays for instance, and for low stakes assessments, it is often appropriate to adopt an open strategy requiring the completion of a variety of small assessments. Subsequent invigilated assessments provided a check on the open assessments.
Advanced mechanisms for personal identification are being tested (biometric keyboard use patterns, iris scans etc) and some have proposed the use of webcams to monitor participants. Dundee found the development of such approaches unnecessary.
Reliability of assessments
All institutions will aim for the highest level of reliability for online assessments. This is crucial, especially in the early days of running a large online assessment programme. If the staff and students experience an unreliable system, and then experience it again, they will loose confidence. At the University of Dundee they take particular care when a department is running a high-stakes summative e-assessment for the first time. Learning technologists and IT specialists visit the assessment for the first 15 minutes to ensure the assessment proceeds smoothly.
No system can be 100% reliable. Backup or redundant procedures should be in place to cope with the unexpected. At Dundee the backup procedures involve an entirely independent e-assessment system that is available for deployment within 15 minutes to a group scheduled to receive an assessment. Some institutions have paper copies of assessments available, but often this is not possible because of the expanding use of multimedia, innovative and flexible questions in assessments that cannot be replicated on paper.
VLE or Specialist e-Assessment systems
Software for the delivery of e-assessments is often contained within commercial VLEs or specialist e-assessment providers. An institution's VLE has the advantage that it is supported, built on a secure servers and familiar to staff and students. Specialist e-assessment systems require additional servers and support and additional tuition for students and staff, but normally offer more flexible and powerful systems.
Experiences of different institutions are varied. A major concern in the use of VLEs for summative e-assessment is that they are large, complex software and often simultaneously accessed by many individuals and groups of students from many different locations. VLE software (in 2005) is still relatively young and some systems are not as reliable or robust as may be reasonably expected. Successful use of VLEs for summative e-assessment delivery can involve timing assessments to the evening or weekends when normal usage is reduced, and specialists are on hand to monitor the systems. At the University of Dundee, the use of specialist e-assessment software (Questionmark Perception) has proved robust, flexible and cost effective. Undoubtably more complex in system design, we find that the ability to run assessments at any time using robust and redundant systems justifies the additional setup and staff development costs.
Load testing on servers
Developers of e-assessment systems are aware that their systems will be utilised with large numbers of students in a complex networked environment. Designing and deploying systems to meet institutional needs is a complex and specialist task. Load testing is frequently difficult as mimicking the activities of (say) 200 students completing an interactive assessment over a busy network cannot be easily replicated. One important tip is that even the most highly specified system, composed of a load balanced array of the latest servers, may struggle to meet the demand of simultaneous access by large numbers of students. It is good practice to make instructors and students aware of this limitation. At Dundee, assessments are started (and therefore finished) within a window of 2 to 5 minutes that effectively spreads the server load and all participants receive fast responses from the server.
Emerging e-Assessment Techniques
Alongside what is now traditional e-assessment used for the delivery of online tests and examinations, are the developing areas of e-assessment which include online submission and marking, plagiarism detection, ePortfolio assessment and assessment of contributions to asynchronous and synchronous discussions. Developing policies in these fields will involve consultation with academic staff as they evaluate these assessment mechanisms and policies should also take in to account accepted practice from published leaders in these fields.
A convention that Dundee aims to develop is that the online submission of student work is acknowledged immediately after the deadline, and that care is taken to ensure that the assessment outline are clear and the mechanism and location for delivery well understood.
A case study of e-assessement at the University of Dundee is available.
