Author: Simon Wilkinson, simon.wilkinson@nottingham.ac.uk
Author: Heather Rai, heather.rai@nottingham.ac.uk
JISC e-Learning Activity Area: e-Assessment
Higher Education Academy Subject Centre: Medicine, Dentistry and Veterinary Medicine
This case study illustrates...use of specialist software, an effect on learning, an effect on exam results, an effect on student personal development, student satisfaction with e-learning, innovation in learning and teaching, an influence on educational research, staff satisfaction with e-learning, staff personal development, a positive effect on retention, an influence on policy, use of resources, modifications to learning spaces, management of learning assets, an effect on social equality
Background & Context
Why did you use this e-learning approach?
The University of Nottingham Medical School wished to switch from optical mark recognition (OMR) method of scoring objectively marked questions to a computer-based assessment (CBA) solution for two reasons. The first came from time pressures of marking increasing cohort sizes. The time it takes to mark using OMR scales linearly with cohort size. With limited numbers of scanners this was impacting on the scheduling of multiple examinations. The second impetus came from a desire to create more realistic questions. Teaching staff wished to incorporate full colour microscope slides and high resolution radiographs which are difficult and expensive to accurately reproduce on paper. Also, the ability to utilise interactive question types such as drag and drop labelling and image hotspots not possible with paper drove the move to CBA.
What was the context in which you used this e-learning approach?
The University of Nottingham runs two Medical degree programmes: a five year undergraduate degree and a four year accelerated graduate-entry degree. The combined number from both programmes is approximately 1,900 students with approximately 1,000 staff registered to use the virtual learning environment. The graduate-entry cohort is based at Derby and is predominantly taught through problem-based learning techniques in 11 modules. The undergraduate cohort complete most modules out of a total choice of 59 and then both groups integrate to study the final 13.
Both programmes utilise a wide variety of assessment forms including: groupwork, essays, projects, OSCEs, OSLERs and objectively marked questions. Historically the true/false/abstain question type dominated this last form of assessment. However, recently there has been a drive towards other more reliable question types such as multiple choice and extended matching. These types were administered on machine readable paper sheets which were then scanned through OMR hardware. The main problem with this form of assessment is the time required to scan is directly proportional to the size of the cohort. Also, unclear changes to answers on the paper form are difficult to process and frequently require human intervention which again slows the process.
Anticipated problems moving from an OMR method to a CBA approach revolved around two central issues. The first was where to find computer labs large enough to examine the cohort sizes of the Medical programme. The solution adopted was to utilise two back-to-back 'sittings' of a particular cohort in the same lab. The second problem regarded additional possibilities for plagiarism afforded by the move online. Staff were worried about: 1) students being able to see other student's monitors, and 2) being able to access forbidden materials during the examination such as Google or their notes off a pen drive.
What was the design?
The initial design three years ago primarily concerned exam delivery online; working with academics to get questions into the system and to correctly set the permissions so that only authorised students could sit each assessment. Building on initial successes the design has expanded considerably to try and generate a complete electronic workflow process to support online summative exams. The workflow starts with question writing and item storage, review and editing with a change audit system in place, providing external examiner access, conducting standards setting, making adaptations for disabled students, through to exam delivery and reporting. Many different stakeholders are involved starting with academics creating the questions, administrators performing room booking/timetabling, subject matter experts conducting standards setting reviews, external examiners, disability experts and IT support personnel. The aim of computerising the whole workflow process is to enhance quality. Utilising a web-server the process can be distributed to any client computer and the data nightly backed up. In the case of any of the key stakeholders being ill or not being able to work, it is easy for another member of staff to log into the system and obtain the information. There are no problems with work being held on different peoples' personal computers that may or may not be password encrypted.
How did you implement and embed this e-learning approach?
Half day summer workshops for staff training were arranged within the medical school alongside a comprehensive online help system with separate areas for staff and students. One-to-one training was given to key administrators, tailored to their roles. Reports of successes and failures were circulated to the main curriculum committees and student questionnaires were collected for non-summative exams
The number of people sitting each assessment was gradually increased to test the load on the system. Initially the system was tested with a few resit students, then a simulated load was tried with around 30 volunteers starting a paper within a few seconds of each other. The next test was with 90 students on the Graduate Entry Programme then 130 for a first year exam and finally a large cohort of 170 students taking their Advanced Clinical Experience (ACE) exams. At each stage the response times of the system were noted.
In response to question 2, the responses to the problems were:
- a large computer lab was found in the School of Computer Science, this lab holds 150 students. Larger numbers can be seated by using two cohorts of students and a second, smaller, computer lab simultaneously.
- An 'exam desktop' was created. This used group policies in Windows XP to restrict access to no applications apart from Internet Explorer and no web site apart from TouchStone. It also restricts access to any drives and system settings.
