Author: Simon Wilkinson, simon.wilkinson@nottingham.ac.uk

Author: Heather Rai, heather.rai@nottingham.ac.uk

JISC e-Learning Activity Area: e-Assessment

Higher Education Academy Subject Centre: Medicine, Dentistry and Veterinary Medicine

This case study illustrates...use of specialist software, an effect on learning, an effect on exam results, an effect on student personal development, student satisfaction with e-learning, innovation in learning and teaching, an influence on educational research, staff satisfaction with e-learning, staff personal development, a positive effect on retention, an influence on policy, use of resources, modifications to learning spaces, management of learning assets, an effect on social equality

Tangible Benefits

What tangible benefits did this e-learning approach produce?

Administrators

As with a 'traditional' assessment life-cycle, involving external examiners to review the academic standard of a paper is equally important when moving to a computer-based approach. TouchStone has been designed specifically with this in mind; it is possible to create special log in accounts for external examiners and to explicitly specify which examiners should review which papers. The advantage of this approach is two-fold: 1) log in details can be easily emailed to externals who have immediate access to the paper(s), and 2) system administrators can check whether an external has reviewed a paper by a particular deadline.

Another advantage is that there is now a single place of storage for questions, past papers, student profiles (year of study, photo etc and marks) and exam results and that these are interconnected e.g. a student profile links to the papers that student has taken and this can in turn link to the marks for a cohort on that paper.

Marking time is greatly reduced with online exams. In March 2007 over two days 330 students each sat Paper I and Paper II for the Advanced Clinical Experience (ACE) module. If OMR was used these exams would have taken a total of around 10 hours to scan; using the CBA approach outlined here required about 2 seconds to bring up each of the two reports. This means that marks can be made available to staff as soon as the exam is finished (and can even be monitored during the exam) and can be viewed on screen or exported into Excel, CSV format or XML. Item analysis can be used immediately after the exam to identify potential problem questions and the marks for each question for every student can also be exported, if any further item analysis is necessary.

It has also been possible to gain flexibility in timetabling as there is no longer a bottleneck in exam marking due to hundreds of papers needing to be fed through one OMR machine. This has meant that one exam could be brought forward by one week this year and therefore provided more time to review marks has been created. A great deal of paper has also been saved with the move away from OMR. This also saves printing time as each OMR answer sheet needs to be printed individually.

Academics

Switching to online assessment has opened opportunities for the use of multimedia and more interactivity in exams. Labelling questions involves the student dragging labels into placeholders over an image or diagram.

This is pedagogically similar to a multiple choice question with a diagram but there is less cognitive load allowing the student to engage more with the subject matter. Hotspot questions are again based on an image but the student is required to place a marker on an area of the image defined by the creator of the question.

Both of these question types are worth highlighting as they are not easily possible on paper. The other advantage with image hotspot question is a reduction in the probability of getting the answer correct through guessing. Instead of the candidate being presented with five discrete options to choose from, they can instead click on literally any pixel on the image.

In fact any kind of image is much easier and cheaper to reproduce on screen than on paper, even for the less unique question types. The Medical School tried printing radiographs (x-rays) as greyscale images on 600dpi laser printers but found the resulting quality insufficient thus rendering questions ambiguous. As a consequence radiographs were removed from any OMR-based exams. However, modern visual display units have no such problem rendering the subtleties and have now been re-introduced into many CBA exams proving particularly useful in the clinical part of the medical curriculum.

In a field as complex as medicine, working collaboratively in teams, potentially interdisciplinary, is important. However, there is always the danger when working in such teams that changes are accidentally overwritten. TouchStone keeps an automatic change log that can be easily interrogated to see the last changes to a question. Also, in a similar way to MS Office, the system will monitor and stop concurrent editing; a second author will be warned about the situation and provided with a read-only version of the question.

Students

It has also been found that it has been simple to accommodate some disabilities such as dyslexia and some visual problems (see Case Study: University of Nottingham, e-Assessment, Disability Support).

Students who prefer a non-white background (some dyslexic students find it considerably easier to read text on a coloured background) can change the presentation of the exam to suit their needs and no matter where they sit in the exam (or taking formative tests at home) this is provided. Text size can also be altered.

Students have also reported that they found the online exam cleaner. Frequently incompletely rubbed out answers on an OMR form can cause the scanner to throw up a multi-answer warning. This takes time to correct as the member of staff administering the scanning process has to manually re-scan. Online the exam paper utilises radio buttons, check boxes and dropdown menus that unambiguously store only one answer. Students can change their answer selection as many times as they choose until finally submitting the form. The other advantage, especially with radio buttons, is that only one option can be selected for a multiple choice question, the interface constrains the permitted interactions. Sometimes on paper candidates mark too many options which cause problems when scanned.

Did implementation of this e-learning approach have any disadvantages or drawbacks?

This approach is inherently more risky than OMR as there are more things that can potentially go wrong than using paper. A successful examination is also reliant on more parts of the chain running successfully. These include servers, client computers, the network and power. Without any of these being in place at the right time, the exam will not run. Large computer labs are also required to sit a large cohort of students simultaneously and these need to have either enough distance between screens or physical barriers to prevent plagiarism.

Powerful servers are also required to cope with a large number of students taking an exam, especially at the start of the exam when they all send a request for the first page almost simultaneously. A backup server is also very useful so that, if there is a problem with the main server, the backup can be used instead and the exam does not need to be rescheduled. The configuration of these primary and backup servers is complex and requires specialists; the salaries of such personnel could be considered a financial drawback.

Some anxiety was seen in the students, especially for their first online exam, though it is predicted that this would rapidly decrease as their familiarity with the system grows. Alongside this there has been some anxiety also seen amongst staff. Although online exams should be very similar to paper based exams in the way in which they are organised by the institution (obviously apart from the type of delivery), it seems that the unfamiliarity of the online approach and the perceived lack of confidence in IT abilities can lead some staff to be shy of the new system. This can lead to more responsibility for the organisation of exams falling to IT staff, who wouldn't normally be involved in these stages.

How did this e-learning approach accord with or differ from any relevant departmental and/or institutional strategies?

It is the Medical Education Unit's role to lead the departmental strategy with respect to assessment. There are staff within the unit who specialise in medical education (including assessment) and staff whose responsibility it is to program and run the online assessment system and therefore lead the departmental strategy. These members of staff liaise with staff within the schools within the Faculty of Medicine to run online assessment.

The MEU started summative online exams 3 years ahead of the University's central IT team. When the University's Information Services started to look at software to begin summative online assessments, Touchstone was considered as an option to be adopted by the University but was beaten to second place by Questionmark Perception at the final stage. The Medical School has continued with Touchstone as there are a large number of questions within the system, staff are trained to use it and it has been designed around the question types used commonly in medicine.