Author: Michael Begg, michael.begg@ed.ac.uk
Author: David Dewhurst, d.dewhurst@ed.ac.uk
JISC e-Learning Activity Area: e-Assessment
Higher Education Academy Subject Centre: Medicine, Dentistry and Veterinary Medicine
This case study illustrates...use of specialist software, an effect on exam results, an effect on student personal development, innovation in learning and teaching, an influence on educational research, staff satisfaction with e-learning, a positive effect on retention, an influence on policy, use of resources, modifications to learning spaces, management of learning assets
Background & Context
The Online System for Clinical Assessment (OSCA) was developed in-house by the College of Medicine and Veterinary Medicine's Learning Technology Section to support the delivery of online OSCE assessments in the undergraduate medical degree programme (MBChB) and undergraduate veterinary degree programmes (BVM&S). OSCEs (Objective, Structured, Clinical Examinations) are an essential component of assessments for medical and veterinary students.
Why did you use this e-learning approach?
The system was developed in 2004/5 in order to specifically take over some of the stations of the year 3 clinical OSCE in the MBChB course which had previously been executed face-to-face at manned stations.
The drivers for adopting an online approach were:
- to reduce the possibility for student collusion
- to reduce the costs of running OSCEs which are extremely expensive
- to provide a system-based standard for marking
- to reduce the complex administration underpinning assessment processes
- to address the growing sense that existing assessment structures were not capturing as accurate a picture of student ability as was possible
What was the context in which you used this e-learning approach?
There are currently approximately 2000 students on the 2 undergraduate programmes in Medicine and Veterinary Medicine at Edinburgh.
Many of the 1500+ teaching and clinical staff are involved with assessment to some degree, though the figure is certainly smaller for those directly involved with authoring examinations within OSCA. There are approximately 40 teaching/admin staff registered as authors within the OSCA database though administrative staff often input assessments created by academic staff into the system on their behalf.
OSCA was initially used in the specific context of the Year 3 clinical exams of the MBChB programme. However, the system has, since 2005, expanded to accommodate other summative and in-course examinations in both the MBChB and the BVM&S. To date, OSCA has been used for 40 formal examinations - comprising events with around 100 simultaneous users to resit events with a single student. In the 2007-08 academic session OSCA will be the vehicle for 19 discrete assessment events across the 2 programmes (not including resits).
The institution is a licensed user of QuestionMark Perception (QMP), but the working group of academic staff, support staff and learning technologists tasked with evaluating the usefulness of QMP for OSCE examinations concluded that it lacked the flexibility to accommodate all of the assessment requirements of the relevant courses.
This was backed up by a comparative review of OSCA and QMP by a teaching fellow in 2006 where it was clearly established that the OSCA environment - both in terms of authoring and delivery - was largely more appropriate, usable, and popular with those staff creating assessments than QMP.
It should be noted, however, that one course within the MBChB continues to use QMP for some examinations.
What was the design?
The project was originally funded by the University of Edinburgh's Principal's eLearning Fund (PeLF), an annual fund which distributes seedcorn grants through a competitive bidding process.
Typically an OSCE clinical examination comprises around 20 OSCE stations and requires students to move from OSCE station to OSCE station each one providing a different clinical assessment and lasting approximately 5 minutes. Examination rooms required significant setting-up and each station needed to be supervised by one or more academic/clinical staff. A full cohort of students (230+ in Medicine alone) would pass through this examination making it extremely time (often extending over two days) and resource intensive. It was considered that some of the OSCE stations could be replaced by assessments, testing the same competencies/skills but delivered in a secure technological environment requiring no staff support that would allow a full cohort of students to be examined in two sittings over a single day.
The working group tasked with planning this project comprised course module leaders, clinical teaching staff, clinical skills facilitators, members of the Medical Teaching Organisation, College Office representatives (assessment officer) and representatives from the Learning Technology Section.
As the embedding of OSCA within the College's assessment processes has grown the management of the application has led to the establishment of an e-Assessment group comprising members of the Medical Teaching Organisation, the Veterinary Teaching Organisation, administrative staff, assessment officer and Learning Technology Section representatives. There is also a CMVM Teaching Fellow who undertakes ongoing review and evaluation of e-Assessment activity and steers much of the required staff development across the college.
How did you implement and embed this e-learning approach?
Alternatives were considered:
- PowerPoint-delivered exam in a lecture theatre setting (visibility of slides and timing of slide rollover were prohibitive elements against uptake)
- Paper based exam (poor quality reproduction of images, poor cost effectiveness, logistically problematic for marking)
- MCQ format exam compatible with OSCE to derive a valid overall clinical assessment mark (C. van der Vleuten, Validity of final examinations in undergraduate medical training. BMJ 2000;321)
Another deciding factor explored by the initial project group was that the written/typed word does standardise the examination in a way that viva/consultation cannot.
An online solution - using an appropriately aligned system - offers a written word-based examination, automated marking, reduction in logistical problems and production costs.
With regard to practical implementation - a teaching fellow, closely liaising with learning technologists was able to steer content authors towards writing effective questions, with appropriate resources, that would be accommodated by the OSCA interface (see ongoing issue about question writing below).
Students were made familiar with the interface initially through a mock exam (which allowed for system testing) and subsequently through a video outlining the main interface features.
At the time of the original implementation one of the main concerns was the level of institutional support for online assessment. The level of support required from the central IT support departments had to be negotiated as essentially, with the increasing use of both QMP and OSCA for online examinations across the University they were/are being asked to support two different secure kiosk environments for assessment events.
Provision of physical spaces for conducting online assessment remains an issue for the Institution and work is being undertaken at institutional level to address this issue.
