Author: David Davies, david.davies@warwick.ac.uk
JISC e-Learning Activity Area: e-Assessment
Higher Education Academy Subject Centre: Medicine, Dentistry and Veterinary Medicine
This case study illustrates...an effect on learning, an effect on student personal development, innovation in learning and teaching, an influence on policy, use of resources
Background & Context
Why did you use this e-learning approach?
In Warwick Medical School (WMS) we have adopted an assessment blueprinting approach to help match learning outcomes and competencies against examination question items. In Phase I of the WMS undergraduate medical course students are assessed at the end of each semester by written answer exams and objective structured clinical examination (OSCE). In the written exams integrated questions are coded against General Medical Council (GMC) competency themes. Students receive feedback on their written exam performance that guides them to plan learning strategies to help attain GMC competencies.
What was the context in which you used this e-learning approach?
WMS offers the largest graduate-only medical degree in the UK. The four year MBChB (Bachelor of Medicine and Surgery) programme is designed for graduates in biological, natural and physical sciences. We produce graduates with the clinical competence to work as Foundation Year 1 and 2 doctors with the understanding, intellectual skills, and attitudes to practice medicine along with the habit of lifelong learning necessary for a successful, continually developing career. UK medical schools are accredited on their ability to meet GMC standards for medical graduate knowledge, skills and attitudes. Assessment is a key part of learning and an integral element of assessment is feedback to the learner. The GMC have identified in Tomorrow's Doctors 'Feedback about performance in assessments helps to identify strengths and weaknesses, both in students and in the curriculum, that allow changes to be made'. The GMC have identified a number of curriculum competency areas that define the knowledge, skills, and attitudes of a graduate medical student.
In WMS we wanted to develop an assessment feedback system that gave students insight into their exam performance from the perspective of attainment in modules and in relation to GMC competency areas. We believe that giving students personalised feedback that addresses the GMC competency themes will help individuals to become better learners. Prior to taking this approach written assessments did not use integrated questions. Instead questions were based on module teaching and feedback was simply in terms of a score in each module. As modules are essentially arbitrary divisions of the course this approach did not encourage integrated thinking and neither did it seek to address the broader GMC competencies expected to today's graduate students.
What was the design?
The blueprinting approach was developed by a small number of key staff led by the Associate Dean for Teaching. Consultation during the blueprinting process involved all module coordinators. The written assessment sub-group of WMS Phase I teachers developed each integrated exam paper. The Associate Dean for Teaching and Course Director then coded these items against GMC competency areas as well as identifying which module was the logical 'home' for each question sub-item. The optically marked sheets for exam markers were developed in-house in consultation with the University's eLab IT team. The database for producing individualised feedback was created in-house. The WMS admin officer for Phase I of the course was a key player in developing the procedure for getting exam papers marked, scanned and processed. The Examinations Officer for WMS scrutinised the whole process.
How did you implement and embed this e-learning approach?
Moving towards a competency-based assessment was not natural for all teaching staff in WMS. There were 'hearts and minds' that had to be won to establish the need for this approach. For a start, not everyone was aware of the GMC competency areas, including module coordinators. Personal tutors were not initially equipped with the skills to provide support for their students once they had received their exam feedback. We did not have the necessary IT in place before starting this experiment, and although we have ironed out many bugs we are still actively developing our approach.
We have so far piloted this blueprinting approach on two end of semester exams, semester 1 (December 2006) and semester 2 (June 2007). Because of time constraints we had not arranged a training session for personal tutors the first time we ran this new feedback system at the end of semester 1. Consequently we had students unsure how to interpret their results and personal tutors who did not feel equipped to offer support. We learnt rapidly from this and made sure we arranged training sessions before feedback was given for the end of second semester results.
We plan to conduct a full evaluation of this new feedback system at the end of semester 3 when the system will have run for the full Phase I of the course. Preliminary results indicate that students welcome the new system as a way of thinking beyond modules.
Because of the requirement for timely processing of exam marks we are contemplating purchasing our own optical mark reader system. Currently we have been using the central IT Services system.
We recognise that to be truly successful in offering a competency framework for students that allows them to look across all four years of the course then we need to blueprint all assessments, not just written exams and not just in Phase I. We also want to develop an assessment item bank, a better automated feedback system (instead of email) and a competency element to our portfolio system. To that end we have bid for funds to create a new IT post within WMS to help develop these systems. Funds were from the Central University Education Innovation Fund. We hope to appoint the new post in the third quarter of 2007.
