Author: Syeve Jones, s.r.jones@leedsmet.ac.uk

JISC e-Learning Activity Area: e-Assessment

Higher Education Academy Subject Centre: Hospitality, Leisure, Sport and Tourism

This case study illustrates...an effect on exam results, an effect on student personal development, innovation in learning and teaching, staff satisfaction with e-learning, use of resources

Tangible Benefits

What tangible benefits did this e-learning approach produce?

This approach produced a good number of tangible benefits for all involved. For the institution this was the first large-scale test of the VLE when used for computer assisted assessment. This has proved important for the achievement of the strategy target outlined at the top of this case study.

For the point of view of the module tutors the most tangible benefits were seen at the end of the module. Marks for all 350 students were submitted within three hours of the completion of the exam. This could be potentially reduced but the module team took the decision to review statistics for each question in each test and exam to ensure that they were appropriate in terms of wording and discrimination. In previous years marking and standardisation of paper-based scripts from 350 students took a considerable amount of time. Even if each paper based script took 20 minutes to mark, it might have taken a total time of 115 hours... add moderation/standardisation times and the process could easily take 120 hours. With this approach the time spent is no more than 20 hours including exam supervision (by the course team) and checking results at the end. These savings in cost and time are now apparent after three years of developing this module in this form, but increased use of reusable learning objects and question banks could reduce initial development time. It is important to recognise the need for this upfront investment of time and for institutions to take account of this when looking at this type of CAA.

The delivery of tests every two weeks had the benefits of giving rise to a marked improvement in attendance during weeks where tests took place and to a lesser extent between the tests. The results were sent to personal tutors immediately after each set of tests that have the benefit of allowing personal tutors to see how their tutees were progressing, to follow up on any absences and head off early attrition.

Perhaps the biggest improvement was in the mean marks for the students of the module. Prior to the implementation of this approach a typical mark is around 53%, rising to 63% in 2006. The biggest area of benefit was in the final exam where a decision had been taken to put all previous years exams online along with full feedback and allow students to take these practice exams as many times as they wished. Those taking advantage of this had an average exam mark 15% higher than those who did not.

One important qualitative benefit of this approach has been that students are less apprehensive of taking a Finance subject since they get regular feedback on how they are doing.

Savings in cost and time are now apparent after three years of developing this module in this form, but increased use of reusable learning objects and question banks could reduce initial development time.

Did implementation of this e-learning approach have any disadvantages or drawbacks?

It should be noted that testing and preparation of large numbers of questions has a very considerable upfront outlay of time and effort over and above the normal deployment for the module - this came out of the personal time of staff teaching on the module but three years after the initial pilot this is now paying off. As this module is made available to a wider number of students the point that little additional effort will be required will be an important benefit to the tutors teaching on this module.

One interesting problem of this approach is that any new staff teaching on the module require an induction into the approach and some development in terms of producing questions and understanding the overall process of ensuring quality. However, it is important that this is not seen as a one off because the skills can be applied in other modules.

It should also be noted that the module team consider that this approach is best suited to level 1 students and that there would be considerable difficulties in testing level 3 students. Although this may be possible, efforts to extend computer assisted assessment are currently focused at level 1.

How did this e-learning approach accord with or differ from any relevant departmental and/or institutional strategies?

This approach is in line with departmental and institutional strategies. However, it does vary from discussions of how these strategies might be implemented in that initial expectations were that computer assisted assessments would be formative in nature rather than summative. The development team however considered that due to the strategic approach to learning often seen in these level 1 students that significant numbers of students would not engage with frequent formative assessment since it carried no apparent credit. The high levels of engagement on this module at least indicate that small amounts of summative assessment spread over the module is something students will engage with very successfully in their first year.

Implementation of approaches such as this at departmental or institutional level is not straightforward or appropriate for all modules. This approach works best in modules with large numbers of students where the initial outlay of time and effort will have a bigger pay off at a later date. It also assumes either a reasonable level of skill in module tutors or the presence of learning technologists within either departments or institutions. As staff within the faculty acquire the skills and knowledge of approaches and adapt their modules to allow this then embedding these approaches in teaching becomes possible - it is likely that more of this approach will be adopted during the year 2007-08.