Author: Mel Powell, m.j.powell@derby.ac.uk

JISC e-Learning Activity Area: e-Assessment

Higher Education Academy Subject Centre: Business, Management, Accountancy and Finance

This case study illustrates...an effect on learning, an effect on exam results, an effect on student personal development, student satisfaction with e-learning, innovation in learning and teaching, an influence on educational research, staff satisfaction with e-learning, staff personal development, a positive effect on retention

Tangible Benefits

What tangible benefits did this e-learning approach produce?

The evaluation data showed improvements in pass rates and grades achieved in the first two computer-based assessments as well as in the final analytical assessment compared with the previous two years. Changes in the distribution of outcome values were examined using the mean scores for summative assessments over a period of 3 years up to 2005 when the new formative assessments were introduced. The pattern shows an increase in mean scores for all assessments relative to both 2003 and 2004, indicating a potential positive impact on student outcomes. The data also showed an increase in the skew of the distribution of assessment grades towards high grades compared with the previous 2 years. There was a rise in the top end performance at A and B grades and a fall in the failure rates in the final assessment (see evaluation document).

The evaluation of formal student feedback showed improvements in most indicators. The feedback results had been good for most factors in prior years but evaluation scores increased compared to the previous year. A higher proportion of students were satisfied with teaching and learning, module workload and the development of their interest in the subject. Informal feedback from students suggested they were very keen to use this type of learning support. They liked being able to access computer-based materials and to test themselves whenever they liked. They liked the feedback within the learning and formative assessment and the clear links to other materials in the module and assessment. Many students said it helped them to gain the confidence to answer questions in tutorials and hence the confidence to attend tutorials in the first semester of their first year.

We found no evidence of a reduction in attendance. There was no evidence to suggest that fewer students bought or accessed an appropriate text book as a result of using formative e-materials. The drop out rate from the module declined relative to previous years, suggesting retention on the module had improved (see drop-out rates). The formative assessment system allowed us to identify weaker students at an early stage in the module and to force non-engaging students to discuss their approach to learning with a tutor. We found that the majority of students who were recalled to a clinic for lack of engagement started to use the computer-based materials as a result with a positive benefit in the first assessment.

The project greatly improved the expertise of the staff involved in its design and implementation and formed an important part of their personal development in e-learning. Our experience has also resulted in wider dissemination of the approach to our colleagues in the School, the University, and the wider academic community. It has resulted in stronger uptake of e-learning approaches within full-time modules within the school. We can identify 3 members of the business teaching team that took up computer-based assessment as a direct result of encouragement by the project team. We strongly believe that the formative materials aid student learning and provide an important source of support for first year first semester students who are unfamiliar with University learning environments. The materials also provide additional support for students whose first language is not English.

The project is part of the School's retention strategy and all the evidence suggests that it has had a positive impact. The project enabled the module staff to move towards a complete computer assessment mode for this module. When the module was introduced, the final assessment was a formal analytical examination using a real world case study. The development of the formative materials lead directly to a new computer-based final assessment. This cut the marking time considerably for the module staff. However, the additional time required to monitor and upgrade the materials have probably outweighed this reduction in marking time.

Did implementation of this e-learning approach have any disadvantages or drawbacks?

Although we applied for and received some funding to reduce our teaching hours in order to develop the materials, we hugely underestimated the amount of time required to create initial materials, test and pilot materials, and to resolve errors. In addition, we underestimated the time required to develop the data set for evaluation. In addition, this project would not have been possible without the expertise of the CIAD department. With this in mind, I do not think this type of development would be appropriate for courses with small numbers of students.

Looking back, we did not anticipate the need to continuously develop the material each year and the time cost that would involve on a regular basis. The original design involved the implementation of additional clinics for weaker students. This was funded by the retention strategy funding. The following year, the clinic system was dropped as the additional staff time was not funded.

How did this e-learning approach accord with or differ from any relevant departmental and/or institutional strategies?

The use of e-learning is quite widespread within the Business programmes as e-learning materials were developed several years ago for an e-learning mode delivery of our Business Studies Degree. Hence this project is consistent with the development of e-learning within the School. At the time of the project, however, the project team were 2 of a small number of staff who used computer-based assessment. What was entirely new in this project was the use of computer-based formative learning and assessment. However, the CIAD support staff are an indication of Derby University's commitment to developing computer-based and e-learning strategies and this project is an example of this. The materials now form the central component of the teaching and learning strategy for first year economics and data analysis within the business school.

Internal dissemination of the project results encouraged a wider group of staff within the business school to develop computer-based assessments and the experience of developing these formative materials provided good experience to support other staff. The module leader for this project also began the development of similar computer-based formative assessment and learning materials for a first year data analysis module within the School. This material has been run successfully using a similar model but is incomplete due to lack of time.