Author: Mel Powell, m.j.powell@derby.ac.uk
JISC e-Learning Activity Area: e-Assessment
Higher Education Academy Subject Centre: Business, Management, Accountancy and Finance
This case study illustrates...an effect on learning, an effect on exam results, an effect on student personal development, student satisfaction with e-learning, innovation in learning and teaching, an influence on educational research, staff satisfaction with e-learning, staff personal development, a positive effect on retention
Background & Context
Why did you use this e-learning approach?
A progressive set of computer-based formative assessment materials with extensive learning materials and feedback were created for a cohort of first year full-time students studying economics on business programmes. The materials were used to support private study and did not replace lectures or classes. The aim was to reduce failure rates amongst full-time students through improving self-regulating learning through computer-based formative private study. Recent literature suggests high failure rates may result from inadequate self-regulated learning skills among first year students and the limitations of traditional lecture and class formats in developing these skills. Increasing the degree of formative assessment and self-regulated learning can have positive impacts on learning outcomes for such modules. In particular, practice and learning structure can be reinforced through formative assessment to improve self-regulated learning and learning outcomes.
The formative e-assessment format was chosen because it was relatively easy to apply to the subject material (interactive graphics, calculations and links to Excel files) and because Derby University has an established department and expertise in designing and producing computer-based learning and assessment materials. In addition, prior experience of developing and using summative computer-based assessment on the module had shown that students were keen to use computer-based assessment and learning materials. The project started in response to (and was funded by) a University call for projects to increase student retention. The approach also met general University criteria to develop e-learning within existing modules.
What was the context in which you used this e-learning approach?
Derby University has a wide range of undergraduate business related degrees, some specialist such as Marketing or Business Management, and some joint honours Business and Law. All students on business-related degrees take a micro-economics first year module. This module is the focus for this case study. Student numbers on the module range from 250 to 350 a year. It was a one semester module taken in the Autumn, assessed by 2 phased (invigilated) computer-based summative assessments taken in weeks 6 and 9 of the semester and a final written examination. The assessment starts with simple multiple choice formats (assessment 1), moves on to calculations and model manipulation (assessment 2) and ends with an applied analysis using the economic tools to analyse a real world business case (examination).
The students come from a wide range of backgrounds, including home, EU and overseas students. Most students have either not studied economics before or only studied economics as part of a business A-level or equivalent qualification. They are generally unprepared for the model building approach and use of graphical analysis, and lack basic numeracy skills and the ability to apply a generalised model to a specific context. Failure rates for this module have been high. This type of subject requires regular practice and development of knowledge and skills over short time spans to achieve learning outcomes. The module teaching comprises a set of 11 lectures, supported by 11 tutorial classes (in groups of up to 20 students. Both lectures and tutorials are supported through the University's VLE. Online materials include study guidance, information on assessment and use of computer-based learning materials and formative assessment, lecture notes, tutorial materials (with answer sets), study guidance, recommended reading for each lecture and class. The tutorials are designed around a set of problems and activities which include analysing newspaper articles and using graphical analysis of relevant models. The lecturer is one of the two or three tutorial tutors. This format was used before and after the introduction of the computer-based formative learning and assessment materials. After introducing the materials, we also provided a weekly clinic for students who needed additional support.
The main challenge for the module staff was developing the content for the e-learning and formative assessment. It was important that the materials used were closely tied to the lecture materials and the tutorial materials, whilst developing the appropriate skills and knowledge for the summative assessments. They also needed to be designed to provide extensive feedback loops and developmental learning as the students would be progressing at their own pace. Because the size of the files in use would be large, the materials would only run on computers with sufficient memory. At the time when the materials were first introduced, many of the computers available in student labs had insufficient memory. The University was also unable to mount the materials on the intranet so that students could access it externally. This meant that students would have to access the materials while in the University. This posed more of a problem for part-time than full-time students. We were concerned that students might use the computer-based formative learning and assessment materials and decide not to attend tutorials. Finally, we were aware that students rarely read any of the guidance material provided online and we anticipated that many students would never find out about the computer-based formative materials unless a wide range of information sources and incentives were used.
What was the design?
The computer-based formative assessment system was designed and created by the module leader and one tutor in conjunction with the Derby University's computer assessment department (CIAD). The process began with the creation of e-learning materials and formative assessment activities by the module leader and tutor in Word format. The CIAD team used this material to generate the formative materials using the TRIAD software. We used the first unit as a pilot with current students asking for feedback on ease-of-use, relevance, and suggested improvements. It was refined and developed and then sent to an external for moderation. The remaining units were designed in a similar way. The two tutors compared notes on successful methods of communicating what was required and we quickly created a format that the CIAD team could easily interpret.
The materials contained instructions for progression at all levels. They also provided conceptual knowledge, used both interactive learning activities and interactive formative assessment with extensive feedback loops, automated advice on incorrect responses, advice and activities for improvement, multiple attempts, staged development and continuous formative grades. The materials were mainly interactive activities such as simulations, drag and pull diagrams, recognition exercises, calculation activities, and concept identification activities, leading to staged applied analysis of real world newspaper articles. The aim was to keep the student active and provide continuous feedback whilst building knowledge and skills. Students would require the knowledge and skills for the three computer-based summative assessments for the module. Access was by username and password and was not time-constrained. However, students had to access the materials on a campus computer. Information on date of access, time taken, activities undertaken and scores achieved were recorded with each student access for monitoring.
The formative learning and assessment materials were explicitly related to other learning materials identified for the module and to the three summative assessments. Each lecture was supported by a unit of computer-based materials. Students received information on the screen about the links to lectures and classes as well as to other support materials. Some activities were designed to help students prepare for class activities and discussions, others to prepare for summative assessments and to promote the use of other learning materials such as lecture notes and books. Students were told to use the materials throughout the module and were told the outcomes and usage would be monitored. The overall design was created to follow good practice and theoretical developments in formative assessments and self-regulated learning.
How did you implement and embed this e-learning approach?
The first set of students to use the e-learning and assessment materials were told about the support materials in the first lectures and given details on how to access the system, reasons why it would help them to study and the benefits of using the materials. They were also reminded that the information was available in handbooks and also on the university intranet. They were told that they should start to access the material that week to prepare for the first tutorial in the second week and that access would be monitored. We had created a new weekly clinic time for students on the module to attend for individual or group help. Anyone failing to access by week three would be asked to attend additional clinics. In the second week, when the first tutorial started, tutors reminded students about the e-learning and assessment materials and asked students who had tried it what they thought about it and whether they had any difficulties. Students were disappointed that they could not access the system remotely from home, but soon realised that they could use the system in short bursts. It was a useful activity to fill in gaps in their timetables when they were attending lectures and tutorials.
The teaching team liaised with the CIAD team on a regular basis to monitor any errors or problems with the materials identified by students. These were dealt with contemporaneously. As no other staff were involved in the roll-out, no additional training was required.
The approach was evaluated in two ways. Firstly, students were asked to provide informal feedback in tutorials on the materials and the system of formative assessment and learning. Secondly, the module tutor created a data set on outcome and input variables by student for the module. The data set was based on individual student information. It included information on student attendance at tutorials, student access and outcomes from the formative e-assessment materials, and student achievement in summative assessment for the module. In addition, each student was asked to complete a skills audit questionnaire before and after the module. The scores from this audit were entered into the data set by student. Evidence of the statistical analysis of the data is provided in the linked document.
Students also provided formal end of module feedback in the form of a standardised student questionnaire which could be compared to previous cohorts on the module. The formal feedback responses and the achievement standards in summative assessment were compared for the cohort that used the formative e-learning and assessment materials were compared to previous cohort years. The results are provided in the linked document. In addition, a theoretical model for developing student self-regulated learning through formative assessment was created from the current literature and applied in the analysis of the individual student data. A paper was written on the outcomes for an appropriate conference on student teaching and learning and is provided in related analysis.
In general, the strategy for implementation worked well. We found some students missed out on the early instructions because they started the module late. These students failed to read the available information online and in module handbooks and did not pick up the information because they had not established friendship groups. Some students could not understand that the system could only be accessed through a password and username on a campus computer. They tried to access the system remotely and failed. Some students tried to use their own username and password rather than the one provided. In general, the problems were lack of familiarity with University processes and systems and not taking responsibility to read instructions provided.
