What Do We Need to Consider?
Advantages of e-Assessment
Some of the advantages of e-assessment that you might want to consider are:
immediate feedback to students,
allows rehearsal and revision,
immediate feedback to staff,
allows evaluation of a course's strengths and weaknesses,
can be linked to other computer-based or online materials.
These are characteristics of good assessment technique and have links to a strong well evaluated pedagogy, as well as providing support for both staff and students - and of course, online assessment has all the other advantages of remote access and choice of time and place of assessment (although the latter may be limited for summative assessments that require security).
When looking to use e-assessment we can find that grading swiftly is one of its strongest points. Test feedback can be on a question by question basis and with the use of a 'knowledge tracking system' students can follow their progression and self determine their weaknesses (and strengths). This aspect of tracking progression combined with careful nurturing of student expectations can assist in developing students as autonomous learners.
Will e-assessment actually save time for the average lecturer? If you look at it in the short term then almost certainly not. Look at this as a longer term investment. While in the first year you won't save any time; in the second and third years of that material's life span you can save considerable development and support time. Another issue is that as well as the question setting, feedback also has to be created before students sit the test, something that tutors will have to factor into their busy schedules.
Sim et al (2004: 217) 'The emotional and subjectivity issues that are evident in human centred marking may be removed via automatic marking offered by CAA software'.
Concerns and Issues Associated with e-Assessment
Some of the concerns associated with e-assessment that you might want to consider are:
Time Required
One of the claims most often made for e-assessment is that it saves time. This is perfectly true at the point of delivery - it is possible to process the results for a summative assessment for a class of, for example around 700 students within a couple of hours of the last one logging off, including error checking and results analysis. This has to be balanced against the time, and skill, needed to create the assessment in the first place. This may not be so important for formative assessments which can be discussed with students later (and where failings may actually be of educational interest) but it is obviously vital that an end-of-course assessment should be reliable. The time and expertise for this cannot be underestimated, nor the need for 'shredding and vetting' by colleagues. There are times when an open-ended exercise (whether we call it an essay, project or report) may be more suitable for your purposes. There is of course no reason why this cannot be delivered online, with students uploading written materials into virtual learning environments to be assessed off-line.
Misleading Clues
There is a danger that by picking out particular areas (either deliberately or inadvertently), the quizzes could send misleading clues to students about what is and isn't important. This is exacerbated by the students' tendency to be very strategic and exam-focused when considering how best to spend their study time.' Clarke et al 2004: 253.
Equity and Diversity
Equity and diversity - when computers are involved in the assessment process, there are equity issues for different student groups relating to language status and gender and issues around computer anxiety and exam equivalence. Brosnan (1999: 48-49) suggests that: 'computer anxiety can lead to simplification of conceptual reasoning, heightened polarisation and extremity of judgement and pre-emption of attention, working memory and processing resources. Individuals high in computer anxiety will therefore under-perform in computer-based versions of assessment'. Brosnan (1999) asserts that even those who are using computers effectively will still exhibit computer anxiety and he contends that female students exhibit higher levels of anxiety, and so poorer levels of performance. Ricketts and Wilks (2002) suggest that student performance in tests should be monitored to ensure fairness and consistency when there are any changes in delivery, whether this is a change to CAA (computer-aided assessment) or changes in the way that the CAA is presented.
Issues of Equivalence
The issues of equivalence between different forms of assessment are highlighted by Clariana and Wallace (2002) who assert that you cannot necessarily expect that equivalent measures of student learning will be produced from computer-based and paper-based tests, even if you use the same questions. They assume that the 'test mode effect' will diminish when students become as familiar with the medium of the computer as they are with paper, for assessment, and that computer familiarity might be an issue for some groups of students. McDonald (2002) concurs, expressing the belief that inconsistent findings relating to student scores in computer-based and paper-based tests often result from different levels of exposure to changing technologies. It is probably fair to observe generally that students perform differently under different conditions of assessment, and that innovations in CAA simply introduce a new range of variants on this construct theme.
Effect of Using Full Range of Marks
Sim et al (2004: 217) 'CAA, like mathematics and some science subjects, also tends to use the full range of marks therefore the trend towards a higher proportion of First Class Degrees may occur in other subject domains adopting this technique in the future.'
Attracts Greater Scrutiny
While problems with objective testing can occur whether the tests are offered on paper or online, it is the online testing that tends to attract greater scrutiny. Don Mackenzie in Brown et al. (1997: 217) contends that CAA has produced quality and efficiency gains in assessment, but for many there have been marginally lower pass rates than for essay-type assessments. He suggests that this is because there is a larger spread of marks (typically a standard deviation of 15 per cent with a mean of 50 per cent).
Design of Questions
Problems in the use of computers for multiple choice questions could derive from the design of the questions and the skills of the designer (Mackenzie, 2003), rather than from the software or the use of the computer per se, or it could be that some tutors may be reluctant to relinquish traditional modes of assessment (Mackenzie, 2003).
Disparity
Research by Clariana and Wallace (2002) has shown that the use of CAA has a positive impact on the test scores of high attaining pupils, when compared to those from paper-based tests, because they assert that higher-attaining students more quickly adapt to new assessment approaches. Noyes et al. (2004) suggest that lower-performing individuals will be disadvantaged when CAA is used because they assert that a greater workload and additional effort is required to complete a computer-based test.
Change in Working Practices
The savings in time that might be produced by the automated marking in CAA are instead shifted to the design and construction of the assessment activity, (including the level and amount of feedback to be given). Brown et al. (1997) see this as a profound change in working practices for academics. There is also the issue of defining requisite technical skills for students undertaking CAA such as, who should be involved in that training, and when should it take place, especially in the context of overloaded curricula (Weller, 2002). Macdonald and Twining (2002) found that their students only became competent in the use of a particular piece of software while they were completing an assignment that required its use.
Plagiarism
Plagiarism is a concern for many thinking of using CAA (Weller, 2002), but Rovai (2000) and Carroll (2002) suggest that assessment design is the key to deterring plagiarism. O'Hare and Mackenzie (2004) assert that there is a level of imagination and rigour required for the design of assessment online compared to that for more traditional forms of assessment. Weller (2002) suggests that the use of portfolios can help to counter plagiarism, as this places less reliance on single assessment items. The JISC funded Plagiarism Advisory Service gives advice and guidance on plagiarism prevention.
Off-Campus Assessment
Computer software for CAA allows for questions to be presented to students in different orders, with distracters in different orders, and if sufficient questions have been compiled of sufficient integrity then they can sit different tests. All of this allows for students to sit in adjoining desks in computer laboratories that will at other times be used for learning activities. This is fairly straightforward for on-campus students, but could be more problematic for students taking courses at a distance. However, Rovai (2000) suggests that this difficulty can be overcome by using 'proctored testing' where academics arrange for students to sit online assessments under test conditions in alternative venues.
