8.6. Sustaining and Evaluating the MLE
Embedding implies cultural change - without it things can soon uproot themselves again. Sustainability implies you know what you want and why you want it and - and this is the hard part - that you can tell if you are getting it!
This section looks at the area of sustaining your MLE and addresses such questions as:
How do you evaluate the use of your MLE?
How is evaluation used to inform strategy, policy and practice, and how do you ensure that policy and practice is sufficiently responsive to lessons learned and new demands, customers and external policies?
How is the sustainability of courses and learning resources ensured?
Overall, honest answers from across the FE and HE sectors to these questions would be such things as: 'badly', 'not at all', 'don't know' etc! In particular, one of the great ironies of educational institutions is that they are rarely learning organisations.
The discussion of Administrative Issues in a previous section is also relevant here.
8.6.1. Evaluation of MLEs
This has been discussed in other parts of the infoKit. However some aspects of evaluation in the context of embedding are worth exploring: first a number of (true) generalities about the education context:
Education is an unstable environment, being subject to rapid politically driven change
Developments in educational institutions are unstable, even if well planned, for much the same reason
Senior management tend to lose focus on developments once past the project stage, taking the view 'we've done that' and moving on to the next challenge.
Education is notoriously bad at evaluation
The introduction of an MLE is often treated as a project, whereas it is, if it is going to be sustainable, an on-going an ever changing development. Therefore the management of an MLE needs to include, as apart of post-project thinking' an approach to project management and evaluation which reflects this - moving into attitude of maintaining the MLE is possibly dangerous.
Evaluation needs to be ongoing and needs to evaluate process as well as outcomes, procedures as well as educational impact. This implies it must be both formative and summative, and must be focussed on the users - it whichever context that may be.
Using evaluation to inform strategy, policy and practice and enhance responsiveness
Three major problems afflict evaluation in FE and HE (apart from not doing it, that is!):
Evaluation tends to be overly summative (with an 'after the event' focus)
This tends to prevent institutions understanding what is wrong the processes they use to develop systems, and handicaps the organisation in both learning from its own experience and in responding effectively to sudden change.
Evaluation tends to be too survey-based and 'big bang'.
Institutions seem to love massive surveys. These have many failings - surveys are poor at finding out things you hadn't thought of asking about, and by being too wide in terms of who they address, lack the granularity to pick up on more contextual or local problems. Instruments such as interviews and focus groups can often tell you far more.
Evaluation tends to under-resourced and is often either not finished or incompletely analysed.
All this also assumes that evaluation is itself embedded - it needs to be a fundamental part of both strategy and operational planning and integrated into development planning. Mechanisms must exist to feed local information upwards. This is less common than you might think - for example, it is quite usual for course monitoring to feed upwards only in general terms rather than in detail - this does not always help the institution (as opposed to the course or department) to pick up issues effectively and respond to them.
You may find the discussion and resources on 'Models and Frameworks' useful in this context.
8.6.2. Sustainability of courses and learning resources
This will hinge on a number of things:
Ability to reuse and repurpose content
Do you have in place the means to reuse and repurpose content? - This might be inherent in the VLE or similar component of your MLE, or be achieved via a repository
Can you link to other eResources in local (eBook, eLibrary), regional and national resources?
Can you do this is such a way that once established references can be reused in a way that preserves their context? For example a web site on fish is an information source, whereas a link to a specific item on that site which discusses swim bladders in the context of buoyancy is a learning resource. 'Reading lists' of references and resources are highly contextual, and because they represent the exercise of considerable professional expertise by their creator, also constitute significant intellectual property.
How can you preserve and reuse these without losing their context?
It is also important that it is desirable to be able to reuse and repurpose learning experiences and pedagogic approaches as well as content
Culture change in academic staff
A minefield (also discussed earlier) - this requires a shift in attitude by many staff, who may be very unwilling to 'share' for all sorts of reasons.
What incentive can you put in place to encourage this?
Is the institution's position on IPR clear and known?
Do your mechanisms for reuse and repurposing offer staff real timesaving?
Adherence to standards
Adherence to standards is going to be essential if the goals of the previous two points are to be met.
Do the component parts of your MLE conform to the relevant international and national specifications, standards and frameworks?
Do the software vendors you are relying on have a clear and demonstrable commitment to tracking these. Currently IMS Content Packaging, IMS Metadata, the UK Common Metadata Framework, SCORM and Simple Sequence are relevant here but in the future the IMS Learning Design Speciation will become important.
Exit strategy from MLE components
An MLE is not a single software system and your courses and resources are a huge investment.
Will you be able to move them? You may change VLE or move to something new instead of a VLE in the future. Are you locked in?
Addressing sustainability of courses and resources will have a profound effect on staff roles, this is discussed in the resources for this section.
Follow this link for key resources for this section (these open in a new window)
