Resources and Links
"The National Co-ordinating Centre for Public Engagement [NCCPE] highlight seven 'dimensions' where engagement is known to 'make a difference' and for which measurement tools exist or are in development."
Public access to facilities
Public access to knowledge
Student engagement
Staff engagement
Widening participation
Encouraging economic regeneration
Institutional commitment
Performance Measurement
There are a number of reasons that institutions measure what they do.
- to justify funding
- to monitor quality of provision and service
- to monitor effectiveness and efficiencies
- to assess impact of activities on a range of factors
- to demonstrate value
- to inform decision-making
A review of Business and Community Engagement in an institution should seek to understand the measurements, targets and attainments being monitored and their purpose, any gaps (and reasons for them) between the required and actual figures and whether the measurements being taken inform future activities and deployment of resources.
HEFCE, through HESA, gather data relevant to Business and Community Engagement via the Higher Education - Business and Community (HE-BCI) Survey.
This provides data on engagement activities to different types of community and separates specifically:
- research-related activities
- business and community services
- regeneration and development programmes
- Intellectual Property
- social, community and cultural engagement
Much of the data gathered for the return is directly useable by institutions to monitor themselves as an institution. However monitoring of individual projects, programmes and activities can help inform the resourcing and efficacy of activity.
Measuring Participation
By far, the most widely measured indicators identified by the Embedding BCE project were participation rates. Museum admissions, meetings attendance figures, CPD students taught, conference attendance, number of students placed, number of Knowledge Transfer Partnerships etc.
These quantitative measures give some idea of the number of people who have engaged with the institution, but no qualitative data such as whether they thought the experience worthwhile or whether it led to an impact on their workplace, or within their local or regional community.
In some instances interviewees of the project were able to point to repeat business from employers, although this in itself cannot necessarily provide evidence of any qualitative evaluation by the company.
Qualitative Measurement and Impact
There was a widely accepted perception that attempting to gather qualitative data from participants was time-consuming, expensive and led to small returns of useful data. Surveying participants immediately following the end of an activity produced data on whether they enjoyed the activity and thought it worthwhile and could also gather data on what they hoped to do differently as a result of having taken part. However, collecting data on whether there was any actual impact due to changed behaviours required work that most practitioners felt was not justified by the results of any survey. Lack of changed behaviour was not necessarily felt to prove any failing of the activity, being highly dependent on participants' own circumstances, self motivation and opportunity, yet could easily be construed by others as being symbolic of some shortcomings of the activity.
Measuring eventual impact within communities was fraught with difficulties due to there being necessarily long time intervals between, for example, increased efficiencies or the introduction of new procedures due to training of clinicians or clinical managers and measurable effects on the health and well-being of a community. Another factor affecting the ability to prove connection to cause was the number of other factors that might contribute to such an effect after a number of years.
Customer Perception
Repeat business is one way of measuring customer perception, but it doesn't actually give very much in the way of information. It shows that no one has actually said in a convincing way 'Don't use this supplier again...' but apart from that, anything else is an assumption.
Surveying delegates directly at some point after a training workshop or course can be fraught with potential misunderstanding. If delegates are aware that they are attending a course or training event taught by the institution then they may be willing to give feedback. Where the course seemed (to them) to have been arranged by their own employer, a survey asking them questions about a university or college course can cause them to either throw it straight in the bin or send it back marked 'I haven't been on such a course'.
As an example from our own experience: JISC infoNet runs many one-day workshops. A sizeable proportion of these are offered regionally by the JISC Regional Support Centres (RSCs) to institutions. Delegates as a norm identify with having attended (or having 'been sent on') either an 'RSC course' or more simply a 'JISC course' no matter who actually delivered it or how clearly they tried to say where they were from. Receiving a survey several months later that says 'Our records show you attended a JISC infoNet event...' isn't always met with recognition.
However information given freely some time after an event is the most robust data in terms of evidence of impact. In obtaining such data, institutions will score where they have:
- flagged to delegates or participants that they may be surveyed at some point in the future - this means that they have a better chance of linking the survey to the course or event
- asked questions in such a way as to reassure people their responses are not just a box-ticking exercise - e.g. given examples as to how their data will be used
- made efforts to make the sending of data as straightforward as possible - e.g. by including links to online surveys rather than a URL that must be typed, and by keeping surveys reasonably short and using tick-box answering as much as possible whilst allowing extra comments to be made in text for added value
- given some incentive to complete the survey - it could be a place in a prize-draw, but it could equally be to have the opportunity to have a say in how future events are shaped
- acknowledged a data return - a short word of thanks and/or a list of the person's responses as a record if they want to keep it, is far better than just reaching 'The End'
