A recent trend has been for funding agencies to become more focused on returns on investment (ROI) in sector innovation programmes – looking at both ROI in individual institutions and across the sector as a whole. For instance:

• The Higher Education Academy funded a range of “transfer” projects within its Pathfinder e-learning change programme, which were designed to transfer the lessons learnt, expertise and outputs from individual Pathfinder projects to the wider sector community. Some of the sector Communities of Practice (or Special Interest Groups) arose out of these transfer projects e.g. the QA-QE in e-learning SIG and ELESIG (Evaluations and Investigations of Learners’ Experiences of e-learning).

• JISC has run a “Benefits Realisation” programme associated with various innovations programmes e.g. the Users and Innovation programme and the more recent Lifelong Learning and Workforce Development programme. As with the Higher Education Academy “transfer” programmes, the aim is to transfer the lessons learnt, expertise and outputs from individual innovation projects to the wider sector community to maximise benefits from the investment.

Alongside, these initiatives, the funding agencies have been proactive in encouraging institutions to evaluate the impact of their innovation projects as opposed to purely evaluating the quality of the project outputs. Furthermore, there is increasing emphasis on “sustainable” impact, where institutions are encouraged to sustain the evaluation of impact beyond the official completion date of a project.

Some guidelines for evaluating impact are as follows:

• Be specific – don’t just focus on “benefits” – instead, focus on impacts/benefits to specific stakeholders e.g. students and staff.

• If the project is focusing on improving the student learning experience, you might want to use a framework for this. For instance, the University of Hertfordshire uses a framework based on Chickering and Gamson’s seven principles for good practice in undergraduate education: 

o   encourages contact between students and faculty,

o   develops reciprocity and cooperation among students,

o   encourages active learning,

o   gives prompt feedback,

o   emphasizes time on task,

o   communicates high expectations, and

o   respects diverse talents and ways of learning.

Impact evaluation could therefore be mapped to these seven principles. Similarly, assessment projects could use David Nicol’s principles of good assessment and feedback.

• Don’t forget the importance of base-lining – to enable you to have a “before” assessment on which to judge improvements.

• Where appropriate, define specific impact measures. For example, if a project is focused on developing more agile/lean curriculum design processes, measures could focus on efficiencies e.g. times to develop curricula and costs (e.g. reduced academic or administrative hours).

• When it comes to embedding in faculties/schools/departments, it is the faculty/school budget/business plan that is likely to be more critical than an institutional budget. Therefore project teams will need to convince heads of faculties/schools to budget for sustaining innovation projects. 

JISC infoNet has developed an Impact Calculator for evaluating the efficiency of a change initiative as part of their records and information management programme. This tool (which can be downloaded) could be used as a basis for calculating impact of other types of change initiatives.