Evaluative Research Improves Educational Policy and Practice
Submitting Institution
Lancaster UniversityUnit of Assessment
EducationSummary Impact Type
SocietalResearch Subject Area(s)
Education: Curriculum and Pedagogy, Specialist Studies In Education
Summary of the impact
We have influenced the development and implementation of national higher
education policies and educational practices in Scotland as well as
international policies through the development of a distinctive approach
to evaluation based on social practice theory. Using a novel way of
conceptualising and conducting evaluative research, we have:
- Changed policy makers' conceptions of how policies impact on
practices;
- Improved the management of national teaching and learning policy
initiatives;
- Influenced practices at an institutional level;
- Shaped international policy debates about how to develop useable and
socially just evaluation.
Underpinning research
We have developed an approach to the evaluation of educational systems
and initiatives based on social practice theories. For the first time in
the field of evaluation this distinctive and original approach:
- emphasised that people experience the effects of policy in diverse
ways in specific national and regional contexts;
- involved a focus on dimensions of practice in which people make sense
of their experience in particular places in unintended ways;
- conceptualised practices as using implicit knowledge as well as
explicit knowledge resources;
- focused on the routine and recurrent practices that result from a
policy intervention;
- recognised that whilst evaluation can have progressive enabling
characteristics, it can also be perceived as controlling and part of a
`surveillance' culture.
We have applied this approach to a number of significant national and
international evaluation projects. This has changed the nature of debates
around evaluation research practices and had an impact on educational
policies and practices in the UK and internationally. This approach to
evaluation informed the national evaluations of the Learners' Experiences
of the Skills for Life Policy (for the National Research and Development
Centre for Adult Literacy and Numeracy), the Learning and Teaching Support
Networks and the Centres for Teaching and Learning Excellence (both for
HEFCE). Here we focus on our work evaluating the Scottish Quality
Enhancement Framework (QEF) for Higher Education and our contribution to
international debates around evaluation practices.
The QEF is a joint initiative between the Scottish Funding Council (SFC),
QAA Scotland, Universities Scotland and NUS Scotland to develop an
enhancement-led approach to quality in the Scottish Higher Education
sector (involving 19 institutions and over 200,000 students). It has five
main elements that are designed to improve rather than simply audit the HE
system. These are: a comprehensive programme of institution-led reviews,
carried out by higher education institutions; enhancement-led external
institutional reviews; improved forms of public information about quality;
a greater voice for student representatives in institutional quality
systems; and a national programme of Enhancement
Themes. Our task was to evaluate all five elements, offering
research-based evidence to make it more effective. The evaluative research
work involved 8 national quantitative surveys (students, student
representatives, institutional student reps, middle managers, quality
managers, lecturers). Two waves of in-depth case studies, structured
interviews (with national key informants) and analysis of secondary data
were undertaken in all Universities in Scotland. The quotes from the
Scottish users in section 4 and 5 below refer to the 12 evaluative reports
over the 2003-2011 period which analysed this data (using the approach
embodied in the published research).
The key researcher involved in this work was Professor Murray Saunders.
The underpinning research has been conducted since 2000 and the evaluation
of the QEF took place in 2003-2011 and involved £795,000 of funding from
the SFC. We show its impact on policy and practice since 2008 and then how
this approach has also had an impact internationally.
References to the research
Underpinning Research in International Peer Reviewed Journal
Articles
Saunders, M. (2000) Beginning an Evaluation with RUFDATA: Theorizing a
Practical Approach to Evaluation Planning. Evaluation, 6(1): 7-21.
Saunders M, Bonamy J and Charlier B (2005) Using evaluation to create
`provisional stabilities': bridging innovation in Higher Education change
processes. Evaluation, 11(2): 37-55.
Saunders M (2006) The presence of evaluation theory and practice in
educational and social development: toward an inclusive approach. London
Review of Education 4:197-215.
Saunders, M., (2011) Capturing effects of interventions, policies and
programmes in the European context: a social practice perspective. Evaluation,
17: 89-103.
Saunders, M. (2012) The use and usability of evaluation outputs: A social
practice approach Evaluation 18: 421-436.
Edited book including chapters on the QEF Evaluation
Saunders, M., Trowler P., Bamber, V. (2011) Reconceptualising
Evaluative Practices in Higher Education. McGraw-Hill, Open
University Press. Reviews of this book highlight the strong case that
it makes for "the reconceptualising of evaluative practices that
takes into account the complexity of the educational experience and the
nexus between evaluation and the activities that are evaluated".
Details of the impact
Our social practice approach to evaluation had impact in four major
areas. First, it has had a major impact on the ways in which policy makers
understand the relations between policies and day to day practices.
Second, it enabled them to improve the management of a national policy
initiative aimed at enhancing quality across Scottish higher education.
Third, our approach to evaluation led to changes in quality practices in
Scottish universities and in some cases had a direct impact on the
development of their educational provision. In fact our evaluative
approach was so integral to the success of the QEF that one Scottish
Pro-Vice Chancellor said "I have often referred to it as the sixth element
of the Scottish Quality Enhancement Framework". Finally, our approach to
evaluation has had global reach in the way that it has influenced
international debates about the development of socially just approaches to
evaluation.
1) Changing policy makers conceptions of how policies impact on
practices
"We took up and ran with your model of culture change [set out in
Saunders, M. et al., 2011] through explicit work on developing and
supporting quality cultures and we still conceptualise the model of
institutional development in terms of systems of both shared practice
and shared values" (Head of the QAA Scotland).
Through our evaluative research reports and our frequent discussions with
higher education policy makers in Scotland, our evaluation helped policy
makers to reconceptualise the relations between policies and practices.
For example, in June 2012, the head of the QAA Scotland told us that our
analytical frameworks:
- provided them with a model of cultural change which they have used to
conceptualise institutional development in terms of systems of shared
practices and values;
- this model helped them to develop and support quality cultures across
the higher education sector;
- resulted in them thinking about measuring impact by examining the
alignment between policies and practices at the levels of senior
management, middle management and teaching and learning interactions
rather than by examining the level of awareness these groups had of
individual policies;
- helped them to develop collective models for defining and measuring
impact in this way;
- allowed them to conceptualise the way in which they can embed a
quality culture.
2) Improving the management of national teaching and learning policy
initiatives
The Lancaster University evaluations "provided valuable insights
into the operation of the [Enhancement] Themes which QAA Scotland
officers were able to use to adopt a revised approach to managing this
area of activity" and informed "a change in emphasis in the way in which
the individual Themes are managed" (Scottish QAA 2012 report on
quality assurance and enhancement in the university sector).
Our evaluation reports and discussions with policy makers in Scotland
have led to an improvement in the ways in which national teaching and
learning policy initiatives are managed. We provided high quality
information for policy makers that helped them to adjust aspects of policy
implementation in order to make it more effective. Reports written by
policy makers in Scotland and statements from these policy makers show
that our valuation led to the following improvements in the QEF:
- our concepts offered a powerful focus for the development of
enhancement indicators resulting in a new framework;
- a change in the approach to managing the operation of the enhancement
themes and a change in emphasis in the way that individual themes were
managed which was more responsive to institutional pressures and
circumstances;
- based on our evidence the external institutional reviews were revised
to be less like a QA audit and more of a fully enhancement-led approach
(by re-emphasising supportive and collegiate approaches);
- the enhancement themes were developed to become longer in duration and
to include the requirement to set up cross-disciplinary teams within
each institution;
- our reports were instrumental in the redrafting of the Scottish Higher
Education Enhancement Committee terms of reference;
- our reports played an important role in Student Participation in
Quality Scotland's (SPARQs) development of their approach to student
engagement in quality enhancement processes by analysing and presenting
good practice.
3) Influencing practices at an institutional level
The evaluations "helped frame our developments over a particularly
challenging period when we were engaged with a major project to merge. I
think it would be fair to claim that this work therefore had a direct
input to the re-development of our provision". (Senior lecturer at
the University of the West of Scotland)
Through our reports and the on-going discussions with individual higher
educational institutions, we had a direct influence on the practices of
staff and thus the students in Scottish universities. Statements from
staff in Scottish universities and national reports on enhancement
activities indicate that our evaluation:
- made an important contribution to the development of a Scottish
approach to quality;
- helped institutions to develop a notion of a quality framework;
- was crucial in developing a change-focused approach to enhancement at
the institutional level;
- had a direct impact on the ways in which some institutions approached
the internal evaluation of their educational innovations;
- had a direct input into the ways in which some institutions developed
their educational provision;
- led to the increased integration of students into institutional review
processes and the provision of examples of good practice for their
development.
4) Shaping international policy debates about how to develop useable and
socially just evaluation
"Your work [Saunders, M. 2011; 2012] fed into a new concept which
is reflected in the draft regulation for the 2014-2020 period, which is
aimed to be clearer on the different types of indicators used and the
differentiated roles of monitoring and evaluation... [It confirmed] that
we need to build up practice at different levels of European Governance
and seek to accumulate evidence across different contexts for the
different policy areas" (Head of Evaluation, Directorate General for
Regional Policy, European Commission)
Our approach to evaluation has had a major impact on international
debates on the development of inclusive evaluation practices. The impact
of our social practice approach to evaluation internationally has
included:
- the incorporation of our work on capturing the effects of policy
programmes and interventions into its guidance to all EU member states
for the evaluation of cohesion and integration fund expenditure;
- the emphasis on evaluation practices (through the vice presidency of
the IOCE and the Executive Committee which manages EvalPartners) and the
co-design of the Chiang Mai Declaration on Civil Society Working In
Partnership For Better Evaluation, which was supported by UN agencies,
the OECD and evaluation societies from across the world
http://www.mymande.org/sites/default/files/Declaration_evalpartners_English.pdf
;
- the emphasis on evaluation practices in recent UNICEF publications
(e.g. Bustelo, M. and Saunders, M. 2013) on Evaluation and Civil
Society, which are sponsored by organisations which include the World
Bank, OECD and UN Women.
Sources to corroborate the impact
1) Changing policy makers conceptions of how policies impact on
practices
- Statement from Head of the QAA Scotland 27 June 2012
2) Improving the management of national teaching and learning policy
initiatives
3) Influencing practices at an institutional level
4) Shaping international policy debates about how to develop useable
and socially just evaluation