Evaluative Research Improves Educational Policy and Practice

Submitting Institution

Lancaster University

Unit of Assessment

Education

Summary Impact Type

Societal

Research Subject Area(s)

Education: Curriculum and Pedagogy, Specialist Studies In Education


Download original

PDF

Summary of the impact

We have influenced the development and implementation of national higher education policies and educational practices in Scotland as well as international policies through the development of a distinctive approach to evaluation based on social practice theory. Using a novel way of conceptualising and conducting evaluative research, we have:

  • Changed policy makers' conceptions of how policies impact on practices;
  • Improved the management of national teaching and learning policy initiatives;
  • Influenced practices at an institutional level;
  • Shaped international policy debates about how to develop useable and socially just evaluation.

Underpinning research

We have developed an approach to the evaluation of educational systems and initiatives based on social practice theories. For the first time in the field of evaluation this distinctive and original approach:

  • emphasised that people experience the effects of policy in diverse ways in specific national and regional contexts;
  • involved a focus on dimensions of practice in which people make sense of their experience in particular places in unintended ways;
  • conceptualised practices as using implicit knowledge as well as explicit knowledge resources;
  • focused on the routine and recurrent practices that result from a policy intervention;
  • recognised that whilst evaluation can have progressive enabling characteristics, it can also be perceived as controlling and part of a `surveillance' culture.

We have applied this approach to a number of significant national and international evaluation projects. This has changed the nature of debates around evaluation research practices and had an impact on educational policies and practices in the UK and internationally. This approach to evaluation informed the national evaluations of the Learners' Experiences of the Skills for Life Policy (for the National Research and Development Centre for Adult Literacy and Numeracy), the Learning and Teaching Support Networks and the Centres for Teaching and Learning Excellence (both for HEFCE). Here we focus on our work evaluating the Scottish Quality Enhancement Framework (QEF) for Higher Education and our contribution to international debates around evaluation practices.

The QEF is a joint initiative between the Scottish Funding Council (SFC), QAA Scotland, Universities Scotland and NUS Scotland to develop an enhancement-led approach to quality in the Scottish Higher Education sector (involving 19 institutions and over 200,000 students). It has five main elements that are designed to improve rather than simply audit the HE system. These are: a comprehensive programme of institution-led reviews, carried out by higher education institutions; enhancement-led external institutional reviews; improved forms of public information about quality; a greater voice for student representatives in institutional quality systems; and a national programme of Enhancement Themes. Our task was to evaluate all five elements, offering research-based evidence to make it more effective. The evaluative research work involved 8 national quantitative surveys (students, student representatives, institutional student reps, middle managers, quality managers, lecturers). Two waves of in-depth case studies, structured interviews (with national key informants) and analysis of secondary data were undertaken in all Universities in Scotland. The quotes from the Scottish users in section 4 and 5 below refer to the 12 evaluative reports over the 2003-2011 period which analysed this data (using the approach embodied in the published research).

The key researcher involved in this work was Professor Murray Saunders. The underpinning research has been conducted since 2000 and the evaluation of the QEF took place in 2003-2011 and involved £795,000 of funding from the SFC. We show its impact on policy and practice since 2008 and then how this approach has also had an impact internationally.

References to the research

Underpinning Research in International Peer Reviewed Journal Articles

Saunders, M. (2000) Beginning an Evaluation with RUFDATA: Theorizing a Practical Approach to Evaluation Planning. Evaluation, 6(1): 7-21.

 
 
 

Saunders M, Bonamy J and Charlier B (2005) Using evaluation to create `provisional stabilities': bridging innovation in Higher Education change processes. Evaluation, 11(2): 37-55.

 
 
 

Saunders M (2006) The presence of evaluation theory and practice in educational and social development: toward an inclusive approach. London Review of Education 4:197-215.

 
 
 

Saunders, M., (2011) Capturing effects of interventions, policies and programmes in the European context: a social practice perspective. Evaluation, 17: 89-103.

 
 
 

Saunders, M. (2012) The use and usability of evaluation outputs: A social practice approach Evaluation 18: 421-436.

Bustelo, M. and Saunders, M. (2013) Making a difference: Supporting evaluative practice through the EES. In Rugh, J and Segone, M. (2013) (Eds.) Voluntary Organizations for Professional Evaluation (VOPEs) Learning from Africa, Americas, Asia, Australasia, Europe and Middle East at
http://www.mymande.org/sites/default/files/files/UNICEF%20NY_ECS_Book2_Web(3).pdf

Edited book including chapters on the QEF Evaluation

Saunders, M., Trowler P., Bamber, V. (2011) Reconceptualising Evaluative Practices in Higher Education. McGraw-Hill, Open University Press. Reviews of this book highlight the strong case that it makes for "the reconceptualising of evaluative practices that takes into account the complexity of the educational experience and the nexus between evaluation and the activities that are evaluated".

Details of the impact

Our social practice approach to evaluation had impact in four major areas. First, it has had a major impact on the ways in which policy makers understand the relations between policies and day to day practices. Second, it enabled them to improve the management of a national policy initiative aimed at enhancing quality across Scottish higher education. Third, our approach to evaluation led to changes in quality practices in Scottish universities and in some cases had a direct impact on the development of their educational provision. In fact our evaluative approach was so integral to the success of the QEF that one Scottish Pro-Vice Chancellor said "I have often referred to it as the sixth element of the Scottish Quality Enhancement Framework". Finally, our approach to evaluation has had global reach in the way that it has influenced international debates about the development of socially just approaches to evaluation.

1) Changing policy makers conceptions of how policies impact on practices

"We took up and ran with your model of culture change [set out in Saunders, M. et al., 2011] through explicit work on developing and supporting quality cultures and we still conceptualise the model of institutional development in terms of systems of both shared practice and shared values" (Head of the QAA Scotland).

Through our evaluative research reports and our frequent discussions with higher education policy makers in Scotland, our evaluation helped policy makers to reconceptualise the relations between policies and practices. For example, in June 2012, the head of the QAA Scotland told us that our analytical frameworks:

  • provided them with a model of cultural change which they have used to conceptualise institutional development in terms of systems of shared practices and values;
  • this model helped them to develop and support quality cultures across the higher education sector;
  • resulted in them thinking about measuring impact by examining the alignment between policies and practices at the levels of senior management, middle management and teaching and learning interactions rather than by examining the level of awareness these groups had of individual policies;
  • helped them to develop collective models for defining and measuring impact in this way;
  • allowed them to conceptualise the way in which they can embed a quality culture.

2) Improving the management of national teaching and learning policy initiatives
The Lancaster University evaluations "provided valuable insights into the operation of the [Enhancement] Themes which QAA Scotland officers were able to use to adopt a revised approach to managing this area of activity" and informed "a change in emphasis in the way in which the individual Themes are managed" (Scottish QAA 2012 report on quality assurance and enhancement in the university sector).

Our evaluation reports and discussions with policy makers in Scotland have led to an improvement in the ways in which national teaching and learning policy initiatives are managed. We provided high quality information for policy makers that helped them to adjust aspects of policy implementation in order to make it more effective. Reports written by policy makers in Scotland and statements from these policy makers show that our valuation led to the following improvements in the QEF:

  • our concepts offered a powerful focus for the development of enhancement indicators resulting in a new framework;
  • a change in the approach to managing the operation of the enhancement themes and a change in emphasis in the way that individual themes were managed which was more responsive to institutional pressures and circumstances;
  • based on our evidence the external institutional reviews were revised to be less like a QA audit and more of a fully enhancement-led approach (by re-emphasising supportive and collegiate approaches);
  • the enhancement themes were developed to become longer in duration and to include the requirement to set up cross-disciplinary teams within each institution;
  • our reports were instrumental in the redrafting of the Scottish Higher Education Enhancement Committee terms of reference;
  • our reports played an important role in Student Participation in Quality Scotland's (SPARQs) development of their approach to student engagement in quality enhancement processes by analysing and presenting good practice.

3) Influencing practices at an institutional level
The evaluations "helped frame our developments over a particularly challenging period when we were engaged with a major project to merge. I think it would be fair to claim that this work therefore had a direct input to the re-development of our provision". (Senior lecturer at the University of the West of Scotland)

Through our reports and the on-going discussions with individual higher educational institutions, we had a direct influence on the practices of staff and thus the students in Scottish universities. Statements from staff in Scottish universities and national reports on enhancement activities indicate that our evaluation:

  • made an important contribution to the development of a Scottish approach to quality;
  • helped institutions to develop a notion of a quality framework;
  • was crucial in developing a change-focused approach to enhancement at the institutional level;
  • had a direct impact on the ways in which some institutions approached the internal evaluation of their educational innovations;
  • had a direct input into the ways in which some institutions developed their educational provision;
  • led to the increased integration of students into institutional review processes and the provision of examples of good practice for their development.

4) Shaping international policy debates about how to develop useable and socially just evaluation
"Your work [Saunders, M. 2011; 2012] fed into a new concept which is reflected in the draft regulation for the 2014-2020 period, which is aimed to be clearer on the different types of indicators used and the differentiated roles of monitoring and evaluation... [It confirmed] that we need to build up practice at different levels of European Governance and seek to accumulate evidence across different contexts for the different policy areas" (Head of Evaluation, Directorate General for Regional Policy, European Commission)

Our approach to evaluation has had a major impact on international debates on the development of inclusive evaluation practices. The impact of our social practice approach to evaluation internationally has included:

  • the incorporation of our work on capturing the effects of policy programmes and interventions into its guidance to all EU member states for the evaluation of cohesion and integration fund expenditure;
  • the emphasis on evaluation practices (through the vice presidency of the IOCE and the Executive Committee which manages EvalPartners) and the co-design of the Chiang Mai Declaration on Civil Society Working In Partnership For Better Evaluation, which was supported by UN agencies, the OECD and evaluation societies from across the world
    http://www.mymande.org/sites/default/files/Declaration_evalpartners_English.pdf ;
  • the emphasis on evaluation practices in recent UNICEF publications (e.g. Bustelo, M. and Saunders, M. 2013) on Evaluation and Civil Society, which are sponsored by organisations which include the World Bank, OECD and UN Women.

Sources to corroborate the impact

1) Changing policy makers conceptions of how policies impact on practices

  • Statement from Head of the QAA Scotland 27 June 2012

2) Improving the management of national teaching and learning policy initiatives

3) Influencing practices at an institutional level

4) Shaping international policy debates about how to develop useable and socially just evaluation