Summary of the impact

As a direct result of methodological research led by Professor Ray Pawson at Leeds, `realist evaluation' has provided a new lens through which to assess and develop social programmes. It has critically changed the apparatus of evidenced-based policy and the way in which policy research is commissioned and utilised. Through advisory work, training package provision, partnership-research and professional exchange, this `realist' perspective has formed a new standard in social programme evaluation, and is used by commissioners in the UK and internationally to frame their interventions across policy domains, including education, environment, criminal justice, and health and social care.

Underpinning research

Beginning in 1994, the underpinning research introduced a `theory-driven' approach to programme evaluation. Pawson (SL, reader and professor, University of Leeds, 1990 — present) set out to tackle one of policy evaluation's most pressing challenges — how to research complex and multifaceted programmes, located in complex and changing environments, designed to tackle complex and deep-seated problems.

Realist evaluation penetrates the `black box' of interventions seeking to understand what it is about programmes that makes them work. Social programmes work and only work through the reasoning of their stakeholders and subjects. The crucial task for the evaluator is thus to discover how programme resources impact upon on the subject's reasoning. Such choices are always constrained by a range of local and circumstantial factors (rendering problematic any assumption that existing programmes will `work' universally). The key issue for policy makers and practitioners is therefore to find out how and under what conditions a given measure will produce its impacts, and consequently, the `realist evaluation' paradigm looked to transform the basic question of evaluation from `what works?' to `what works for whom in what circumstances?'

This research, which originated in a book Pawson co-authored with criminologist Tilley (then at Nottingham Trent), established not only the theoretical and conceptual basis for `realist evaluation' but also protocols and procedures for how to conduct it [1]. Turning the oil-tanker of social inquiry requires perseverance. It also requires proposing, debating, refining and consolidating new research methodologies. Pawson thus led further inquiries in which the method was enlarged via two further books covering research synthesis [2] and complex policy analysis [3]. In addition there are 24 book chapters, 41 refereed journal papers, 8 articles in professional journals/newsletters, 10 working papers, 10 research reports and one Festschrift, which add detail to the research strategy and provide practical application of the method across a range of programmes and policy domains ([4] [5] as examples). Pawson's research was supported by external funding specifically earmarked for methodological development on realist methods, including ESRC awards [G1-G4] and NHS NIHR awards [G5-G6].

Methodological advances cannot be developed in a vacuum. Research principles become honed in practice. Accordingly, Pawson carried out a series of parallel substantive inquiries into the efficacy of a variety of ongoing UK and Canadian programmes, positioning him as an expert in this field. The interventions studied (between 1995 and 2013) include: prisoner education programmes, Megan's Law, youth mentoring and employment initiatives, urban regeneration schemes, public disclosure (naming and shaming), public health law, NHS service modernisation and referral management schemes. Most of these involved research contracts won in competitive tender and through this exposure the influence of the approach grew within the commissioning infrastructure. Income from these projects approached £2.0m. Clients included:

• HMP Prison Service, Home Office (1990-98)
• SSHRC / Canadian Corrections Service (1993-96)
• Transport Research Laboratory (1998-99)
• Department of Environment, Transport and Regions (2000)
• Social Care Institute for Excellence (2002-03)
• The Connexions Service (2003-04)
• Guy's/St Thomas' Hospital Trust (2005-06)
• Department for Education and Skills (2006-07)
• NHS NIHR HS&DR (2012 ongoing)

Research and writing were also supported and funded by three periods as visiting professor (University of Rome, 2005; University of Victoria, 2006; RMIT, 2007). These helped to provide an international footing for the research. Save for these three short periods and a year on secondment to the ESRC centre for Evidence-Based Policy and Practice, Queen Mary, University of London, all the research was led by Pawson at the University of Leeds.

References to the research

Supported by ESRC fellowships and awards:

G1. 1995 — 1996, Senior Fellowship — Fellowship Programme , Principles and Practice of Evaluation Research, awarded to Pawson, £20,780.00, Grant no: H52427501995

G2. 2001-2002, Visiting Fellowship — Centre for Evidence-based Policy and Practice, Queen Mary, University of London, £1,293,124.62, Grant No: H141251005

G3. 2003-2004, Senior Fellowship — Evidence-Based Policy, Theory-Based Synthesis, Practice-Based Reviews, awarded to Pawson, £51,481.50, Grant No: H333250055

G4. 2010-2011, Follow-on Funding Award — Research Synthesis in Action: Investigating the Utility of the Realist Approach in Supporting NICE's Public Health Guidance, awarded to Pawson, £95,881.33, Grant No: RES-189-25-0007

and NHS NIHR awards:

G5. 2003-2004, Joint CHSF/SDO project, Systematically reviewing qualitative and quantitative evidence to inform management and policy making in the health field, £50,000, HS&DR Project — 08/1317/058

G6. 2011-2014, Development of methodological guidance, publication standards and training materials for realist and meta-narrative reviews, co-investigator Pawson, £217,042, HS&DR Project — 10/1008/07

Details of the impact

Based on Pawson's foundational research, `realist evaluation' provided a new lens through which to examine social programmes. Not only does it provide a novel means of testing interventions but, significantly, it does so in a manner that improves targeting and refinement of social policy and practice. Armed with a greater awareness of how measures produce varying impacts in different circumstances, policy makers and practitioners benefit from a better understanding of what policies to implement in particular conditions.

Pawson's insights have subsequently been adopted widely by major national and international organisations charged with policy development and service delivery improvements. Whilst the broad footprint `methodological influence' is challenging to depict, the following key users and beneficiaries can be used as examples to evidence the impact of Pawson's work on research agendas, commissioning priorities and policy guidance.

National Institute for Health and Clinical Excellence (NICE)

NICE sets the standards for healthcare by developing evidence-based clinical guidance and advice for NHS, local authorities, charities, and those with a responsibility for commissioning or providing healthcare, public health or social care services. Pawson's research on `realistic evaluation' is a key part of NICE's methods deployed in developing public health guidance, as noted by its Director of the Centre of Public Health Excellence:

`Firstly, "realistic evaluation" has, and will continue to, revolutionise the field of evaluation on social and health programmes. Second, this idea has profoundly changed, at least potentially, the way that social scientists think about causation. Third, the work has challenged the prevailing damaging and powerful orthodoxies, held widely in government, about the way interventions in human affairs work' [A] (Director of the Centre of Public Health, NICE)

NICE has also made extensive usage of Pawson's research throughout its committee structure, with Pawson serving as a member of its R&D Advisory Committee (2005-10) as well as serving on the Programme Development Group (PDG), which developed guidance on Behavioural Change Interventions (2008-9). Pawson has been called as an expert witness for the PGD on Unintentional Injuries (2009) and for the revisions of the Behavioural Change Guidelines (2013), as well as providing training and workshops to the Public Health Group (2009/10). The NICE Citizen's Council imparts a public perspective on their operations and Pawson was recently invited to consult with them on the role of evidence (2013). As a result of this sustained involvement and professional exchange, realist approaches has found application in much of NICE's guidance. [B]

A recent project, funded by [G4], provides further evidence of the research's impact through professional exchange. This 2009-2011 ESRC project was supported by NICE who seconded a senior analyst to the project. She co-authored many of the research outputs and now plays an anchor role in developing project ideas into official NICE Guidelines Development Manuals.

NHS NIHR HS&DR

The NHS NIHR is the UK's largest funder of research into health service delivery, organisation and management. Methods of evaluation and synthesis are at the core of its commissions and the impact of the realist strategy is evidenced in the current projects portfolio, which include a growing number of realist inquiries [C]. Reciprocal support has been provided by HS&DR, through [G6], a project producing methodological advice, publication standards and training resources for future users seeking to apply the realist and/or meta-narrative approach to systematic review. The corroboration statement [D] from the Associate Director and Chair of the NHS HS&DR Board notes that `the impact of [Pawson's] work can be seen in the research proposals that come before our commissioning board and in the debate at board meetings ... which tend to make more explicit and intelligent use of theory-driven approaches than would have been the case a few years ago'.

Wetenschappelijk Onderzoek — en Documentatiecentrum (WODC), Netherlands.

As an example of the international reach of `realist evaluation' methods, WODC produces recommendations on matters related to all aspects of criminal justice policy in the Netherlands, and is also responsible for research and evaluation regarding legislation and regulations in general for national security and migration. The Director of WODC notes that its advice `carries the Pawson "fingerprints"' and that Pawson's work forms `citation "classics" that are used often when my organization designs evaluations to be commissioned to outside research organizations or are carried out in-house by researchers.' [E].

As a former President of the European Evaluation Society, which aims to stimulate and promote theory, practice and utilisation of high quality evaluation, the Director of WODC also notes that Pawson's work has had a `profound effect on research utilization and policy formation across Europe', bringing a level of `sophistication to the understanding of how interventions work.' [E].

Institute of Health Improvement (IHI), Harvard.

The IHI is a leading innovator in health care improvement worldwide. It offers a wide range of resources and teaching tools to help health care professionals enhance service and clinical outcomes and realist strategies are now a key part of that portfolio. Pawson (April 2008) was invited to give a lecture of realist applications in healthcare at the IHI followed by a two-day workshop at Dartmouth University. Training was continued in a week-long IHI `summer camp' (Vermont, July 2009) in which senior practitioners met to discuss the evaluation of their own interventions (infection control, early response teams, etc.). Research designs were put into place to be implemented over the following years and this work is evidenced in a number of publications (notably JAMA, impact factor 30) that have already emerged from the early consultations. [F,G,H]

Sources to corroborate the impact

A. Letter from Director of the Centre of Public Health Excellence at NICE, received 13 June 2013.

B. Killoran, A. and Kelly, M.P. (2009) Evidence-based Public Health: Effectiveness and efficiency. Oxford: Oxford University Press ISBN-13: 9780199563623. A major publication from the NICE public health directorate including commendations and a chapter on realist approaches.

C. NHS NIHR HS&DR, Catalogue of current projects including realist inquiries:
http://www.netscc.ac.uk/hsdr/projlisting.php?cat=inprogress

D. Letter from Associate Director and Chair of the NHS HS&DR Board, received 19 August 2013.

E. Letter from Director of Wetenschappelijk Onderzoek — en Documentatiecentrum, Netherlands, available on request.

F. IHI publications advocating realist approach: Berwick, D. (2008) The Science of Improvement JAMA 299(10):1182-1184 http://jama.jamanetwork.com/article.aspx?articleid=181589

G. IHI publications advocating realist approach: Davidoff F (2009) Heterogeneity is Not Always Noise: Lessons from Improvement JAMA 302(23) 2580-2586 http://jama.jamanetwork.com/article.aspx?articleid=185078

H. IHI publications advocating realist approach: Ogrinc, G. and Batalden,P. (2009). Realist evaluation as a framework for the assessment of teaching about the improvement of care. The Journal of Nursing Education, 48(12):661-667, http://europepmc.org/abstract/MED/20000246