Summary of the impact

The Research Unit for Research Utilisation (RURU) has had wide-ranging impact on the ways in which policymakers, research funders, intermediary bodies and practitioners think about research use, the strategies they employ to enhance research influence, and their assessment of research impact. RURU has helped to transform thinking from ideas of one-way `knowledge transfer' towards more situated and interactive models, which are about influencing organisational as well as individual behaviour. The reach of the impact has been international (e.g. Australia, Canada, the USA and Scandinavia, as well as the UK), and cross-sectoral (encompassing the criminal justice, education, healthcare and social care sectors). The overarching contribution has been towards more effective research policy, better public policy making and improved public service delivery.

Underpinning research

The body of research underpinning this impact has been undertaken by members of RURU since 1996, with RURU itself being established in 2001 with extensive funding from the ESRC. The key researchers all work in the School of Management at the University of St Andrews: Sandra Nutley (professor; 1992-2006 and again since April 2012); Huw Davies (professor; 1996 to present); Isabel Walter (research fellow; 2001 to 2013); Alison Powell (research fellow; 2008 to present). RURU's research responded to growing international interest in evidence-based policy and practice in the late 1990s/early 2000s. It has focused on increasing our understanding of research use in public policy and practice settings, and how such use can be enhanced. RURU has drawn on these understandings to investigate and elaborate various approaches to assessing research impact. The nature of the research insights that underpin the impact described here are summarised as follows.

• Articulation of research use as a complex, social, interactive, highly contingent and context-dependent process in which research is more likely to be adapted than simply adopted (e.g. Davies, Nutley & Smith 2000; Nutley, Walter & Davies 2007).
• Production and refinement of a taxonomy of strategies to improve the use of research, which identifies five key underlying mechanisms: dissemination; interaction; social influence; facilitation; and incentives/reinforcement (e.g. Nutley, Walter & Davies 2007).
• Reviews of the evidence about the success or otherwise of different strategies and mechanisms for increasing research use and impact, leading to eight guiding principles to support the use of research in practice (e.g. Walter, Nutley & Davies 2005; Nutley, Walter & Davies 2007).
• Identification that research use strategies benefit from adopting a wider target audience than just individual research users and from focusing on more than just instrumental research use (e.g. Nutley, Walter & Davies 2007; Nutley, Walter & Davies 2009).
• Articulation of different types of research impact, identification of the multiple routes by which research can have impact, and reviews of the appropriateness of different approaches to assessing research impact (e.g. Davies, Nutley & Walter 2005; Nutley, Walter & Davies 2007).
• Discussion of the implications of the above for research commissioning processes, research intermediaries and policy-making bodies (e.g. Walshe & Davies 2010).

Pathways to impact were developed through multiple workshops, symposia and network events; collaborative empirical and research synthesis projects; collaborations and bespoke training with government agencies and funding bodies; invited consultancy; and a wide range of peer-reviewed and practitioner-oriented publications.

References to the research

The core references What Works? (Davies, Nutley & Smith 2000) and Using Evidence (Nutley, Walter & Davies 2007) built on a raft of previously published peer-reviewed work that meets or exceeds the 2* quality threshold; the peer-reviewed references listed are also of similar quality. Using Evidence was the main output from an ESRC Research Resources grant (PI Nutley; Co-I Davies; Ref. H141251001; 2001-2005, £232K). The ESRC provided no formal grading for the end of award report, but the anonymous peer reviewers described our work as `world class' and of `lasting value for both academic and practitioner communities'. Indeed, one reviewer commented that, `overall, the work of RURU is some of the best in the world on addressing the growing area of evidence-based decision-making in the broad social policy area'.

Details of the impact

Two broad spheres of impact are outlined below along with some of the pathways (links) to these impacts. Impact is seen in changes in the ways key actors view `research-use processes' and the actions that followed from such improved understandings.

Impact has been facilitated by the way in which members of RURU have sought to engage with policy and practice audiences both during the research process and following the publication of findings. This has involved working with many bodies (see examples below) as they seek to understand the implications of our research for their organisations. RURU's impact has also been achieved by `secondary links' as existing users apply, cite and recommend our work to others (see examples below). In this way the reach of our influence, direct and indirect, has crossed country and sector boundaries (e.g. criminal justice, education, health and social care). The ultimate over-arching significance of our impact lies in more effective research investment, redesigned knowledge sharing activities, improved public policy making and enhanced public service delivery. The reach and significance of this impact was recognised in 2011 by the Campbell Collaboration when Nutley was presented with the Robert Boruch Award for research that informs public policy, see http://www.campbellcollaboration.org/news_/Boruch_Award_to_Sandra_Nutley.php.

Influence on the strategies and practices of research funding bodies
RURU's influence is evident in the strategies and activities of at least six research funding bodies and details of four of these are provided below: the ESRC (UK); the WT Grant Foundation (US), the NIHR SDO Programme (UK) and the Australian NHMRC.

In 2005 RURU was approached by the ESRC to carry out underpinning research for an international symposium on `Assessing the non-academic impact of social science research'. We co-wrote a briefing paper for that symposium as well as a subsequent report (Davies, Nutley & Walter 2005). This report and subsequent research (Nutley, Walter & Davies 2007) is widely cited in two ESRC policy documents on research impact assessment: Taking Stock (2009 - S6) and Branching Out (2010 - S7). The influence of RURU's research has also been confirmed by testimony from ESRC's Head of Evaluation, Strategy and Analysis who commented that `RURU's work has underpinned the way in which we conceptualise and understand research impact... we are now much better able to articulate the linkages between research findings, application, influence and impact' (S1)

In 2008, the WT Grant Foundation (New York) drew on RURU's work when initiating a new stream of funding exploring the use of evidence in the United States. RURU's influence is evidenced in the 2008 and 2010 Calls for Proposals, which refer extensively to RURU publications (S8). These formal statements are heavily influenced by direct dialogue with RURU and the provision of bespoke reviews drawing on RURU research. The Vice President, WT Grant Foundation, has commented that RURU's research has `played a defining role' in the Foundation's `support of the use of research evidence in policy and practice... More than any other body of work, [it] has shaped our understanding of the field and promising directions for future work' (S2).

In 2008-2009, RURU's work had a major influence on the National Institute for Health Research Service Delivery and Organisation (NIHR SDO) programme's decision to invest around £3.5m in knowledge mobilisation activities, and Davies was invited to direct this programme on secondment (40%fte; 2008-10). Other NIHR investments, such as the CLAHRCs (collaborative partnerships between universities and surrounding NHS organisations) have been directly influenced by RURU's work: Davies sat on the original funding panel (c. £90m; 2008) and chaired the panel for second round funding (c. £124m; 2013); he also commissioned the external evaluations of the original CLAHRCs through NIHR SDO (c. £2m of research investment; 2009), and committed extensive time to supporting all nine CLAHRCs during their establishment while on secondment to NIHR (20%fte; 2008-2010). Evidence of the influence of RURU's work on the practices of the CLAHRCs is provided by testimony from one of their directors, who reported that `the social and interactive model of knowledge co-production articulated in Using Evidence and promulgated by Davies & Nutley through RURU ... provided a guiding philosophy for much of our work ... for which we have been commended nationally and internationally' (S3).

Influenced by the CLAHRCs, a similar combined `research, policy and practice' initiative was launched by the Australian National Health and Medical Research Council (NHMRC) in 2012. Due to the importance of RURU's work as an underpinning guide, Davies was invited to sit on the Expert Advisory Panel to help shape this investment (c. A$70m) in line with the research insights identified in Section 2.

Influence on the thinking and actions of those supporting policy and practice development
RURU's influence has been particularly marked amongst analysts and trainers working in government departments, and those working in intermediary bodies tasked with improving the use of research in public services. Indirectly, therefore, the underpinning research in this case supports improved public services through the better application of evidence. Examples below are drawn from Canada, the US and the UK.

RURU's work directly shaped the thinking and practices of the Research Division of Ontario's Ministry of Education 2008-2012, as it began to place more emphasis on knowledge mobilisation. `RURU's work was influential in the design and development of a new Ontario Research and Evaluation Strategy by the Ontario Ministry of Education. Particularly important... was evidence from RURU about the need to pay attention to developing individual and organisational capacity for research use and to developing a strategy for the systemic use of research' (Founding Director of the Education Research and Evaluation Strategy Branch, Ontario Ministry of Education - S4). This impact was facilitated through the provision of web-based resources and site visits (Nutley, 2011). In addition, RURU's work shaped the thinking and activities of Quebec's National Institute for Public Health (INSPQ). In 2009, INSPQ produced a knowledge transfer guide, which built on and cited RURU's work (S10). RURU's wide dissemination of research outputs led to the initial impact, and Nutley subsequently visited Quebec in June 2011 to work directly with staff from INSPQ and other colleagues interested in further developing knowledge transfer activities in Quebec.

Further west in Canada, RURU's work influenced the development and delivery of British Columbia's internal training programme on evidence-informed policy (2009), and a copy of Using Evidence (Nutley, Walter & Davies 2007) was provided by the Province to all civil servants on this course (250 copies in total). The dissemination activities of RURU enabled this impact and RURU's influence was reinforced when Nutley took up the offer to work with members of the Office of the Chief Information Officer for several weeks in 2009.

More recently, in 2012, the US National Research Council of the National Academies (`Advisers to the Nation on Science, Engineering and Medicine') produced an authoritative digest on `Using Science as Evidence in Public Policy', a distillation of expert input from a committee of sixteen US experts meeting over a three-year period (2009-12). The work of RURU is cited eight times in the document, with extensive quotes drawn from Using Evidence (Nutley, Walter & Davies 2007) [S9].

A final example is RURU's impact on the UK Alliance for Useful Evidence and the What Works Evidence Centres for Social Policy (www.alliance4usefulevidence.org). RURU has `contributed to and shaped Alliance debates on what counts as evidence and how research use can be improved' (Manager, Alliance for Useful Evidence - S5). Davies, Nutley and Powell have been actively engaged with the emerging activities of the Alliance during 2012 and 2013 through dialogue, commissioned pieces of work, blogs and participation in events. They co-wrote an Alliance provocation paper on What counts as good evidence? (which can be found at: http://www.alliance4usefulevidence.org/assets/What-Counts-as-Good-Evidence-WEB.pdf). This was subsequently picked up by UK Cabinet Office advisors who sought RURU's advice on the operation of the What Works centres (announced by the UK Government in March 2013).

Sources to corroborate the impact

Testimonies
Testimonies providing further evidence of the impact claimed in this case study have been received from the following people/organisations (held in a central repository by the University):

[S1] Head of Evaluation, Strategy and Analysis, Economic and Social Research Council, UK

[S2] Vice President, Programs, WT Grant Foundation, USA

[S3] Director, National Institute for Health Research (NIHR) Collaboration for Leadership in Health Research and Care (CLAHRC) for South Yorkshire, UK.

[S4] Chief Research Officer and Founding Director of the Education Research and Evaluation Strategy Branch, Ontario Ministry of Education, Canada

[S5] Manager, Alliance for Useful Evidence, UK.

Report citations.

[S6] ESRC (2009) Taking Stock: A summary of ESRC's work to evaluate the impact of research on policy and practice, http://www.esrc.ac.uk/_images/Taking%20Stock_tcm8-4545.pdf. Pages 3, 4, 11 and 20 corroborate claims of impact on ESRC's approach to assessing research impact.

[S7] ESRC (2010) Branching out: New direction in impact evaluation from the ESRC's Evaluation Committee, http://www.esrc.ac.uk/_images/Branching%20Out_tcm8-14881.pdf. Page 5 corroborates claims of on-going impact on ESRC's approach to assessing research impact.

[S8] WT Grant Foundation's 2010 call for proposals (available from University of St Andrews). Pages 3 & 7 corroborate claims of impact on the WT Grant Foundation's new stream of funding to support studies of the use of research in policy and practice in the USA.

[S9] National Research Council (2012) Using Science as Evidence in Public Policy, Committee on the Use of Social Science Knowledge in Public Policy, The National Academies Press. http://www.nap.edu/catalog.php?record_id=13460
Pages 37-39, 43, 49-50, 79 and 85-86 corroborate claims of impact on this committee and its recommendations as it reviewed ways of strengthening the use of social science.

[S10] Lemire, N., Souffez, K., Laurendeau, M.-C. (2009). Animer un processus de transfert des connaissances: Bilan des connaissances et outil d'animation, Institut national de santé publique du Québec (http://www.inspq.qc.ca/pdf/publications/1012_AnimerTransfertConn_Bilan.pdf). Pages 23, 25, 37 and 44 corroborate claims of impact on Quebec's National Institute for Public Health's knowledge transfer guide 2009.