Summary of the impact

The IOE's Evidence for Policy and Practice Information and Co-ordinating (EPPI) Centre has helped UK and overseas governments to make better-informed decisions on education, public health and health education, social welfare and international development, through the use of systematic reviews. The internationally-respected research team has done this by developing and improving methods and tools that produce a broader range of high quality, reliable studies better able to answer policy-makers' questions. EPPI trains civil servants to formulate effective research questions, supports NICE and WHO in ensuring the validity of their guidelines, and works with governments across the EU to build capacity in evidence-informed policy and practice in education.

Underpinning research

Context: Research reviews inform decisions that affect people's lives, so it is important that they are done well. In addition to producing and supporting research reviews for a range of government and third sector clients — some 180 papers between 1996 and 2013 — EPPI leads the development of methodology in this young field. Its innovations — particularly in synthesising qualitative research and conducting mixed methods reviews — have enabled reviews to extend their reach from medicine (their original purpose) to social care, education and international development.

Systematic reviewing: Led by Professor Ann Oakley from its inception in 1993 until 2005 and Professor David Gough since 2005, EPPI has extended systematic review methodology to embrace a broader range of studies for understanding policy problems, comparing policy options and informing policy implementation. Innovations include ways to draw out new themes and narratives from qualitative research and ways to integrate data from experimental studies with data from other study types through multi method and multi component reviews. EPPI's trademark two-stage reviews start by drawing a broad picture of what evidence is available to help clients identify the most useful focus for the second stage, an in-depth analysis.

Tools for research reviews: EPPI has developed a framework to help researchers and funders select the most appropriate review method for their study (see reference R1 p66) as well as the state-of-the-art EPPI-Reviewer, a web application that enables researchers (who may be geographically dispersed) to manage the entire lifecycle of a review and to analyse the results. With 2,700 user accounts worldwide, it incorporates pioneering data-mining techniques, and supports the identification and coding of studies, followed by statistical and/or conceptual synthesis of the findings. Paper R5 is an innovative `how to' guide for synthesising qualitative research.

User involvement: EPPI researchers have improved systematic reviewing by tapping into the expertise of research users — such as health service patients, practitioners, funders and policymakers (R1 p17). Involving stakeholders in designing research questions and frameworks helps systematic reviews address their priorities (e.g. R3). Studies focusing specifically on user involvement in guiding research (e.g. R6) have used systematic methods to identify successful approaches.

Exemplar studies: Microfinance in sub-Saharan Africa: Key findings: This 2010 DFID-funded research review by Ruth Stewart and colleagues from the University of Johannesburg was the first major study to cast doubt on the benefits of microfinance for the very poor in the developing world (R2). Researchers concluded that these small-loan aid projects make some people poorer.

Microfinance also harmed some children's education (particularly girls) because parents re-prioritised spending and could not pay school charges. How the research was conducted: Guided by a peer-reviewed protocol, researchers drew on the expertise of potential users to find relevant literature, and searched specialist libraries and databases. The quantitative findings from 15 studies deemed `good enough' were synthesised into categories (positive, negative or varied effects of microfinance) and qualitative findings into a narrative. Lastly, researchers developed a `causal chain' to unpack the different ways microfinance affects poor people.

Smoking cessation programmes in pregnancy: Key findings: There are effective strategies to help pregnant women to stop smoking that lead to fewer premature babies and better birth weights. The most effective intervention — providing incentives to stop — helped nearly a quarter of women to quit smoking during pregnancy. How the research was conducted: The study was designed in consultation with health professionals and women who had smoked before or during pregnancy to make sure it addressed issues important to those with the greatest interest in its findings (R3).

Researchers found 64 appropriate randomised and quasi-randomised trials. They also extracted qualitative data and discovered the views of pregnant women seldom informed research design. This tranche of research, including several updates, was led from Australia, with support from EPPI and its researchers. Sandy Oliver (since named Professor) was involved throughout and designed the stakeholder involvement.

References to the research

Indicators of quality:

IQ1: R2 is DFID's most downloaded systematic review - 14,180 from 04/12-07/13.

IQ2: EPPI is the Methods for Research Synthesis node of the ESRC National Centre for Research Methods.

IQ3: "I have very little hesitation in recommending the work of the EPPI centre as being of the highest quality. The work undertaken in the last period of funding has been published in high quality peer reviewed journals and widely cited thereafter. It is frequently used in reviews of reviews and held by other agencies like NICE to be of the best quality." From a set of independent external anonymous reviews commissioned by the DH in 2009 (see impact source S5).

Funding/Grants: EPPI Centre has a wide range of funders, including the Cochrane Collaboration, ESRC, Nuffield Foundation, Wellcome Trust, EC and six government departments. DH has funded EPPI continually for 20 years, with current grants extending until 2019. The Centre's total grant funding during the REF period is £5,641,197.

Details of the impact

Principal beneficiaries include UK and worldwide policy-makers, including WHO, international development NGOs and education ministries in Europe; users of public services (i.e. the general public, including children) and those who work in them.

Dates of impact: From 2008-13, affecting many different policies at different times.

Reach and significance: EPPI's continuous development of synthesis methodology has made it possible for an ever-broadening range of decisions to be informed by research — from EPPI itself and others using its training, methods and tools. These include policy decisions made by governments and agencies globally and decisions by individuals considering their own health choices. The impact has been instrumental¹, supporting changes in policy, guidance and practice for services provided to different groups within society; conceptual — helping change the way politicians and civil servants develop policy; and capacity-building — by training and supporting others in review methods and use (R5). Its reach is global: for instance, work with WHO and DFID extends this training and support to low-income countries.

Impact on research-informed policy-making capacity: Independent external anonymous reviews of the programme, commissioned by DH in 2009, reveal the extent of EPPI's influence (S5). Reviewers praised its pioneering innovations in methodology, its approach to working with clients and its commitment to building capacity around the world:

• "They have contributed significantly to our ability to spot `what works best for whom in what circumstances'."
• "The two-stage approach to systematic reviews is critical ... Not only does it provide a key way of enhancing the dialogue with policymakers ... it also promotes the effective use of resources".
• EPPI developed "methods to synthesise the kind of complex interventions that are so urgently needed to promote health and reduce health inequalities".
• The Centre "led the way in persuading other review groups to take seriously a broader range of research evidence ... Ultimately this has to be to the benefit of the policy-making community".

UK Government: EPPI trains policy-makers and academics to enable government to make better use of systematic reviewing. For example, they teach civil servants at DFID, DH and other departments how to develop focused research questions that will result in more useful reviews. Its methods enabled the benefits of systematic reviewing — which formerly analysed randomised controlled trials of health interventions — to extend its reach to other areas, particularly education.

[text removed for publication]

Health guidance: EPPI's expertise in research synthesis methods and in reviewing social care literatures led to two NICE contracts in 2013: a) to be a part of NICE's National Collaborating Centre in Social Care that develops and disseminates guidance and standards and b) to lead a collaboration between several universities that acts as NICE's Research Support Unit. The Centre is a formal partner of both the internationally prestigious Campbell Collaboration (social policy reviews) and the Cochrane Collaboration (healthcare reviews). Oliver is a member of the WHO Expert Advisory Panel on Clinical Practice Guidelines and Research Methods and Ethics and of the NICE Accreditation Advisory Committee.

Public involvement: Oliver's work has helped ensure that user perspectives inform health guidance. With colleagues she developed new research-informed resources and procedures to support patient and public involvement in shaping health technology assessments to meet NHS needs. Guidance authored by Oliver was publicly available on the National Institute for Health Research website until the summer of 2013. An evaluation by the IOE's Social Science Research Unit (2006) showed that public involvement changed the focus of research commissions and added new outcomes — patient organisations and charities highlighted patients' views, social contexts, long-term implications and the provision of information to consumers — issues not usually addressed by professionals. The same body of research (e.g. R6) informed the WHO handbook for guideline development (2012, p15, S3) and the James Lind Alliance guidebook, co-authored by Oliver, showing step-by-step how to bring patients, carers and clinicians together to identify treatments needing urgent study (jlaguidebook.org, updated 2013).

Education Impact in Europe: EPPI leads the EU-funded Evidence Informed Policy and Practice in Education in Europe (EIPPEE) project (2011-13), which links 36 partner countries and 600 individual network members, and helps countries across Europe develop their research-informed policy-making capabilities. Thirty-nine people from 20 countries have benefited from its free online training courses and more than 250 people from eight countries from its tailored workshops — including many from education ministries. EPPI's consultancy service helped education policy-makers in the City of York select the highest impact strategies for narrowing the learning gap and develop the `York Challenge' approach for 2013-14 (S7). The consultancy "helped to unpack the research so that we have been able to plan implementation processes with schools", they said.

Wider international collaboration: From 2008-13 EPPI provided training for policy-making bodies around the world, including education and justice ministries in New Zealand, India's National Council for Educational Research and Training, the European Food Standards Agency and Japan's Economic and Social Research Institute. EPPI is a partner in an international Global Evidence Initiative to build review capacity for policy and practice in low and middle income countries and supports WHO-funded Centres in Bangladesh, Chile, China, Lebanon, India and South Africa to carry out systematic reviews relevant to health systems policy.

Professional engagement: EPPI has partnerships or strong links with the UK's What Works Centres including NICE, the Education Endowment Fund and the new What Works Centre for Crime. Impact of individual reports: EPPI studies have broad national and international impact, taking in neonatal health in developing countries, sex education interventions and plain tobacco packaging. We have chosen two very different high impact reports as examples of EPPI's diverse impact. Education for smoking cessation in pregnancy: Beneficiaries include infants and mothers in at least five countries. By addressing health-worker and patient concerns, the researchers shaped questions around how to educate and help women to stop smoking, not just the impact of smoking (R3). The findings inform clinical guidelines in Australia, Brazil, South Africa and the UK, as well as those published by the International Primary Care Respiratory Group. The 2010 NICE and NHS Wales guidelines say all pregnant smokers should be automatically referred to stop smoking services and offered help to quit. This was a key recommendation of the EPPI research (R4), which is cited in the guidelines' background and briefing papers (e.g., S4). The research also informed the American 5As clinical programme (Ask, Advise, Assess, Assist, Arrange) (S6).

Microfinance in Sub-Saharan Africa: The 2010 study (R2) and the 2012 follow-up sparked debate and helped bring a more questioning approach to microfinance to the international development community. Policy change in Norway: The research contributed to the 2012 decision of the country's aid agency Norad to stop funding most new microfinance institutions, after more than a decade as a key donor. The decision followed a TV exposé by a Danish journalist, to which Stewart contributed. In its response, Norad stated it was "well aware of the new research in the microfinance area, including the systematic reviews" (S8). The Anglican Communion asked Stewart to contribute to its Economic Empowerment Workshop in Nairobi in 2012. Priorities agreed included "development of new products and services that can provide access to finance for the most poor" and better financial literacy education. Professional and public engagement: Impact was heightened by an intensive programme of meetings, briefings and colloquia in 2011-12, including with Comic Relief, FSA, World Bank, Cochrane Colloquium, the South African government and the House of Commons Microfinance All Party Parliamentary Group. [text removed for publication]

Sources to corroborate the impact

S1: [text removed for publication]

S2: [text removed for publication]

S3: Preparatory work for WHO guidelines: Fretheim A, Schünemann HJ and Oxman AD (2006) Improving the use of research evidence in guideline development: 3. Group composition and consultation process. Health Research Policy and Systems 4:15

S4: Bauld, L., Coleman, T. (2009), The Effectiveness of Smoking Cessation Interventions during Pregnancy: A Briefing Paper http://www.nice.org.uk/nicemedia/live/13023/49422/49422.pdf

S5: Independent external anonymous reviews commissioned by the DH in 2009 (available)

S6: http://www.acog.org/~/media/Departments/Tobacco%20Alcohol%20and%20Substance%20Abuse/SCDP.pdf?dmc=1&ts=20130730T0540537089 (US guidance cites R4 in its own right; R4 is also a key source for other underpinning research)

S7: City of York Challenge (comment available)

S8: http://bdnews24.com/bangladesh/2012/06/19/norad-won-t-back-micro-lenders

---

1 Using Evidence: How Research can Inform Public Services (Nutley, S., Walter, I., Davis, H. 2007)

2 All web links accessed 29/10/13