Summary of the impact

The Pupil Premium Toolkit is an evidence-based resource for schools in England looking for guidance on spending their premium, which is in turn a funding policy to address the effects of poverty on attainment. The continuously developing Toolkit, created by researchers at Durham University, provides a unique cost/benefit summary of the relative impact of different teaching approaches in schools. Independent research suggests it is now used by at least 36% of school leaders in England in determining their spending priorities for the Pupil Premium and to review their support for disadvantaged pupils. It has had a direct impact on the Education Endowment Foundation (EEF) and its funding strategy for the £200 million it will spend over 15 years to reduce inequalities in school outcomes. The EEF's approach to commissioning research and evaluation is explicitly based on this synthesis of research evidence. The Toolkit has also directly influenced Government spending on education and the policy decisions of governments outside England. In March 2013, the Toolkit was identified as a model for the `What Works' network for social policy, which will inform over £200 billion of Government spending.

Underpinning research

The Toolkit is a synthesis of research evidence from meta-analyses and other quantitative studies [R1; R3]. It aims to support schools in spending their resources, especially their Pupil Premium allocation, more thoughtfully and more effectively. The Pupil Premium will account for an estimated £6.25 billion of education spending between 2011 and 2015 so is a significant element of resourcing for disadvantaged pupils.

The contribution of the Toolkit is that it provides estimates of the relative benefit of the impact of different approaches on pupils' attainment, using effect size as a common metric [R1; R2; R3]. In addition it includes an estimate of the financial costs for each of the different approaches. The resulting findings about the relative cost/benefit of adopting different educational approaches on attainment in schools provides highly valued support to schools. The initial research drew on data from over 60 meta-analyses and systematic reviews of approaches and interventions to improve learning in schools. Clear criteria for selecting the meta-analyses have been identified by the authors so that comparable studies with quantitative evidence from well-controlled experimental studies are included [R3], which is a further distinctive feature.

The research insight provided by this synthesis is that many of the approaches initially chosen by schools are either ineffective, or unrealistic to implement, on the basis of the initial levels of funding made available to schools. It advises that some of the most popular uses of the Pupil Premium, such as appointing additional teachers or teaching assistants, are unlikely, on average, to increase pupils' attainment, based on the analysis of research findings. The Toolkit identifies other approaches, such as providing feedback, or developing pupils' skills in planning, monitoring and evaluating learning (meta-cognition), or interventions that have been successful that teaching assistants and others could be trained to carry out, which are all more likely to be successful. It recommends that schools use the information to inform their decision making about the most effective support they can provide for disadvantaged learners, and encourages them to evaluate whatever they select.

The Toolkit was produced from funding awarded by the Sutton Trust to a team from Durham (Higgins, Coe & Kokotsaki) to undertake a new synthesis of `Strategies for Improving Learning'. This research was undertaken between November 2010 and May 2011 and the resulting Toolkit was published in July 2011 [R1].

The research has been adopted by the EEF who have subsequently commissioned Higgins and Coe to extend and update the review annually [R3; S1; S2] for three years (2012-14), and to use it as a basis to develop a methodology for the comparative analysis of EEF projects. The Toolkit is now presented as a public website and called the Sutton Trust/EEF Teaching and Learning Toolkit, with an overall synthesis of the findings, detailed information about each of the areas covered, references to the sources used and a summary of the synthesis of quantitative evidence used to estimate overall effects. It is routinely updated with new and emerging evidence. Other links aim to support take-up and implementation in schools [S1].

The findings in the Toolkit draw on conceptual work undertaken at Durham for the ESRC-funded Researcher Development Initiative (RDI) `Training in the Quantitative Synthesis of Intervention Research Findings in Education and Social Sciences' (April 2008 - March 2011). An extensive database of educational meta-analyses was produced for this project and enabled the preliminary analysis of effect sizes for the Sutton Trust.

Higgins and Coe are Professors in the School of Education, with Coe also the Director of the Centre for Evaluation and Monitoring (CEM). Coe has worked at Durham since 1996, Higgins since 2006. Kokotsaki is a lecturer in the School of Education and a researcher in CEM.

References to the research

Details of the impact

Pupils in England from less affluent families do not achieve as well in school as their more affluent peers, particularly compared with other countries. In 2011, in recognition of this, the Government established a policy, the Pupil Premium, to target resource for these pupils. However it is difficult for schools to decide how to spend this additional resource effectively to improve learning as there is no simple link between more spending and better learning. It is this challenge the Toolkit aims to address.

Impact on the EEF: the impetus for impact from the Pupil Premium Toolkit has been through the work of the Education Endowment Foundation (EEF) and is an example of a creative research partnership between Durham University and an influential educational organisation. Established in 2011, the EEF is an independent grant-making charity dedicated to raising the attainment of disadvantaged pupils in English primary and secondary schools by challenging educational disadvantage, sharing evidence and finding out what is effective in improving pupils' attainment [S1]. The EEF has adopted the Toolkit to inform their decisions about project funding and about evaluation and are committed to developing and extending the Toolkit with the emerging findings and evidence from their own research [S2], exemplifying co-production of research knowledge. The Sutton Trust, one of the founding partners in the EEF, advocated the adoption and development of the Toolkit as a means to develop rigorous comparative evidence about the impact of different educational approaches on disadvantaged pupils in schools. The School of Education and CEM have worked in partnership with the Sutton Trust since 2008. EEF was funded with £135m from the Department for Education (DfE). With investment and fundraising income, the EEF intends to award over £200 million over the 15-year life of the Foundation. The EEF's vision is to break the link between family background and educational achievement, ensuring that pupils from all backgrounds have the opportunity to fulfil their aspirations and make the most of their talents. By the end of May 2013 it had funded 56 projects at a cost of £28.7 million, reaching about 1,800 primary and secondary schools and 300,000 pupils [S3]. Evidence in the Toolkit informs decisions about which projects the EEF funds and the type of evaluations commissioned [S2, p 16; S4]. The Foundation acknowledges the Toolkit as their "primary means of disseminating knowledge of what works in education" [S2, p. 19]. The success of the Pupil Premium Toolkit has encouraged the Sutton Trust to commission other similar projects, such as the `Sutton Trust Access Toolkit' for Higher Education.

Impact on national and local education policy: in March 2013, the Cabinet Office announced the creation of a `What Works' network for social policy, to inform decision-making on £200 billion of public spending [S3]. The EEF and Sutton Trust were designated the national `What Works' centre for schooling and the Toolkit forms the heart of this work [S3; S4]. The Toolkit was cited as an exemplary model of the presentation of clear, high-quality evidence which the four new centres (in crime, economic growth, ageing, and early intervention) should aim to emulate [S3]. The Toolkit has also had a direct influence on policy spending with £50M spent on funding Summer Schools for disadvantaged pupils in 2012 and another £50M committed for 2013. This policy was developed by the Deputy Prime Minister's Office who used the evidence in the Toolkit to identify the policy focus on summer school provision [S4]. Ofsted also cited the evidence summarized in the Toolkit as influencing their judgments about effective use of the Pupil Premium [S5]. This has already had a significant effect on schools as Ofsted have responsibility for reporting on how effectively schools are spending their Pupil Premium allocation. Since 2011, a significant number of English Local Authorities have also endorsed the Toolkit. These include Barnet, Bradford, Bristol, Derby, East Sussex, Gloucestershire, Harringay, Kent, Leicestershire, Medway, Northumberland, Shropshire, Staffordshire, Suffolk, Wiltshire, Wolverhampton and York, all linking to the Toolkit through their websites and online support for schools [S6]. The Toolkit is also recommended on the Welsh Government's `Learning Wales' website for effective allocation of funds from the Pupil Deprivation Grant and the Toolkit was reviewed by the Research and Information Service in a briefing for the Northern Ireland Assembly's discussion of the Pupil Premium in England.

Impact on schools: The most significant impact of the Toolkit has been on schools. It has been used voluntarily and directly by hundreds of schools to guide their spending and their teaching priorities when allocating Pupil Premium funds and informing parents about their rationale. A representative survey of schools commissioned by the Sutton Trust, and undertaken by NFER in 2012, and repeated in 2013, asked schools about their priorities in spending the Pupil Premium. The report states [S7, p 10] that the "responses from 2013 about how decisions are made about approaches to adopt to improve pupil learning were very similar to those given in 2012...Among senior leaders there were three options which showed a change... The option, reading the pupil premium toolkit published by the Sutton Trust, ... saw an increase in response among senior leaders, from one in ten in 2012 (11%) to over a third in 2013 (36%)." This level of take up in schools is corroborated by the Evaluation of the Pupil Premium report by Manchester and Newcastle Universities for the Department for Education.

In 2012-13, the total Pupil Premium spending was £1.099 billion and £1.875 billion in 2013-14. This suggests that if 11% of senior leaders used the Toolkit to guide decision making in 2012, it influenced £120 million of spending in schools (11% of £1.099 bn), and that if 36% of senior leaders said they used the toolkit to guide spending in March 2013, it has influenced the allocation of about £675 million of school spending for 2013-14 (36% of £1.875 bn).

Google searching reveals at least 120 schools which have acknowledged on their website the contribution of the Sutton Trust/EEF Toolkit to their rationale for spending the Pupil Premium and how they have allocated the funds. Most of these websites describe how their decisions for allocating funding have been influenced by the Toolkit, both in terms of identifying strategies which tend to be more effective, but also outlining how they will address the potential disadvantages of approaches which have been identified as less effective, such as using teaching assistants for intensive one-to-one support, rather than as general classroom help. A typical example of this is Longfield Academy in Kent [S8], which allocated about £180,000 between 2011-13, and developed its plan based on Toolkit guidance. The Academy does not have any direct links with Durham University, EEF or the Sutton Trust. Overall this indicates that the Toolkit has made a clear and distinctive contribution to schools and has had a direct impact on their spending of the Pupil Premium and their priorities for supporting their disadvantaged pupils.

Impact on the Media and on politics: the Toolkit has been widely reported in the educational media with more than 20 articles in online newspapers, professional journals and magazines, such as the Guardian Online, Times Educational Supplement, The House, Headteacher Update, Governing Matters, Teaching Leaders and Primary Headship. It is recommended on the websites of more than 20 educational organisations, from charities such as the Campaign for Learning, to publishers such as Pearson and Oxford University Press, including endorsements from influential bodies like the Association of School and College Leaders and the National Association of Headteachers [S9]. It has also been acknowledged in the House of Commons as a helpful summary of approaches to narrow the gap between rich and poor (28^th June, 2012, 143WH) and in the Welsh Assembly by the Minister for Education and Skills, (01/05/2013, 17.29 pm) as "clear evidence of what really works in terms of turning around performance". The success of the Toolkit has influenced the Sutton Trust's commissioning of research, such as their evidence review to create a `Higher Education Access Toolkit' in 2012. In June 2013 the Toolkit was awarded an `Inspiration for Government Award' by the The Institute for Government (IfG), which is an independent charity and think tank with funding from the Gatsby Foundation, promoting more effective government with cross-party support [S10].

Sources to corroborate the impact

S1. EEF website and the Toolkit: http://educationendowmentfoundation.org.uk/toolkit/ The evidence analysis is updated annually and links to the EEF's research and evaluation projects.

S2. EEF Annual Report 2012:
http://educationendowmentfoundation.org.uk/uploads/pdf/Annual_Report_2011-12.pdf "The Sutton Trust-EEF Teaching and Learning Toolkit is our primary means of disseminating knowledge of what works in education" (page 19).

S3. Government announcement of the evidence centres (4/3/2013):
https://www.gov.uk/government/news/new-world-leading-evidence-centres-to-drive-better-decisions-across-200bn-of-public-services.

S4. Letter from the Chief Executive of the Education Endowment Foundation.

S5. Ofsted (2012) The Pupil Premium: How schools are using the Pupil Premium funding to raise achievement for disadvantaged pupils September 2012, No. 120197 Ofsted London:
http://www.ofsted.gov.uk/resources/pupil-premium (page 11).

S6. In 2012 the Suffolk LA website providing the Toolkit to Schools stated: "The Sutton Trust have published excellent independent guidance on the resources that improve attainment for disadvantaged pupils. It is in the form of a toolkit on what works, drawing on research evidence and linking this to a cost benefit analysis." The updated website points to the current version of the Toolkit available through the EEF:
http://www.suffolklearning.co.uk/content.asp?did=6728 Other similar LA websites pointing schools to the Toolkit or indicating LA endorsement have been identified for Barnet, Bradford, Bristol, Derby, East Sussex, Gloucestershire, Harringay, Kent, Leicestershire, Medway, Northumberland, Shropshire, Staffordshire, Wiltshire, Wolverhampton and York.

S7. NFER Report commissioned by the Sutton Trust: http://www.suttontrust.com/research/nfer-teacher-voice-omnibus/

S8. Longfield Academy website and rationale for spending allocation based on the Toolkit: http://www.longfieldacademy.org/aboutus/pupil-premium.php.

S9. NAHT endorsement of the toolkit: http://www.naht.org.uk/welcome/news-and-media/key-topics/funding/naht-welcomes-advice-on-targeting-funds-for-disadvantaged-pupils/

S10. The Institute for Government award: http://www.instituteforgovernment.org.uk/inspiration-government-award.