Summary of the impact

School accountability is a crucial part of most education systems. The UK is a world leader in the use of school accountability and Professor Simon Burgess's research at Bristol has had a significant impact on national policy on school accountability in England and Wales. He has also provided advice on this policy internationally. His research showed that the abolition of comparative school performance information in Wales significantly damaged pupil attainment, and widespread public reporting of his results contributed to the reversal of this policy in 2011. In England, his work influenced changes in the content of the school `league tables' in 2011, leading directly to an improvement in their usefulness for parents in their school choice and government in its role as funder and regulator. In both cases, pupils are likely to benefit from the changes following Burgess's work by achieving higher grades in their crucial GCSE exams and thereby from higher lifetime income.

Underpinning research

The research was published initially in two University of Bristol CMPO Working Papers, [1] and [2]. Together these papers showed that school league tables play a key role as an accountability mechanism in regulating the market for schools. Paper [1] showed that the removal of comparative performance information from the public domain had a substantial negative effect on pupil attainment, while paper [2] showed that the nature of the performance information significantly affects its functionality. This research is part of a substantial ongoing programme of research and policy engagement on schools markets led by Burgess, and funded as part of the ESRC's Centre grant to CMPO [6]. Burgess joined the University of Bristol in 1987.

a. The impact of abolishing school league tables in Wales

The initial research [1] was carried out in 2010 with Deborah Wilson, a Reader at the University of Bristol, and Jack Worth, then a Research Assistant at CMPO, now a Research Manager at NFER.

When control over education was passed to the Welsh Government in 2000, one of their first acts was to abolish `school league tables'. This reform provided an opportunity to test decisively for the first time the theory that such information is important to school performance. Burgess and co-authors collected data on all secondary school pupils over a decade, before and after the reform, in both countries. The data included GCSE performance, prior attainment, pupil demographics, school expenditure, and neighbourhood characteristics. The analysis involved taking each secondary school in Wales and comparing its performance to a very similar school in England, both before and after the reform. It found significant and robust evidence that the policy change substantially reduced school effectiveness in Wales, on average by about two GCSE grades per pupil. The strongest effect was on schools in more disadvantaged neighbourhoods. So the policy resulted in both lower overall school effectiveness and greater inequality.

This research was subsequently published in 2013 in the Journal of Public Economics, [3].

b. The functionality of school league tables for parents

The initial research [2] was carried out in 2010 with Rebecca Allen, a Reader at the Institute of Education, University of London (both authors were equally responsible for the initiating idea and for the empirical implementation).

The debate about the best metrics to use in school performance tables has largely been conducted by assertion. In 2010, Burgess and Allen set out a model to determine the answer statistically by asking which metric was most useful to parents trying to decide which school their child would do best in academically. They used the entire cohort of more than half a million pupils who chose secondary school in 2003 and tracked their progress. The results showed that some school performance metrics were very useful, significantly and substantially producing better choices than choosing at random, while others were much less helpful.

The research has subsequently been published in a technical form [4] and has also been incorporated in a broader discussion of the merits of different forms of performance table, [5]. This latter paper proposes some new metrics that were incorporated in the published league tables from 2011.

References to the research

The high quality of the research is corroborated by the quality of the journal publications and the high amount of associated peer-reviewed competitive grant funding.

Grants

[6] Burgess, S. (PI): ESRC CMPO Centre Grants, RES-343-28-3001 and RES-343-28-0001, £4.8m, 2004-2014.

Details of the impact

a. Broad and sustained influence on national and international policy

Simon Burgess has had a broad and sustained influence on policy for the schools market in the UK, of which the two specific cases on school accountability are leading examples. Carole Willis, Director of Research and Analysis at the Department for Education writes [a]: "Simon Burgess's work on school accountability has had a major impact on thinking in the Department on this issue over a number of years. His research was cited in the "Importance of Teaching" White Paper, and has fundamentally changed Ofsted's risk assessments of which schools to inspect."

This influence is ongoing. Tim Leunig, Special Advisor to Secretary of State Michael Gove, writes in September 2013 [b] "On the basis of his research leadership in this field, we asked Simon Burgess to contribute to the development of the new pupil progress measure, and the school floor targets that will form the basis for interventions in schools". Burgess was also asked to talk at the Permanent Secretary's Seminar on school markets in May 2013. Strengthening school accountability is also a key policy issue in many other countries and Burgess was asked to present his results to the Spanish Minister for Education and Secretary of State for Education [c].

In addition to school accountability, Burgess's wider research on education has also influenced the thinking in the Department for Education on key policy issues including teacher effectiveness, school choice, school financial decisions and teacher performance information through briefings, seminars, presentations and reports.

b. Impact of the research on the abolition of performance tables in Wales

Publication of [1] was the lead item in the main evening news on BBC Wales, 2 November 2010. The next day, questions on the research were put to the First Minister of Wales on the floor of the Assembly. The findings were described as a "slap in the face with good data" by the Senior Policy Advisor to the Welsh Assembly, Professor David Reynolds [d]. The research was further reported in other news and documentary programmes for BBC radio and television, in the Financial Times and in The Economist (twice).

In February 2011, the Welsh Minister for Education announced a major change of policy, to reverse the decade-long policy of not publishing performance information [e]. The new system was introduced on 8 December 2011.

Burgess's research was influential in this decision as David Reynolds confirms [f]: "I can testify to the major impact that the research of Simon Burgess and colleagues had on our educational thinking in general and specific policies in particular. The Minister Leighton Andrews found the research chimed with much of his own thinking about the need for transparency on school performance and there was much internal discussion within WG after the publication of the particular paper in October 2010. The research helped to prepare for the introduction of the Banding system we launched in 2012 which of course involved the publication of school results. It was for us quite seminal research."

Burgess's research suggests that this policy reversal will have a major impact on pupil attainment and life chances in Wales. For pupils on the margin of achieving 5 good passes, the reform will raise them above this crucial threshold, bringing them an estimated 25% earnings increase. The policy change will also reduce educational inequality in Wales. The research showed that the greatest fall in attainment following the removal of league tables was for low-attaining pupils in deprived areas; hence the reinstatement is likely to show the greatest benefit there too, leading to greater earnings gains at the lower end of the distribution.

c. Impact of the research on the design of school performance tables

Initial discussions between Burgess and Allen and officials at the Department took place in February 2010 on reforming school league tables. Graham Stuart MP, Chair of the House of Commons Education Select Committee, requested a meeting with Burgess and Allen in November 2011.

In November 2011 new league tables were released that adopted key components of their proposal. Carole Willis states [a]: "Specifically, his work with Rebecca Allen of IOE was very influential in the re-casting of the school performance tables from 2011. Their findings on contextual value added (CVA) led directly to the Government withdrawing CVA from the tables and the wider school accountability framework. Their thinking also contributed to new measures of GCSE performance for pupils of different levels of prior attainment alongside other expert views"

The new tables included a measure showing the GCSE performance of students with differing levels of initial ability. For each school the new tables report the percentage of pupils attaining at least five A* to C grades separately for low-attaining pupils, high attainers and a middle group.

This change improves pupil attainment directly and indirectly. The new form of tables will allow parents to make a better-informed decision on where to send their child to school. Better matching between child and school will raise attainment. The new measure also gives schools more of an incentive to focus across the ability distribution, which benefits students at the top and bottom of the ability distribution as well as those near the crucial threshold, who were the focus of the previous accountability system. Higher attainment will lead to higher incomes and better life chances for individuals, and at a macro level will raise the skill base of the economy.

Sources to corroborate the impact

[a] Factual statement, Director, Department for Education.

[b] Factual statement, Ministerial Policy Advisor, Department for Education.

[c] Education in the 21^st Century: International Experiences, 1-2 July 2013, Madrid.
http://www.uimp.es/uimp/home/homeUIMPdina.php?jcj=ACTIVIDADES_ACADEMICAS&juj=3002&lan=es&jpj=plan=61WW&any=2013-14&verasi=N&lan=es&tipo=ACA¶metros=actividad=$oficial=S$sede=Santander$tipoact=$fecini=01/07/2013$fecfin=02/07/2013$director=$conferenciante=$areas=$busqueda=1$mostrar=25$pagina=1$encurso=

[d] Wales Online article by Gareth Evans, 3 November 2010.
http://www.walesonline.co.uk/news/wales-news/school-league-tables-abolishment-hitting-1883571

[e] Written Statement by the Welsh Assembly Government, 2 February 2011.
http://www.assemblywales.org/bus-home/bus-third-assembly/bus-guide-docs-pub/bus-business-documents/bus-business-documents-written-min-state/dat20110202-e1.pdf?langoption=3&ttl=Performance%20and%20Standards%20(PDF%2C%2088-1KB.

[f] Factual statement, Senior Policy Advisor, Education and Skills Department of Welsh Government.