Summary of the impact

The research impacted on the Coalition Government's decisions to (a) discontinue the Social and Emotional Aspects of Learning (SEAL) national strategy (2011), and (b) roll out the Achievement for All (AfA) pilot programme at a national level (2011). Both of these major decisions on public policy and professional practice were underpinned by the findings of large-scale national evaluation projects.

Underpinning research

The impact is based on research that took place in Manchester between 2007 and 2011, with the first major publication in 2008. The key researchers (returned staff names in bold): Barlow (2009-date); Farrell (1991-date); Humphrey (2002-date); Kalambouka (2003-date); Lendrum (2007-date); Lennie (2003-2013); Squires (2007-date); Wigelsworth (2007-date). This case study is located in the Special Education and Additional Needs (SEAN) Thematic Programme of Research (TPR) in the UoA25 submission. The primary aim of the research was to evaluate two major policy interventions relating to vulnerable groups of learners. The studies were carried out by research teams led by Humphrey yielding funding of £1.1 million. The two projects are:

Social and Emotional Aspects of Learning (SEAL) evaluations (2007-2010): in the primary SEAL study (2007-2008) [3.1] a sample of over 600 children from 37 primary schools was used in a quasi-experimental design to assess the impact of SEAL small group work on a range of outcomes, and detailed case study work was undertaken in six schools. In the secondary SEAL study (2007-2010) [3.2, 3.4] a sample of nearly 9,000 pupils from 50 schools was used in a quasi-experimental design to assess the impact of the SEAL programme on social and emotional competence, pro-social behaviour and mental health difficulties over the first three years of secondary school. This was supplemented by detailed case study work in nine schools, with a focus on implementation. The results of these evaluations ranged from mixed [3.1] to null [3.2, 3.4], and demonstrated that the preferred `bottom-up' approach to school-level implementation was impractical.

Achievement for All (AfA) pilot evaluation (2009-2011): an assessment of the impact of the AfA pilot initiative on a range of outcomes using a sample of about 16,000 pupils with special educational needs and disabilities (SEND) took place. A comparison group of pupils attending schools that were not implementing AfA, and national comparison datasets were included. Academic attainment and attendance data, and as well as school, teacher and parent surveys were also used [3.3, 3.5]. An implementation study using interviews was used, with longitudinal case studies of 19 AfA schools, and case profiles of more than 80 individual pupils and their parents. The findings demonstrated that the AfA pilot was successful in narrowing the academic attainment gap between children with and without SEND. The findings also showed that AfA promoted better relationships between schools and parents, and signposted wider outcomes for pupils with SEND [3.3, 3.5]. The study also highlighted the key factors (in terms of school processes and practices) that facilitated or impeded changes in these various pupil outcomes.

References to the research

(AOR- Available on request)

The quality of the research is evidenced by peer review of competitive grant applications and steering group advisors, and subsequent outputs are in high quality, ISI-indexed journals (e.g. Research in Developmental Disabilities). The projects upon which the outputs are based involved the application of rigorous evaluation protocols (e.g. pre-test-post-test control group design) and analytical techniques (e.g. hierarchical linear modelling).

Key outputs:

Other relevant outputs:

Details of the impact

Context: the policy landscape prior to the publication of the research findings was one in which the SEAL programme was viewed as integral to school improvement (for example, the National Strategies website made reference to its ability to `transform' schools). By contrast, the AfA pilot was viewed as a rather costly (£31 million) experiment. The publication of the research findings challenged both of these views, where rigorous evidence demonstrates the failing of SEAL and the success of AfA.

Pathways to impact: following the approach outlined in REF3a this case study illustrates an instrumental change model based on the direct reporting of commissioned research to the UK Department for Education. A two-staged reporting process was used with written and oral feedback in interim and final reports. This enabled the transmission of findings, with the opportunity to discuss and clarify. The robust evidence base combined with expert reporting facilitated research engagement by users, particularly through supporting interpretations. This approach to research legibility ensured that the research was effectively translated and transmitted to ministers, and so impacted on decision-making with major changes to policy. Two witness statements from civil servants confirm this: "the final report was very well received by the Department and Ministers...the research on AfA was very robust but the report was written in a way that helped people turn it into practical use", and a second reports that the DfE values research relationships with higher education and "Professor Humphrey's research is a good example of this mechanism working well" [5.1].

Additional pathways included media interviews, e.g. BBC Radios 4 (2010) and 5 (2011) in relation to the SEAL research, and presentations at regional conferences hosted by the National College for School Leadership (2010) in relation to the AfA research. Wider professional engagement with schools and local authorities was established through the dissemination of project outputs, for example, Humphrey et al.'s report [3.2] has been downloaded over 10,000 times [5.6]

Reach and Significance of the Impact: in terms of reach, the government's decision to discontinue the SEAL programme affects all learners of compulsory education age in England. The decision to continue and subsequently roll-out the AfA initiative affects all learners with special educational needs and disabilities in England, an estimated 20% of the school-aged population. In terms of significance, both national policy decisions were in direct contrast to the prevailing `zeitgeist'. The ending of SEAL is a reversal of a flagship initiative intended to capitalise upon growing interest in emotional intelligence. AfA survived a major spending review (which saw £670 million cut from the education budget), and was prioritised for national roll-out at a time when the Coalition government had given a clear steer that there would be few or no centralised school improvement strategies.

Discontinuation of the SEAL programme (2011): the decision to discontinue the SEAL programme in light of the research findings has been confirmed by the Department: "the Coalition government's decision to cease directly promoting the SEAL programme in 2010 is particularly significant...The work of Humphrey and colleagues... influenced the development of Coalition government policy in this area. The Department's general policy is to give further choice to schools in the approaches they wish to use. The results from this study reinforced the Department's decision to make the use of SEAL voluntary, whilst adding to the information that schools can use in their choice of this or other approaches" [5.2]. This decision is particularly significant for several reasons. From its launch in 2005 (2007 at secondary level) SEAL quickly became a dominant orthodoxy in the English education system, adopted in up to 90% of primary schools and 70% of secondary schools by 2010 [3.2]. It was one of the government's flagship initiatives, and was promoted as a panacea for improving a range of outcomes for pupils and staff. Thus, the work of Humphrey and colleagues directly challenged this and the Government's decision to discontinue SEAL needs to be considered in this context.

The secondary SEAL national evaluation [3.2, 3.5] has been particularly influential. Indeed, statements made by the Department for Education (DfE) regarding the future of SEAL can be seen to change in response to the publication of the final report for the evaluation. In autumn 2010, the DfE told the BBC: "The Coalition is continuing with SEAL. SEAL, and what schools decide to include in PSHE lessons, are what the Department (for Education) has set out in terms of encouraging emotional intelligence" [5.4]. The secondary SEAL evaluation [3.2] was subsequently published, reporting that the programme had no impact on pupils' social and emotional skills, pro-social behaviour or mental health difficulties, and furthermore that the underlying programme theory (based on a `bottom-up' view of school-level implementation) was flawed. Following this, there was clear change in direction by the DfE. In January 2011 they stated: "We trust head teachers and teachers to decide whether schemes such as SEAL work for their school. Each school will have its own unique issues to address and it is only right that these decisions are made at a local level based on the needs of the pupils" [5.4]. By March 2011 a decision was made to discontinue the programme entirely, stating: "the lack of any overall positive impact from SEAL reinforces the need to prevent further time and resource expenditure on this project" [5.5]. Coverage of the research clearly attributed this change in policy direction to the research carried out at Manchester: "Neil Humphrey's research seemed to have hit home" [5.4].

Since the discontinuation of SEAL, the research has impacted on approaches to professional practice: Hale, Coleman and Layard (2011) published a model for the delivery of Personal, Social and Health Education (PSHE) in secondary schools. The document cites the Manchester evaluation [3.2], including direct quotes focusing on the need to provide guidance for the adoption of evidence-based practice in social and emotional learning in schools [5.5]. Their model, "based around this call for guidance" (p.8), developed into a collaborative project with the How To Thrive organization funded by the Education Endowment Foundation that is currently being trialed. In regard to this one of the authors states that "Professor Humphrey's work was instrumental in helping develop a framework for the identification of evidence-based programmes" and he goes on to say "Professor Humphrey's conclusions about SEAL helped us to identify potential pitfalls in the development of our own programme" [5.5]. Furthermore, in October 2012, Humphrey was consulted by the senior adviser to Ed Miliband MP (Leader of Labour Party) in relation to the development of the Opposition's policy proposals in this area, and impact has been acknowledged: "Professor Humphrey's research and advice has made and is making a very important contribution to our thinking in these areas — including through a major speech by the Leader of the Opposition in October 2012, through the work of our Taskforce on Mental Health in Society (established in November 2012), and through the Labour Party's wider policy review activity over the last year in the areas of curriculum reform, child well-being, mental health and public health. In summary, Professor Humphrey's work is having a significant impact on the Labour Party's thinking and policymaking, and also — through our interventions on these issues — on wider public and policy debates more generally, both within Parliament and civil society. We will certainly be following his ongoing and future research in these areas with great interest, and hope to draw on his expertise again in future" [5.7].

Continuation (2010) and national roll-out (2011) of the AfA programme: the government's decision to continue and subsequently roll out the AfA programme at a national level in light of Humphrey and his team's research findings [3.3, 3.5] has been confirmed: "the research evidence was so strong in the interim report, backed up by what the SEN and schools sector were saying, that it was clear that rolling the programme out nationally was the right way forward" [5.1]. This is significant for a number of reasons. First, AfA was one of many initiatives instigated by New Labour (1997-2010) that came under close scrutiny when the Coalition government took office in 2010. Second, the Coalition government gave a clear steer that they would not be continuing the pattern of centralized approaches to school improvement established by New Labour. That the AfA pilot survived the Coalition's initial spending review (which saw £670 million cut from the education budget) and was subsequently rolled out nationally needs to be seen in this context. The role played by evidence gathered by the research team is recognized by government [5.1] and the AfA organization [5.3]. A civil servant has confirmed, "the outcome of the research has challenged long held perceptions on pupils with SEN at policy and practice level... It also led to Ministers committing to grow the AfA programme along with continued government funding. The AfA approach is now being adopted in other countries too" [5.1]. AfA is included as an example of effective practice in the SEND Green Paper, Support and Aspiration: A New Approach to SEND [5.8], which directly references the first interim evaluation report (p.44). Statements made by the DfE since then clearly highlight the impact of the national evaluation in their decision to bring AfA to scale nationally. For example, commenting on the findings of the final evaluation report [3.3], the former Children's Minister Sarah Teather said, "We are able to provide £14 million to help fund the roll-out of Achievement for All. Schools can now see for themselves the evidence that this programme works" (DfE Press Notice, 27/11/11) [5.9]. A letter from the current DfE team states: "The evidence gathered through the research helped provide a business case for expanding the programme beyond the pilot, despite the change of government" [5.1].

Brian Lamb OBE, chair of the 3As Achievement for All charity that is supporting schools in the roll out process, said, "For the first time we have a proven intervention that can improve outcomes" (DfE Press Notice, 27/11/11) [5.9]. This has been supported by the Founder and CEO of AfA, who identifies the impact of the research on public policy: "The independent evaluation, carried out by the University of Manchester and Professor Humphrey's team, provided important evidence that children with special educational needs and disabilities (SEND) had made remarkable progress in attainment, behaviour and attendance under a Government-funded pilot. As a result of this research, the Government provided £14 million to help fund the roll-out of the Achievement for All programme nationally" [5.3]. The impact on professionals is also confirmed: "from discussion with my colleagues I am aware that circa 2000 schools take into account Professor Humphrey's work in their decision making process or in the development of provision" [5.3]. We have evidence from professionals about this impact on practice and learning: "I have been involved in implementing a great deal of the AfA findings... and I can confirm that policy and practice were directly affected by the outcomes of Professor Humphrey's work" (Deputy Headteacher); and "I have been involved in AfA since the beginning of the pilot project and I can confirm that policy and practice were directly affected by the areas outlined in Professor Humphrey's research" (SENCO) [5.10].

Sources to corroborate the impact

All claims referenced in the text.

[5.1] Two letters from senior civil servants at the DfE regarding AfA.

[5.2] One letter from a senior civil servant at the DfE regarding SEAL.

[5.3] Letter from the Founder and CEO of AfA.

[5.4] BBC Radio 4, Analysis: Testing the Emotions, 07/03/11 Transcript available at:
http://news.bbc.co.uk/nol/shared/spl/hi/programmes/analysis/transcripts/07_03_112.txt

[5.5] Hale, D., Coleman, J. and Layard, R. (2011). A model for the delivery of evidence-based PSHE (personal wellbeing) in secondary schools (CEP Discussion Paper No. 1071). London: LSE. Plus a letter from one of the authors updating the impact of the research.

[5.6] www.education.gov.uk/researchandstatistics.

[5.7] Letter from the Senior Policy Adviser to the Leader of the Opposition.

[5.8] Department for Education (2011). Support and aspiration: A new approach to special educational needs and disability. London: DFE.

[5.9] DFE Press Notice, 27/11/11, available at: https://www.gov.uk/government/news/sen-children-make-big-step-forward-in-english-and-maths-according-to-new-report

[5.10] Two letters from school practitioners: a SENCO and a Deputy Headteacher.