Summary of the impact

Improving the quality of school leadership has been a key priority for both New Labour and Coalition education policy (see for example `The Importance of Teaching' White paper) since 1997. This led to the establishment of the National College for School Leadership (NC) in 2000. Between 2004 and 2009 the NC commissioned six external evaluations of its programmes from Sheffield Hallam University (total value: £276k). The studies have impacted on the range and quality of the College's provision and hence have on the quality of leadership in schools by enabling the NC to:

• take decisions about programme continuation and development by assessing the effectiveness of programmes: their contribution to school improvement and value for money;
• redesign specific programmes to meet system and individual leadership development needs by enabling the NC to understand factors influencing programme outcomes
• take strategic decisions about its portfolio of courses and patterns of delivery by providing a range of evaluation studies that placed individual programmes within the broader context of leadership development needs and provision.

Underpinning research

The studies involved a range of research methods and the scale of work (2,000 survey responses; 200 telephone interviews with participants and others; school case studies), as well as its cumulative nature, provided a robust basis for findings. Prior to the research reported here, there was a considerable literature on leadership development and its effectiveness, but detailed work on impact and the factors that influenced this was patchy (see review article R7). This research complemented and took forward this work in a number of ways.

First, the studies indicated the overall effectiveness of NC programmes in terms of aggregated outcomes (G1, G2, G3, R2), although this was differentiated in relation to a number of key variables associated with participant and school characteristics (see below). In relation to the Multi- Agency Team Development programme (MATD) (the only programme that was deemed unsuccessful and was discontinued soon after our evaluation), our three studies, while positively evaluating programme design and delivery, contributed primarily to understanding the challenging policy environment of multi-agency team working, identifying issues relating to team formation and characteristics, local authority policies, structures and procedures and programme cost, all of which could impede the success of such programme (G4, R6).

Secondly, the studies contributed to understanding factors influencing the effectiveness of leadership development activities. This understanding extended previous work in a number of ways. First, the ways in which participants were selected and their consequent motivations were significant factors in determining the nature of their engagement with programmes and consequently the benefits they obtained from them. Key variables here included the degree to which participants' motivations were instrumental or developmental and the degree to which the programmes were perceived to match participants' level of experience (R2). Second, the role of the participant's sponsor (school or local authority) was critical in determining programme effectiveness. The studies showed that schools engaged in different ways, with some adopting a more strategic approach (which the studies modelled), while others were more opportunistic (R3, R4). In contrast, for Local Authorities (LAs), in relation to MATD, there were deeper strategic issues concerning ways in which responsibilities for multi-agency working were located structurally and culturally (R6). Third, the role of the coach in the school-focused programmes was seen as critical, with the coaching process varying considerably (both in approach and quality) amongst schools. A model for classifying coaching processes was developed (R1). Finally, the studies explored the blended learning designs embodied in the programmes, drawing conclusions about differences between the impact of face-to-face and online components and factors that influenced this (R4). A commissioned study of the design process for Leadership Pathways (G2a) analysed the concept of `personalisation' which underpinned later developments in College programmes, identifying in particular four key factors that needed to be embodied in programme design: challenge, contextual relevance, motivational ability (especially in relation to participants' learning preferences), and relationship to changed leadership behaviour.

Third, as the research findings accumulated, and the above factors were identified, we developed a model for analysing the impact of professional development activities in general, and leadership development activities in particular. This model was refined over a number of studies and provided a robust basis for understanding not just whether impact had occurred but the key variables determining this (R2, R5). It built on and modified earlier work by writers such as Kirkpatrick and Guskey in developing a more detailed specification of key variables and relationships with particular emphasis on variables external to the programme in the school and the wider environment as well as those internal to programmes.

The studies were led by Professor Tim Simkins, Professor of Education Management, with other main contributions made by Paul Close, Senior Lecturer in Education Leadership Mike Coldwell, Head of Centre for Education and Inclusion Research and Ros Garrick, Principal Lecturer in Early Years Education. All were employed at Sheffield Hallam University throughout the REF assessment period.

References to the research

Key refereed papers

All these papers were published in peer reviewed journals. R4, R5 and R7 are included in Simkins outputs in REF1; R1 was included in his 2006 RAE return.

Research Grants (all from National College for School Leadership) (total value £276k)

G1. Evaluations of Leading from the Middle (LftM) (Value £75k)

a. Evaluation of Cohort 1. Final Report date 2004

b. Evaluation of Cohort 3: 2004-05.

G2. Evaluations of Leadership Pathways (LP) (Value £90k)

a. Design Study. Final report date 2006.

b. Evaluation of Pilot. Final Report date 2007.

c. Evaluation of Roll-Out. Final report date 2008.

G3. A comparative study of the impact of the in-school components of three of the College's core programmes (LftM; National Professional Qualification for Headship [NPQH]; and Leadership Programme for Serving Heads [LPSH]). Final Report date 2006. (Value £60k)

G4. Evaluations of the Multi-Agency Team Development Programme (MATD) (Value £51k)

a. Follow-up study of pilot participants. Final report date 2009.

b. Evaluation of roll-out. Final report date 2009.

c. Study of programme marketing. Final report date 2009.

Following standard NC practice, reports from these studies are not in the public domain, but they can be provided to the panel on request.

Details of the impact

The research insights achieved impact through their influence on the policies and programmes of the NC. Although this impact is limited to one organisation, the potential range and scope of indirect second-level impact on school leaders can be gauged from the size of the NC and its programmes. By 2008/09 participants on the four core programmes we evaluated (LftM; LP; NPQH; LPSH) numbered more than 10,000, one of the largest bodies of sustained leadership development activity for education in the world.

Overall impact
The research described here had a broad impact across the leadership development work of the NC for a number of reasons: the design of each study, the conclusions reached as well as underpinning conceptualisations and theorisations contributed to the accumulation of knowledge and understanding about the impact of the NC's leadership development provision; our approach to evaluating impact, refined over a number of studies, provided a robust basis for understanding not just whether impact had occurred, but key variables determining this; and the number and scale of the studies provided a robust evidence base. Although a number of the studies were undertaken prior to the REF period, these factors meant that their impact was cumulative at least until 2011. Key NC contacts emphasised the quality and robustness of the studies and the various study teams' 'ability to provide timely feedback and enter into dialogue' on issues that were key to programme development. This was done through meetings during which interim results were presented and discussed and policy implications drawn out. These strengths were reflected in the amount of repeat commissions in a competitive environment and the development of a sustained relationship between SHU and the NC that still continues. A NC informant stated `I would wish to note the high degree of professionalism, insight and objectiveness of the Sheffield Hallam research teams. The way they interacted with College on an on-going basis was very important in ensuring that the findings from the research impacted on the College's leadership design And development work in a timely way' (S2).

Specific impact: programme design and development
Evidence about programme effectiveness and the factors that influenced programme outcomes impacted on NC provision in a number of ways. In terms of the portfolio of provision, our earlier reviews of the NC's programmes (G1) led to the decision to develop a new programme - Leadership Pathways (LP - piloted in 2006 and rolled out in 2007). This was designed: (i) to meet the needs of more experienced leaders than LftM; (ii) to place a much greater emphasis on on-line materials and support; and (iii) to be delivered on a commissioned basis rather than directly by the College. SHU was closely engaged with the NC in the implementation of LP as it evolved over the period 2006-2009. We were asked to undertake a study of the design process in 2006, looking in particular at ways in which `personalisation' was embodied in the programme and the implications of the commissioning strategy. Following implementation, SHU evaluated the LP Pilot and Roll-Out in 2007 and 2008. These studies together influenced evolving programme design as it developed from 2008 in a number of ways, including informing: i) the move towards a modular curriculum; ii) developments in the blended learning approach; iii) the evolution of more effective coaching models and practices. These outcomes drew on analyses derived from an understanding of key variables in programme design described above.

The evaluations of the MATD Programme (2009) contributed to a rather different scenario. The programme was not successful: our studies demonstrated that the reasons lay in misunderstanding of the market rather than programme content or design. The studies contributed to the decision to discontinue the programme in 2011. A key contact in the NC (S3) stated that the programme would have continued longer without the evaluation's conclusions about the reasons for its under-recruitment - it made them `bite the bullet' of discontinuation. Beyond this, however, the studies informed the College's general thinking about how to support leadership development across a range of service contexts, such as through its programme for Directors of Children's Services, developing `a more mature way of operating across a group of agencies' as a key NC informant put it (S1).

Specific impact: strategic thinking
Contributions to programme design and delivery described above were complemented by more in- depth contributions to the NC's understanding of factors impacting on leadership development outcomes, and, as the scale of College activity expanded, to its strategic thinking as it moved, in 2011, from a centralised model of leadership development to one where consortia of schools and others were licensed to provide leadership development on a commissioned basis within a centrally designed curriculum framework.

First, as part of the process of moving towards a licensing approach to programme delivery the College developed a leadership development framework comprising five levels from middle leader to experienced headteacher, embodied in a modular curriculum. The research informed this framework, both through `distilling understanding of the signature characteristics of good leadership development' as one of our College informants put it (S1), and, more specifically, by feeding into the design of three levels of the programme which evolved from previous programmes that we had evaluated (Leading from the Middle [G1, G3], Leadership Pathways [G2], National Professional Qualification for Headship [G3]). Another informant stated: `Through detailed evaluations of existing provision and of the context within which programmes had to be delivered, the studies informed our move towards the development, in 2011, of a more modular curriculum supported by a blended learning approach' (S2).

Second, it became increasingly clear that schools were key players in the design and delivery of programmes, through in-school projects of various kinds and the provision of coach support. However, research conducted by SHU indicated the challenges that school-based delivery involved, with some evidence indicating that the quality of support provided by schools varied considerably (G1, G2). It was recognised that little was known about the school as a site of leadership development. Consequently, SHU was commissioned to undertake a comparative study of in-school components of three major core programmes: Leading from the Middle, the National Professional Qualification for Headship and the Leadership programme for Serving Headteachers (G3). This study identified similarities and differences between the three programmes' in-school components, and contributed to the implementation of the NC's involvement in the broader national policy shift towards a school-led system. For example, the evolution of the NC's approach to coaching was underpinned by the availability of detailed information provided by our studies about what was actually happening in schools. This highlighted the key issue of coaching capacity, factors that affected this and how it needed to be developed. As one informant put it: `It helped us to see a picture of what was actually happening in schools' and `that there had often been more rhetoric than good practice' (S4). These understandings contributed to the NC's evolving strategy to support coaching and coaches. A NC informant summarised the overall impact of our work on NC strategy as follows: `More generally - and of particular significance - the range of research done by SHU in relation to the in-school aspect of leadership development helped provide the basis for a major shift of strategic focus towards a more school-based and commissioned provision' (S2).

Sources to corroborate the impact

S1. Former Director of Evaluation and Performance

S2. Managing Director of International Unit

S3. Former Research and Evaluation Officer

S4. Former Research and Evaluation Officer
(All from the National College of School Leadership)