Summary of the impact

Research by the Oxford Public Policy Group addresses a widespread concern among governments, international organisations, and NGOs that the pervasive use of targets and performance indicators in public service management can have important distortive effects. The group's work clarifies the different effects and shortcomings of performance indicators and develops a pioneering approach to assess their reliability. The research has been influential in generating better informed approaches to the use of indicators and targets in public service management — in the UK and by international institutions — and has shaped the wider policy and public debate.

Underpinning research

The research was conducted at the University of Oxford by the Oxford Public Policy Group, led by Professor Christopher Hood (appointed to the Gladstone Chair of Government in 2001) and by Professor Iain McLean (Official Fellow in Politics at Nuffield College since 1993). Dr Ruth Dixon joined the group in 2006 as a Postdoctoral Researcher. The work was embedded in the wider ESRC Public Services Programme, directed by Hood (2004 to 2010), who provided leadership and co-ordination for 47 ESRC-funded research projects at Oxford and across a number of UK universities.

Quantitative indicators for performance management are a central feature of modern public services. Arising out of a long-term research interest in public policy and public administration, Hood and McLean's work (with colleagues in each case) examined the scope and limits of performance management `by numbers' in public services. Building on Goodhart's Law, which was formulated in the mid-70s and which questions the value of a measure that becomes a target, their research a) clarified how the effects of performance indicators vary with the manner in which they are used, b) analysed the nature of their distortive effects, and c) developed approaches to assess their validity and reliability. The group's work in each of these areas is central to the impacts described below.

(a) Clarifying how the effects of performance indicators differ with the type of use
Research by Hood examines how the effect of performance indicators varies with the use to which they are put by policymakers. The work analyses the use of quantitative indicators as targets (threshold standards), rankings (comparative leagues), and intelligence (background information for policy and management) and demonstrates that effects associated with each of these applications do not apply to them all [Section 3: R1&R2].

(b) Analysing and demonstrating unintended effects
The second strand of the research probes the unintended effects of `management by numbers', and shows that the use of targets and performance indicators can produce distortions that are akin to those observed by analysts of socialist planned economies. Drawing on that work, Hood demonstrates that contemporary public service management systems in the UK produce threshold effects (incentives to reduce effort once performance has met the target), ratchet effects (supressing current outputs to reduce future targets), and output distortions (a focus on meeting targets to the detriment of substantive quality) for example in healthcare - waiting times and ambulance response times. The research highlights these risks and explores ways to reduce them. It also examines other implications of target-focussed systems such as credit-claiming (claiming responsibility for performance boosts that do not derive from improvements within the organisation) and negativity bias (a focus on limiting the risk of falling short of the performance standard over the pursuit of excellence) and proposes ways to mitigate these effects [R4&R5].

The practical implications of such unintended effects are demonstrated in a number of ways by the research on the Comprehensive Performance Assessment (CPA) that was applied to English local government from 2002. McLean (et al.) showed that the CPA gave rise to potentially conflicting incentives and perverse effects. Conflicting incentives arose because meeting performance thresholds in one area could lead to a loss of support in another. Perverse effects resulted because the CPA rewarded efficiency in the operation of local authorities. However, authorities in deprived parts of the country found it harder to operate efficiently (recruit staff, collect local taxes) than did authorities in wealthier areas. Thus, the CPA regime could compound disadvantage by contributing to withdrawal of grants from poor areas of the country [R6].

(c) Developing an approach to assess the validity and reliability of performance indicators
Despite their potentially negative effects, however, indicators and benchmarking remain central to evidence-based public policy processes. Hood and Dixon's work therefore developed a pioneering and systematic approach to assess the validity and reliability of performance indicators. This work on `rating the rankings' shows that it is possible to compare and assess the usefulness of ranking surveys as tools of public management. It also demonstrates a trade-off between the measurement validity of a ranking and its reliability, a possibility that had not previously been demonstrated [R3].

References to the research

Christopher Hood was selected by a rigorous process of open competition as the Director of the ESRC Public Services Programme (2004-10). The Director's programme grant of £1,131,443.46 (http://www.esrc.ac.uk/my-esrc/grants/RES-153-34-1003/read) included funding for Postdoctoral Researcher Ruth Dixon (2006-). Hood collaborated in some of his research on use of indicators in healthcare with Gwyn Bevan (LSE) and Deborah Wilson (Bristol University, but also associated with the Department of Politics and International Relations on a part-time basis 2007-2010). McLean had a project grant from the programme (£45,860.25) for the CPA study. His co- researchers were Dr Dirk Haubrich (Research Officer, 2005-2007) and Dr Roxana Gutierrez- Romero (Research Officer, 2005-2007).

The quality of the group's research is demonstrated by:

(i) The large number of citations (especially R1, R2, and R5) and the influence of [R1] as the most-read article in this journal (journal's web-page accessed 4th August 2013). http://onlinelibrary.wiley.com/journal/10.1111/%28ISSN%291467-9299

(ii) The award of the IRAS Best Article Prize for 2010 for [R6] and of the June Pallot Award for best article in IPMJ 2008 for [R3].

(iii) The award of a CBE in 2011 to Hood for services to social science.

Details of the impact

The practical applicability of the group's research findings accounts for its broad impacts on governance in the UK, and on approaches to public sector management by international institutions and other agencies. In the UK, the research was disseminated widely through the ESRC Public Services Programme, which drew the findings to the attention of senior policymakers especially the Cabinet Office and HM Treasury. Articles, briefing papers, events, the programme's website, and media coverage created visibility for the research and increased engagement with stakeholders.

(a) Domestic impacts

Informing the government's performance framework: The research findings concerning the use and effects of targets and indicators fed into the re-assessment of the government's performance framework by the Prime Minister's Delivery Unit. Hood and colleagues were invited in 2009 and 2010 to brief officials who used the insights on the use of indicators in preparing policy briefs for ministers about the 2010 Performance Framework (published in 2011 as part of the new Coalition approach to the performance management of government). The framework proposals referenced Hood's work [C1], and senior civil servants confirm that the research led politicians to question the widespread use of targets and performance indicators and informed a more nuanced understanding of how these tools can be used effectively [C2].

Contributing to the debate that led to the termination of the CPA regime for English local authorities: The group's research on the effects of the Comprehensive Performance Assessment regime deepened concerns about the large number of targets applied to local authority management. The research findings created intense policy and public debate. In an article in the Guardian's `Public' magazine (3.05.2007), McLean (et al.) published the findings and called for the CPA `to be scrapped' [C3]. The Audit Commission (the body that oversaw the CPA) responded with a defence of the CPA. However, the research, particularly the insight that the CPA methodology risked compounding socio-economic deprivation, contributed to the weight of evidence against the CPA regime, which was replaced in 2009.

(b) Shaping approaches to public sector management internationally

The World Bank Group: At the invitation of the World Bank's public sector governance team, Hood contributed to an international meeting on performance budgeting (Mexico, June 2008), attended by senior finance officials and development experts from across the Organisation for Economic Co-operation and Development and South America. Hood's contribution (by video) explained key research findings on performance indicators and encouraged pragmatism and nuance in their use (viewed 2,940 times on Youtube, as of July 2013). The research shaped the discussion, and Hood and Dixon's work was frequently referred to during the proceedings [C4]. The ensuing World Bank publication particularly highlights the research findings on the potential for performance indicators to create unintended consequences and the risks of reform overload [R2] [C5]. The World Bank uses these insights in its approach to monitoring and assessment frameworks for public sector management [C4].

Hood's research on the effective use of performance frameworks is also used in the Small Countries Financial Management Programme supported by the World Bank, the Commonwealth of Nations, and the University of Oxford [R1 to R5]. The programme addresses the special financial management challenges faced by small countries and serves senior officials of ministries of finance, financial regulatory bodies and the central banks of small countries [C6].

The Inter-American Development Bank: Like the World Bank, the Inter-American Development Bank sponsors programmes to support effective public service management. It used the research of the Public Policy Group in programmes and workshops carried out over the last few years on implementing performance frameworks in central governments. In particular, the work on the ratchet, threshold, and distortive effects of performance indicators [R4& R5] was considered in the preparation of the Bank's framework for performance management in central governments and has subsequently been taken forward [C7].

(c) Shaping the wider policy and public debate about the use of performance indicators
Under the aegis of the ESRC Public Services Programme the Public Policy Group released a range of policy briefs including Hood, C. Dixon, R and Wilson, D (2009) Managing by Numbers: the Way to Make Public Services Better? (ESRC Public Services Programme Policy Briefing and referenced in C5). Dixon, R, Hood, C and Wilson, D were asked by School Leadership Today (2010) to write Keeping Up the Standards? The Use of Targets and Rankings to Improve Performance. These articles and the underlying research informed wider professional and public debates about the use of performance indicators. In the education sector, for example, the group's research on indicators and targets was used by the European School Heads Association and the National Association of Head Teachers in preparation for consultations with government and trade unions over the revision of SATs inspection regimes in schools (2010-2012) [C8]. The group's research also impacted on the work of the 2020 Public Services Trust (a cross-party think tank 2008-10) concerned to explore the role of performance indicators in public service management in a period of fiscal austerity. The Trust published an article by Hood in its own series on the future of public services (2010), which it later re-published as an edited collection (July 2013) [C9]. In addition, the research informed the wider public debate in the media about performance indicators including in The Financial Times, Guardian Public and The Scotsman.

In sum, the research of the Oxford Public Policy Group has shaped conceptual understandings of the uses, shortcomings and reliability of targets and indicators in public sector management for a wide range of major stakeholders in the UK and in the international community. The impact has been to induce a more nuanced approach to the use of indicators and substantially greater awareness of the unintended and perverse effects of over-reliance on such data; insights that are now widely applied in performance management. The impacts of the research were leveraged by the fact that the work was embedded in the broader, Hood-directed ESRC's Public Services Research Programme under which it was conducted. As Ian Diamond, ESRC Chief Executive, stated, `the programme has done everything the ESRC had hoped for, managing to combine academic excellence and practical impact' [C10].

Sources to corroborate the impact

[C1] Former Head of the Prime Minister's Delivery Unit on their use of the research in the development of policy proposals.

[C2] Former Head of the Civil Service observes that the research was used by officials in Whitehall and influenced advice to Ministers.

[C3] Copy of Guardian's magazine for public sector professionals `Public' (3.05.2007) held on file. No longer publicly available.

[C4] Head of the Governance and Public Sector Management (GPSM) group at the World Bank confirming Hood & Dixon's contribution to the international meeting in Mexico.

[C5] Arizti, P., Brumby, J., Manning, N., Senderowitsch, R., and Thomas, T. (eds.) (2010). Results, Performance Budgeting and Trust in Government. Washington: World Bank http://siteresources.worldbank.org/INTLACREGTOPPOVANA/Resources/840442 1264801888176/Results_Performance_Budgeting_Final.pdf

[C6] SCFP Programme Director confirms use of Hood's research and the conclusions of the programme's evaluation report.

[C7] Inter-American Development Bank Institutions for Development Technical Note No. IDB-TN- 581 page 25. http://www.iadb.org/wmsfiles/products/publications/documents/38051851.pdf

[C8] President of the European School Heads Association confirms the use of Hood's work by the Association.

[C9] Christopher Hood, Reflections on Public Service Reform in a Cold Fiscal Climate. Re- published in Henry Kippin, Gerry Stoker, Simon Griffiths (ed.) Public Services a new Reform Agenda (Bloomsbury July 2013). http://www.2020publicservicestrust.org/publications/item.asp?d=2606

[C10] Ian Diamond speaking at the final ESRC Public Service Programme conference December 11, 2009. Reported in Public Services Programme newsletter number 9, Winter 2010 http://www.publicservices.ac.uk/index.php/pastevents/public-services-in-the-2010s-prosperity-austerity-and-recovery/