Improving the Understanding and Use of Performance Indicators in Public Sector Management
Submitting InstitutionUniversity of Oxford
Unit of AssessmentPolitics and International Studies
Summary Impact TypeSocietal
Research Subject Area(s)
Economics: Applied Economics
Studies In Human Society: Policy and Administration
Summary of the impact
Research by the Oxford Public Policy Group addresses a widespread concern
among governments, international organisations, and NGOs that the
pervasive use of targets and performance indicators in public service
management can have important distortive effects. The group's work
clarifies the different effects and shortcomings of performance indicators
and develops a pioneering approach to assess their reliability. The
research has been influential in generating better informed approaches to
the use of indicators and targets in public service management — in the UK
and by international institutions — and has shaped the wider policy and
The research was conducted at the University of Oxford by the Oxford
Public Policy Group, led by Professor Christopher Hood (appointed to the
Gladstone Chair of Government in 2001) and by Professor Iain McLean
(Official Fellow in Politics at Nuffield College since 1993). Dr Ruth
Dixon joined the group in 2006 as a Postdoctoral Researcher. The work was
embedded in the wider ESRC Public Services Programme, directed by Hood
(2004 to 2010), who provided leadership and co-ordination for 47
ESRC-funded research projects at Oxford and across a number of UK
Quantitative indicators for performance management are a central feature
of modern public services. Arising out of a long-term research interest in
public policy and public administration, Hood and McLean's work (with
colleagues in each case) examined the scope and limits of performance
management `by numbers' in public services. Building on Goodhart's Law,
which was formulated in the mid-70s and which questions the value of a
measure that becomes a target, their research a) clarified how the effects
of performance indicators vary with the manner in which they are used, b)
analysed the nature of their distortive effects, and c) developed
approaches to assess their validity and reliability. The group's work in
each of these areas is central to the impacts described below.
(a) Clarifying how the effects of performance indicators differ
with the type of use
Research by Hood examines how the effect of performance indicators varies
with the use to which they are put by policymakers. The work analyses the
use of quantitative indicators as targets (threshold standards), rankings
(comparative leagues), and intelligence (background information for policy
and management) and demonstrates that effects associated with each of
these applications do not apply to them all [Section 3: R1&R2].
(b) Analysing and demonstrating unintended effects
The second strand of the research probes the unintended effects of
`management by numbers', and shows that the use of targets and performance
indicators can produce distortions that are akin to those observed by
analysts of socialist planned economies. Drawing on that work, Hood
demonstrates that contemporary public service management systems in the UK
produce threshold effects (incentives to reduce effort once performance
has met the target), ratchet effects (supressing current outputs to reduce
future targets), and output distortions (a focus on meeting targets to the
detriment of substantive quality) for example in healthcare - waiting
times and ambulance response times. The research highlights these risks
and explores ways to reduce them. It also examines other implications of
target-focussed systems such as credit-claiming (claiming responsibility
for performance boosts that do not derive from improvements within the
organisation) and negativity bias (a focus on limiting the risk of falling
short of the performance standard over the pursuit of excellence) and
proposes ways to mitigate these effects [R4&R5].
The practical implications of such unintended effects are demonstrated in
a number of ways by the research on the Comprehensive Performance
Assessment (CPA) that was applied to English local government from 2002.
McLean (et al.) showed that the CPA gave rise to potentially conflicting
incentives and perverse effects. Conflicting incentives arose because
meeting performance thresholds in one area could lead to a loss of support
in another. Perverse effects resulted because the CPA rewarded efficiency
in the operation of local authorities. However, authorities in deprived
parts of the country found it harder to operate efficiently (recruit
staff, collect local taxes) than did authorities in wealthier areas. Thus,
the CPA regime could compound disadvantage by contributing to withdrawal
of grants from poor areas of the country [R6].
(c) Developing an approach to assess the validity and reliability
of performance indicators
Despite their potentially negative effects, however, indicators and
benchmarking remain central to evidence-based public policy processes.
Hood and Dixon's work therefore developed a pioneering and systematic
approach to assess the validity and reliability of performance indicators.
This work on `rating the rankings' shows that it is possible to compare
and assess the usefulness of ranking surveys as tools of public
management. It also demonstrates a trade-off between the measurement
validity of a ranking and its reliability, a possibility that had not
previously been demonstrated [R3].
References to the research
[R1] Bevan, R., G., and Hood, C., (2006) `What's Measured is What
Matters: Targets and Gaming in Healthcare in England', Public
Administration 84(3): 517-538, (Google Scholar 434 citations).
[R2] Hood, C., (2006) `Gaming in Targetworld: The Targets Approach
to Managing British Public Services' Public Administration Review, 2006
66(4): 515-520, (Google Scholar 195 citations).
[R3] Hood, C., Dixon, R., and Beeston, C., (2008) `Rating the
Rankings: Assessing International Rankings of Public Service Performance'
International Public Management Journal 11(3): 298-358.
[R4] Hood, C., and Dixon, R., (2010) `The Political Payoff
from Performance Target Systems: No- Brainer or No-Gainer?' Journal
of Public Administration Research and Theory 20(2): i281- i298.
[R5] Bevan, R., G., and Hood, C., (2006) `Health Policy — Have
Targets Improved Performance in the English NHS?' British Medical
Journal 332(7538): 419-422, (Google Scholar 184 citations).
[R6] Gutierez-Romero, R., Haubrich, D. and McLean, I. (2010) `To
What Extent Does Deprivation Affect the Performance of English Local
Authorities?' International Review of Administrative Sciences 76
Christopher Hood was selected by a rigorous process of open competition
as the Director of the ESRC Public Services Programme (2004-10). The
Director's programme grant of £1,131,443.46 (http://www.esrc.ac.uk/my-esrc/grants/RES-153-34-1003/read)
included funding for Postdoctoral Researcher Ruth Dixon (2006-). Hood
collaborated in some of his research on use of indicators in healthcare
with Gwyn Bevan (LSE) and Deborah Wilson (Bristol University, but also
associated with the Department of Politics and International Relations on
a part-time basis 2007-2010). McLean had a project grant from the
programme (£45,860.25) for the CPA study. His co- researchers were Dr Dirk
Haubrich (Research Officer, 2005-2007) and Dr Roxana Gutierrez- Romero
(Research Officer, 2005-2007).
The quality of the group's research is demonstrated by:
(i) The large number of citations (especially R1, R2, and R5) and
the influence of [R1] as the most-read article in this journal
(journal's web-page accessed 4th August 2013). http://onlinelibrary.wiley.com/journal/10.1111/%28ISSN%291467-9299
(ii) The award of the IRAS Best Article Prize for 2010 for [R6]
and of the June Pallot Award for best article in IPMJ 2008 for [R3].
(iii) The award of a CBE in 2011 to Hood for services to social science.
Details of the impact
The practical applicability of the group's research findings accounts for
its broad impacts on governance in the UK, and on approaches to public
sector management by international institutions and other agencies. In the
UK, the research was disseminated widely through the ESRC Public Services
Programme, which drew the findings to the attention of senior policymakers
especially the Cabinet Office and HM Treasury. Articles, briefing papers,
events, the programme's website, and media coverage created visibility for
the research and increased engagement with stakeholders.
(a) Domestic impacts
Informing the government's performance framework: The
research findings concerning the use and effects of targets and indicators
fed into the re-assessment of the government's performance framework by
the Prime Minister's Delivery Unit. Hood and colleagues were invited in
2009 and 2010 to brief officials who used the insights on the use of
indicators in preparing policy briefs for ministers about the 2010
Performance Framework (published in 2011 as part of the new Coalition
approach to the performance management of government). The framework
proposals referenced Hood's work [C1], and senior civil servants
confirm that the research led politicians to question the widespread use
of targets and performance indicators and informed a more nuanced
understanding of how these tools can be used effectively [C2].
Contributing to the debate that led to the termination of the CPA
regime for English local authorities: The group's research on
the effects of the Comprehensive Performance Assessment regime deepened
concerns about the large number of targets applied to local authority
management. The research findings created intense policy and public
debate. In an article in the Guardian's `Public' magazine
(3.05.2007), McLean (et al.) published the findings and called for the CPA
`to be scrapped' [C3]. The Audit Commission (the body that oversaw
the CPA) responded with a defence of the CPA. However, the research,
particularly the insight that the CPA methodology risked compounding
socio-economic deprivation, contributed to the weight of evidence against
the CPA regime, which was replaced in 2009.
(b) Shaping approaches to public sector management internationally
The World Bank Group: At the invitation of the World Bank's
public sector governance team, Hood contributed to an international
meeting on performance budgeting (Mexico, June 2008), attended by senior
finance officials and development experts from across the Organisation for
Economic Co-operation and Development and South America. Hood's
contribution (by video) explained key research findings on performance
indicators and encouraged pragmatism and nuance in their use (viewed 2,940
times on Youtube, as of July 2013). The research shaped the discussion,
and Hood and Dixon's work was frequently referred to during the
proceedings [C4]. The ensuing World Bank publication particularly
highlights the research findings on the potential for performance
indicators to create unintended consequences and the risks of reform
overload [R2] [C5]. The World Bank uses these insights in
its approach to monitoring and assessment frameworks for public sector
Hood's research on the effective use of performance frameworks is also
used in the Small Countries Financial Management Programme supported by
the World Bank, the Commonwealth of Nations, and the University of Oxford
[R1 to R5]. The programme addresses the special financial
management challenges faced by small countries and serves senior officials
of ministries of finance, financial regulatory bodies and the central
banks of small countries [C6].
The Inter-American Development Bank: Like the World
Bank, the Inter-American Development Bank sponsors programmes to support
effective public service management. It used the research of the Public
Policy Group in programmes and workshops carried out over the last few
years on implementing performance frameworks in central governments. In
particular, the work on the ratchet, threshold, and distortive effects of
performance indicators [R4& R5] was considered in the
preparation of the Bank's framework for performance management in central
governments and has subsequently been taken forward [C7].
(c) Shaping the wider policy and public debate about
the use of performance indicators
Under the aegis of the ESRC Public Services Programme the Public Policy
Group released a range of policy briefs including Hood, C. Dixon, R and
Wilson, D (2009) Managing by Numbers: the Way to Make Public Services
Better? (ESRC Public Services Programme Policy Briefing and
referenced in C5). Dixon, R, Hood, C and Wilson, D were asked by School
Leadership Today (2010) to write Keeping Up the Standards? The
Use of Targets and Rankings to Improve Performance. These articles
and the underlying research informed wider professional and public debates
about the use of performance indicators. In the education sector, for
example, the group's research on indicators and targets was used by the
European School Heads Association and the National Association of Head
Teachers in preparation for consultations with government and trade unions
over the revision of SATs inspection regimes in schools (2010-2012) [C8].
The group's research also impacted on the work of the 2020 Public Services
Trust (a cross-party think tank 2008-10) concerned to explore the role of
performance indicators in public service management in a period of fiscal
austerity. The Trust published an article by Hood in its own series on the
future of public services (2010), which it later re-published as an edited
collection (July 2013) [C9]. In addition, the research informed
the wider public debate in the media about performance indicators
including in The Financial Times, Guardian Public and The
In sum, the research of the Oxford Public Policy Group has shaped
conceptual understandings of the uses, shortcomings and reliability of
targets and indicators in public sector management for a wide range of
major stakeholders in the UK and in the international community. The
impact has been to induce a more nuanced approach to the use of indicators
and substantially greater awareness of the unintended and perverse effects
of over-reliance on such data; insights that are now widely applied in
performance management. The impacts of the research were leveraged by the
fact that the work was embedded in the broader, Hood-directed ESRC's
Public Services Research Programme under which it was conducted. As Ian
Diamond, ESRC Chief Executive, stated, `the programme has done
everything the ESRC had hoped for, managing to combine academic
excellence and practical impact' [C10].
Sources to corroborate the impact
[C1] Former Head of the Prime Minister's Delivery Unit on their
use of the research in the development of policy proposals.
[C2] Former Head of the Civil Service observes that the research
was used by officials in Whitehall and influenced advice to Ministers.
[C3] Copy of Guardian's magazine for public sector professionals `Public'
(3.05.2007) held on file. No longer publicly available.
[C4] Head of the Governance and Public Sector Management (GPSM)
group at the World Bank confirming Hood & Dixon's contribution
to the international meeting in Mexico.
[C5] Arizti, P., Brumby, J., Manning, N., Senderowitsch, R., and
Thomas, T. (eds.) (2010). Results, Performance Budgeting and Trust in
Government. Washington: World Bank http://siteresources.worldbank.org/INTLACREGTOPPOVANA/Resources/840442
[C6] SCFP Programme Director confirms use of Hood's research and
the conclusions of the programme's evaluation report.
[C7] Inter-American Development Bank Institutions for Development
Technical Note No. IDB-TN- 581 page 25. http://www.iadb.org/wmsfiles/products/publications/documents/38051851.pdf
[C8] President of the European School Heads Association confirms
the use of Hood's work by the Association.
[C9] Christopher Hood, Reflections on Public Service Reform in a
Cold Fiscal Climate. Re- published in Henry Kippin, Gerry Stoker, Simon
Griffiths (ed.) Public Services a new Reform Agenda (Bloomsbury
July 2013). http://www.2020publicservicestrust.org/publications/item.asp?d=2606
[C10] Ian Diamond speaking at the final ESRC Public Service
Programme conference December 11, 2009. Reported in Public Services
Programme newsletter number 9, Winter 2010 http://www.publicservices.ac.uk/index.php/pastevents/public-services-in-the-2010s-prosperity-austerity-and-recovery/