Case Study 3: International benchmarking and econometric analysis used to set efficiency targets for Network Rail
Submitting InstitutionUniversity of Leeds
Unit of AssessmentCivil and Construction Engineering
Summary Impact TypeEconomic
Research Subject Area(s)
Economics: Applied Economics
Studies In Human Society: Policy and Administration
Summary of the impact
Reductions in railway infrastructure and operating costs, through
efficiency gains, deliver benefits to taxpayers (via lower subsidies)
and/or passengers (via lower fares). Research undertaken by the Institute
for Transport Studies (ITS) at the University of Leeds from 2005 onwards
revealed a 37% efficiency gap in relation to rail infrastructure costs and
operations, relative to international best practice. The key impact of
this research was to inform the Office of Rail Regulation's (ORR) setting,
in 2008, of annual efficiency targets for Network Rail for the subsequent
five-year period, resulting in a reduction in costs from £18.2bn to
£15.8bn over the five year regulatory `control period' starting 2009/10. A
secondary impact of the ITS Leeds research was to provide key benchmarking
and evidence in more recent ORR efficiency assessments (2010) and Sir Roy
McNulty's long-term policy-setting Rail Value for Money (VfM) study
(2011). Extending the reach of these research impacts, the water and
sewerage regulator OFWAT has, from 2013, adopted the ITS Leeds approach
for its latest periodic review.
A key responsibility of the Office of Rail Regulation (ORR) is to
determine the efficient cost of operating, maintaining, renewing and
developing the British rail infrastructure. The determination of these
costs, referred to as the periodic review, occurs every five years
and, given the structure of the rail industry in Britain, effectively
amounts to a review of the costs incurred by Network Rail. In contrast to
other regulated utilities, previous periodic reviews (prior to 2008) did
not assess the efficiency of Network Rail against external benchmarks.
Recognising this gap in knowledge, ITS Leeds was contracted in 2005 by ORR
to develop two international benchmarking studies.
Econometric modelling — efficiency studies
Between 2005 and 2008, researchers from ITS Leeds carried out new
research funded by the ORR to investigate the cost and efficiency of the
rail system in Britain relative to its international peers. Employing
advanced time-varying efficiency modelling techniques, Dr Andrew Smith
identified drivers of rail infrastructure costs, and the relative
efficiency of rail infrastructure managers (IMs) around the world [1; iii]
using a unique panel dataset of 13 European rail infrastructure managers
collected by the International Union of Railways. This work exposed a
substantial efficiency gap between Network Rail and its international
peers of around 37%.
As part of this work, Smith and his colleague Phill Wheat also adapted
stochastic frontier techniques to deal with `sub-company' datasets (where
firms are sub-divided into several business units). This new approach
allowed inefficiency to be decomposed into a fixed element that applies
across the entire company (external inefficiency) and an element that
varies at the sub-company level (internal inefficiency). The `sub-company'
analysis revealed large variations in performance and practice between
regional units within Network Rail [2; iii], as well as a substantial
efficiency gap between Network Rail and its international peers.
In parallel to the above research, Wheat and Smith developed a new
international collaboration with Professor William Greene at NYU Stern
School of Business to develop statistical methods for assessing the
uncertainties associated with the estimates of inefficiency generated by
stochastic frontier methods [3; i, iii].
Econometric modelling — evaluation of policy reforms
The research referred to above is part of a larger body of rail cost
research on whole train systems (i.e. train operations as well as rail
infrastructure) conducted by ITS Leeds over many years, supported by
funding from EPSRC [i], DfT [ii] and ORR [iii]. Since 1993, ITS
researchers have carried out econometric and wider policy evaluation work
to assess the impact of competitive tendering/franchising on costs.
Introducing competition into passenger rail, via competitive tendering,
forms an important part of the European Commission's proposals for further
reform of the European rail sector. This strand of research was the first
in the literature to demonstrate the sharp rise in train operating company
costs in Britain (previously the policy focus had been on rising
infrastructure costs); see for example . Reference  found that a key
reason behind rising train operating company costs in Britain was the
franchising body's decision to place operators in financial distress on
"cost-plus" type management contracts for extended periods.
Research team with current title/grade and period of employment
Professor Chris Nash (Lecturer, 1975-88; Senior Lecturer, 1988-89;
Professor, 1989-date) Dr Andrew Smith (Lecturer, 2004-11; Senior Lecturer,
2011-date) Phill Wheat (Research Officer, 2005-08; Research Fellow,
2008-10; Senior Research Fellow, 2010-date)
Key grants 1998-2013
i. Rail Research UK, several grants (e.g. RG.TRAN.471326, PI Nash,
RG.TRAN.481002, PI Smith), worth approximately £200k in total.
ii. Department for Transport (RG.TRAN.477546, PI Smith), £38k.
iii. ORR funding, numerous grants (e.g. RG.TRAN.471451, RG.TRAN.473416,
RG.TRAN.475450, RG.TRAN.477940, RG.TRAN.478240, RG.TRAN.478541,
RG.TRAN.482896, RG.TRAN.474920), worth approximately £250k in total. PI Smith.
iv. OFWAT funding, three grants (e.g. RG.TRAN.484975, RG.TRAN.101259. PI
Smith, worth approximately £49k in total.
Note: The repeat funding from Rail Research UK and ORR is indicative of
the quality of the Leeds research and its ability to meet project
objectives, as is the new work for OFWAT following on from the successful
research carried out for ORR. It should be noted that the ORR contracts
were awarded following a competitive tendering process.
References to the research
1. Smith, A.S.J. (2012) `The application of stochastic frontier
panel models in economic regulation: experience from the European rail
sector'. Transportation Research Part E, 48, pp503-515.
The above output is based on an earlier project report produced for ORR
and published on their website: Smith, A.S.J (2008) International
Benchmarking of Network Rail's Maintenance and Renewal Costs: An
Econometric Study Based on the LICB Dataset (1996-2006), Report
for the Office of Rail Regulation, October 2008. http://www.rail-reg.gov.uk/upload/pdf/pr08-itslicb-301008_20081117141529.pdf
2. Smith, A.S.J. and Wheat, P.E. (2012) `Estimation of
cost inefficiency in panel data models with firm specific and
sub-company specific effects'. Journal of Productivity Analysis,
37, pp27-40. doi: 10.1007/s11123-011-0220-8.
The above output is based on an earlier project presentation produced
for ORR (co-authored with ORR) and published on their website: Smith,
Wheat, and Nixon (2008) International Benchmarking of Network
Rail's Maintenance and Renewals Costs. http://www.rail-reg.gov.uk/upload/pdf/pr08-its_010608.pdf
3. Wheat, P.E., Greene, W, and Smith, A.S.J. (2013)
`Understanding prediction intervals for firm specific inefficiency
scores from parametric stochastic frontier models'. Journal of
Productivity Analysis, in press. doi: 10.1007/s11123-013-0346-y.
4. Smith, A.S.J., Nash, C.A. and Wheat, P.E.
(2009) `Passenger rail franchising in Britain: has it been a success?' International
Journal of Transport Economics, 36 (1), pp33-62.
The McNulty report (see Section 4 below) referenced (in footnote 106,
p284) a project report produced for ORR based on the above output, as
well as an earlier conference paper version of output 4: Smith,
A., Nash, C. and Wheat, P. (2009) `The effect of
passenger rail franchising on costs: evidence from the passenger rail
sector in Britain (1996 to 2009)'.
5. Smith, A.S.J. and Wheat, P.E. (2012) `Evaluating
alternative policy responses to franchise failure: evidence from the
passenger rail sector in Britain'. Journal of Transport Economics
and Policy, 46 (1), pp25-49.
The McNulty report (see Section 4 below) referenced (in footnote 25,
page 34 and footnote 108, p285) an earlier conference paper version of
output : Smith, A.S.J. and Wheat, P.E. (2009) `The
effect of franchising on cost efficiency: evidence from the passenger
rail sector in Britain'.
Note: All ITS Leeds researchers are in bold. References , 
and  should be assessed for research quality. All references are
published in international journals with rigorous peer review processes.
Details of the impact
An undesirable consequence of rail reform and privatisation in Britain
has been sharp increases in both rail infrastructure costs (from roughly
£15bn during the first five years after privatisation to £30bn during the
second five year period) and train operating company unit costs (up by
around 15% since privatisation). The ITS Leeds research commissioned by
ORR has enabled the regulator to benchmark the efficiency of Network Rail,
by providing the key evidence for the regulator to set efficiency targets.
The efficiency improvements delivered by Network Rail have directly
benefited taxpayers via lower subsidies (although these benefits could
instead be directed to passengers through lower fares).
Economic impacts — improving the efficiency of rail infrastructure
The first of these studies  was used by ORR as the central piece of
evidence on Network Rail's efficiency performance in the 2008 Periodic
Review [A]. The efficiency gap, estimated at 37% by the ITS Leeds
modelling, was used directly to derive a funding settlement for Network
Rail covering the fourth `control period' (CP4; 2009/10 to 2013/14). A
quote from the ORR's final conclusions document emphasises how the model
was used: "...we [ORR] consider that it is reasonable,
transparent and prudent to adopt the 37% value from the preferred model
as our estimate for the efficiency gap for each of maintenance and
renewals in 2006-07." [A, p157]. Corroborating the role
played by ITS Leeds, ORR [A, p126] noted: "...the econometric analysis
work was undertaken by ITS working closely with both Network Rail and
The value of the efficiency targets set by ORR, based on the ITS Leeds
evidence, was to reduce the cost of maintaining and renewing Network
Rail's infrastructure from around £18.2bn to £15.8bn over the five year
regulatory control period (2009/10 to 2013/14). Note that this reduction
is smaller than 37% because ORR phased the target gradually over a ten
year period. A letter of corroboration from ORR [B] notes: "Following
the PR08 process, assisted by the ITS econometric efficiency model, ORR
deemed that the efficient cost should be only £15.8bn (in 2006-7
prices). This work thus directly influenced the determination by the ORR
of Network Rail's funding for CP4, much of which is provided by
The second study commissioned by ORR  was also an international
econometric modelling exercise, but this time expanding the sample size by
utilising sub-company (regional) data within individual railways (drawn
from a sample of participating European and North American railways). The
model from this second exercise was quoted in ORR's 2008 determination as
a supporting, corroborating study to the first study outlined above [A,
Setting standards for rail regulatory reviews
In its review of ORR's regulatory efficiency analysis, the National Audit
Office in 2011 noted that: "...the Regulator initiated an innovative
international benchmarking exercise with the Institute of Transport
Studies (University of Leeds) during its 2008 Periodic Review" [C,
Since the 2008 Periodic Review, the approach was further developed and
used in subsequent assessments; ITS Leeds modelling supported the ORR's
Annual Efficiency Update process in 2010 [D]. Furthermore, Smith and Wheat
were seconded part-time to ORR during 2012 and 2013 to advise the
regulator on its benchmarking methods for the 2013 Periodic Review; their
contribution is acknowledged in the 2013 Draft Determination [E].
Informing policy — the long-term future of Britain's rail system
The modelling work [1, 2] also formed the basis of future efficiency
targets for Network Rail (CP5; 2014/15 onwards) documented in the Sir Roy
McNulty Rail Value for Money study (which commenced in 2010 and was
published in 2011) [F]. This is arguably the most significant report on
rail costs in Britain, which has assessed efficiency and value for money
since privatisation and takes a long-term perspective on its future.
A letter of corroboration from DfT [G] notes: "The [McNulty] study
represented a major review of rail costs, efficiency and value for
money, and set out a new policy direction based on its analysis and
findings. The study identified savings for the whole industry of between
£2.5bn and £3.5bn per year (or 20-30% of total system costs), of which
the savings identified for Network Rail from the work conducted by ITS
accounted for £1.8bn per year (£1.2bn per year earmarked to be achieved
by the end of CP4 and a further £0.6bn per year by the end of CP5)".
Through implementing the recommendations of this study (and because of
growing passenger revenues), the level of subsidy paid to the railway is
forecast to fall from £4bn in 2010-11 to around £1bn by 2019.
In addition, ITS Leeds research (a project report for ORR based on 
and an earlier conference paper version of ) is also quoted by McNulty
[F] as a key piece of evidence, amongst others, in determining future
targets for train operating costs.
Changing practice in other regulated sectors
Extending the reach of the research impacts to other sectors, ITS Leeds
was commissioned (with consultants CEPA) by the water regulator OFWAT in
summer 2012, to develop new econometric models and advise on how they
could be used to set a `corridor' for allowed cost levels, whilst
complying with the OFWAT's aim to achieve a "light touch" approach to
regulation. The preliminary report [H] released in 2013 sets out the
approach developed by ITS Leeds (with CEPA). This work has fed directly
into OFWAT's 2014 regulatory review (PR14), as OFWAT has now decided to
adopt the ITS Leeds approach to econometric modelling for PR14.
Furthermore, Smith has been appointed (from August 2013) as the
regulator's academic adviser on cost assessment.
Sources to corroborate the impact
A. Office of Rail Regulation (2008) Determination of Network Rail's
Outputs and Funding for 2009- 14 (October 2008). http://www.rail-reg.gov.uk/upload/pdf/383.pdf
B. Letter of corroboration from the Deputy Director, Railway Markets and
C. National Audit Office (2011) Regulating Network Rail's Efficiency.
D. Office of Rail Regulation (2010) International cost efficiency
benchmarking of Network Rail. The main report (see p5) and supporting
paper (the latter, co-authored with ITS) can be found at, respectively:
E. Office of Rail Regulation (2013) Draft determination of Network Rail's
outputs and funding for 2014-19 (see p300). http://www.rail-reg.gov.uk/pr13/PDF/pr13-draft-determination.pdf
F. Sir Roy McNulty report (2011) Realising the Potential of GB Rail:
Final Independent Report of the Rail Value for Money Study: Detailed
report (see, for example, page 30)..
G. Letter of corroboration from the Deputy Director, Rail Analysis,
Department for Transport.
H. OFWAT reference.