Summary of the impact

80% of all government policies are delivered through large-scale projects and programmes. In the private and the public sector alike they are key to innovation, change, and growth. However, they often go wrong. The research has impacted on the performance of a number of projects by changing the way projects are planned, managed, and assured. The impact is the result of the research programme of the BT Centre for Major Programme Management (BTC), a research centre of the Saïd Business School. The research has had an impact on a wide range of management and policy issues in the UK and internationally. This case study highlights three examples. The first is impact on the UK government's assessment of projects through work with the National Audit Office (NAO). The second is innovation of professional services at McKinsey & Company. The third is impact on the largest infrastructure investment in the developed world - the California High Speed Rail project.

Underpinning research

Research team at Oxford includes:

• Prof. Bent Flyvbjerg, BT Professor and Chair of Major Programme Management — Principal Investigator, at Oxford since 04/2009
• Dr. Chantal Cantarelli, Research Fellow, managed the research stream on transport infrastructure, at Oxford since 04/2011
• Alexander Budzier, Doctoral student (2009- to present) — managed the research stream on ICT, at Oxford since 09/2009
• Dr. Atif Ansar, Research Fellow, research assistance on transport infrastructure stream, at Oxford since 09/2006
• Dr. Allison Stewart, Doctoral student (2009-2012), research assistance on ICT and transport infrastructure stream, at Oxford 09/2008-12/2012.

The research team's work aimed to better understand and subsequently innovate in the way megaprojects and major programs are delivered. Megaprojects are transformative, complex, and time-bound undertakings, with an investment of at least USD 1 billion. They can be found in all industries and focus on delivering organisational change, introducing technologies, or building physical assets. Innovations in the last 70 years have, however, not contributed to a significant improvement in performance as cost overruns, schedule delays, and benefit shortfalls are still widespread. Therefore the research program aimed to further understanding of the importance and impact of the under-researched front-end management of these megaprojects.

The research program was designed in multiple streams to use the same methodology to systematically build large comparable datasets of project performance across transport infrastructure, and information and communication technology (ICT). The unique size of the datasets has enabled empirical comparisons that previously were not possible. Furthermore, the data were used to test theories on biases in decision-making, principal-agent relationships, and complexity management. Findings have been fed back into policy and practice via multiple publications, conference presentations, teaching, consultancy, and media outreach.

The findings from the first research stream, transport infrastructure, explored the determinants of budget overruns, schedule delays, and benefit shortfalls. The research project built a database of 1,493 infrastructure projects from 34 different countries, dating back 90 years. The research focussed on structural and temporal determinants of optimism bias and strategic misrepresentation, which are both root causes of cost overruns, schedule delays, and benefits shortfalls in megaprojects. Prior research argued that a key to addressing the root causes of these is to shift from an inside view of projects to the outside view, i.e. to replace subjective judgements with objective empirical data. This research developed an eight-step theoretical process model to conduct quality control and due diligence of decision making on megaprojects. The process was tested on a real-life example, which has been documented in a case study. This research has been published in high ranking articles, among others [Section 3: R1, R2, R3, R4].

The second, more recent, stream of the research, ICT, challenged prevailing assumptions about social and technical complexity as a factor influencing project performance. The research stream collected a global sample of nearly 4,300 projects from 189 public and private sector organisations. The key finding showed that in the performance of ICT projects, outliers matter more than for the typical project. The data showed that the rate of outliers is higher in ICT projects than in any other field of project management. Furthermore the research found that the social complexity of projects dominates the technical complexity. Moreover it showed that a particular indicator of risk is the project's length but not its size. Lastly, the research stream identified early indicators that might help with catching high-risk projects. Preliminary results of this research have only been published very recently as a working paper, conference paper, and in the Harvard Business Review, which featured the work as the No. 1 Idea to Watch [R5, R6]. This research, while still at an early stage, has achieved significant practical impact through pre-publications and conference presentations to academic and practitioner audiences. The practical impact, in turn, generated valuable feedback shaping and enriching the research. The research has been presented at peer-reviewed academic conferences and with the combined feedback is currently being prepared for submission to academic journals, for example MIS Quarterly.

References to the research

British Telecom funded the research through the endowment of the chair of the BT Professor for Major Programme Management and a very substantial 4 year research grant.

Details of the impact

The BTC has had an impact on a wide range of management and policy issues related to megaprojects and major programmes in the UK and internationally. This case study highlights three examples.

Improving Megaproject Decision Making in the UK Public Sector
The UK government concurrently manages circa 200 megaprojects, worth more than GBP 350 billion. BTC research has directly changed the way these investments are prioritized and assessed through working in partnership with the UK's National Audit Office (NAO). Part of the NAO's remit includes value-for-money studies, which look at how projects are implemented and how projects can be improved. This joint interest between the NAO and the BTC led to an active collaboration. In 2012, BTC and NAO held multiple full-day workshops in Oxford and London to share new methods, research findings [e.g., R1, R2, R5], and collected data. In March-April 2012, the BTC wrote a research report for the NAO to identify measures of project complexity with the aim of identifying high-risk projects. The report was based on three years of research into project complexity and early risk indicators [Section 5: C1]. 10 factors were identified, showing that risks are mostly associated with the social and political environment of projects [C1]. In September 2012, the research was included in an internal report entitled `Designing successful projects and programmes'. The report was combined with concurrent work by the Treasury's Infrastructure UK group (IUK) to form the revised complexity assessment of government projects [C2, C3]. In January 2013, the new complexity assessment was rolled out as part of IUK's route map toolkit and applied to IUK's pipeline of 576 projects worth GBP 300 billion. The Director responsible for this work at the NAO, said: "The global research [you] have been undertaking ... has proved an invaluable source of evidence for the NAO ... Your analysis has made a substantial contribution to the evidence base underpinning our research into the factors driving the success, or otherwise, of major projects. An example of the impact which this work [has had is] our environmental complexity analytic [which] forms the front end of the IUK Route map published earlier this year. Your work — and importantly our discussions — was an important contributory evidence source." [C4].

Achieving Impact through a Trusted Mediator
In 2009, the BTC was approached by McKinsey to begin a collaboration to widen the reach, and transfer the research, in particular [e.g., R1, R2] into practice. From 2010-2012, BTC held multiple full-day workshops with McKinsey in Oxford. The impact was two-fold. First, the research allowed McKinsey to re-conceptualise their way of thinking about project management and subsequently changed McKinsey's methodology in the light of the advocated reframing. In turn the impact provided the data sources as well as relevant research questions that lead to [R5] and [R6]. The McKinsey partners, who were involved, wrote that the "[McKinsey-Oxford] surveys of IT executives indicate that the key to success lies in mastering four broad dimensions, which, combined, make up [McKinsey's] methodology for large-scale IT projects that we call `value assurance'" [C5]. The BTC research not only directly shaped the analytical tools of McKinsey's value assurance framework but also identified and prioritized the dimensions for which practical solutions to project problems are lacking. Subsequently, McKinsey has applied their new, BTC research-based, `value assurance' methodology to advise more than 30 global clients between 2012 and 2013. The cooperation with McKinsey allowed the BTC research to achieve a substantially more significant impact with greater reach, by not only shaping a new consulting service, but also providing a pathway for the BTC to achieve impact through other intermediaries, similar to McKinsey [C6].

A key Partner at McKinsey wrote: "Reflecting on the research project I see three areas of impact in particular. First, the research created an empirical base to help us support the planning and decision making of our clients. So far the data from your research was used at more than 20 of our clients, who as a result have made a better informed decision. Secondly, your research has helped us in developing an assessment framework which we use to analyse the complexities a project faces and the organisational capabilities of our clients to deliver against that. Thirdly, your research has helped us to dispel commonly held assumptions about the true risk drivers of projects, for instance your insight that time is a much larger risk driver than project spend has led to an 11th hour decision in one of our clients to not go forward with a 5-year, multimillion IT project" [C7].

De-Biasing the Largest Civil Project in the Developed World
The USD 51.8bn California High Speed Rail project is the US federal government's single largest civil investment. Recent estimates total the cost of the two-phased development at approximately USD 90bn. In 2012, after 5 years of planning, the US congress requested its auditor, the Government Accountability Office (GAO), to review the final project plans. In autumn 2012, drawing on the research [R1, R2] and Flyvbjerg et al. (2003) the GAO approached the BTC for methodological advice on how to best establish an independent and unbiased view of the project. Three high-ranking GAO officials interviewed the BTC Academic Director at length. In particular, they were keen to learn more of the methodology developed by BTC for how to identify optimism bias and strategic misrepresentation in project forecasts [R4]. The impact was two-fold. First, the GAO adopted this methodology to define what it considers best practice. GAO's final report stated "By not following all best practices, there is increased risk of such things as cost overruns, missed deadlines, and unmet performance targets" [C8]. Out of five academic publications listed and used by the GAO to identify global best practice, one article [R4] and two further publications (Flyvbjerg et al. 2003, Flyvbjerg and COWI, 2004) were authored by BTC researchers [C8]. Secondly, the BTC research changed how the GAO viewed the veracity of the plans, in particular the ridership forecasts. GAO's Assistant Director, who led the project review, stated: "your work sensitized us to issues of veracity. I think this was also true for California High Speed Rail Authority and Cambridge Systematics. They seemed obliged to include low-ball ridership estimates, even though this took some of the shine off their presentation.

"Your work was one of several factors we considered in reviewing the process used to prepare ridership and revenue forecasts. It was not our intent to opine on the precise point estimates of the forecasts but rather the process used to prepare the forecasts. If a sensitivity, and an extreme downside, analysis had not been conducted we would have discussed this in the report and suggested that it be addressed in future forecasts." [C9].

In sum, the research has impacted on the performance of a number of projects by changing the way projects are planned, managed, and assured through multiple pathways of impact. The first example achieved impact through changing the planning and decision making procedures of the UK public sector through work with the National Audit Office (NAO). The second example achieved impact to several large-scale projects in the private sector through collaborating with McKinsey & Company, as a trusted mediator. The third example showed the impact on a specific project, California High Speed Rail, which made the project team consider more realistic and de-biased cost and revenue forecasts.

Sources to corroborate the impact

[C1] National Audit Office, 2012. Designing successful projects and programmes, London — unpublished report.

[C2] Infrastructure UK (IUK), 2013. Infrastructure Procurement Routemap: a Guide to Improving Delivery Capability, London: HM Treasury.

[C3] Audit Manager, UK National Audit Office — will confirm the impact on the NAO's internal thinking on how to design and execute successful projects

[C4] Director, UK National Audit Office — Held on File — confirms the impact on IUK route map

[C5] Bloch, Michael, Blumberg, Sven, and Laartz, Jürgen, 2013. Delivering large-scale IT projects on time, on budget, and on value. McKinsey on Business Technology, 6 pp.

[C6] Director, McKinsey & Company, Inc. — will confirm the impact on McKinsey's Value Assurance service line and the impact of the collaboration

[C7] Expert Principal, McKinsey & Company, Inc. — Held on File — confirms the impact on McKinsey's thinking and their clients

[C8] United States Government Accountability Office, March 2013. CALIFORNIA HIGH-SPEED PASSENGER RAIL Project Estimates Could Be Improved to Better Inform Future Decisions, GAO-13-304, Washington, D.C., 90 pp.

[C9] Assistant Director, US Government Accountability Office — Held on File — confirms the impact on California High Speed Rail