Summary of the impact

Competition authorities (CAs) such as the Office for Fair Trading (OFT) are obliged to evaluate how their activities have benefited consumers. On the basis of his prior research in this area, Stephen Davies was invited to produce a public evaluation of methods used by the OFT to estimate the direct consumer benefits resulting from its competition enforcement. This is an obligation to HM Treasury and influences the allocation of resources to, and within, OFT. His published report (2010) has led to changes in OFT's methodologies and has been influential on CAs internationally. It also feeds into how OFT conducts future interventions.

Underpinning research

Stephen Davies is a professor at the University of East Anglia where he has worked since 1981.

The relevant research concerns the quantitative evaluation of anti-trust policy — both in general, and related to specific competition cases. The research underpinnings derive from a stream of previous publications, dating back to the late 1990s: for example [1] is an article that recommends more dynamic measures of competition and [2] is one of the earliest econometric analyses of the decisions of a CA. The most recent published example in this theme [3] is a rare analysis of how a CA evaluates the difficult subject of tacit collusion, using a large database of EC mergers with potential coordinated effects.

Davies' keynote speech at the International Conference on Ex-Post Evaluation of competition, University of Mannheim (2009), drew together some of the implications of his previous research, but also included a critical technical appraisal of the methodologies most frequently used in evaluation. The speech also identified some apparent gaps and weaknesses in how CAs evaluate their own activities and attracted immediate attention from academics and policymakers, setting out a long-term agenda for future research.

Progress on this agenda was documented by the publication of a working paper co-authored with Dr Peter Ormosi, a colleague in the University's Centre for Competition Policy (CCP) [4], which: (i) developed the critique of evaluation methodologies; (ii) highlighted the need for theoretically-informed counterfactuals; (iii) highlighted the relative inattention to policy in the areas of Advocacy and Article 102; and (iv) presented a framework for measuring the benefits from deterrence and the cost of under-detection (both previously ignored in CA evaluations). The journal publication of the working paper [4] forms the basis for current projects and papers in progress in the CCP.

Against the background of this research portfolio, Davies was invited by the OFT to conduct an evaluation and give recommendations (outlined in section 4), to be made publicly available, regarding its own methodologies, used both in initial decision making and subsequent evaluations. The OFT is required to estimate, on an annual basis, the savings to consumers resulting from its interventions in the areas of cartels, abuse of market power, mergers and consumer protection. In each of these areas, policy and the Law are heavily influenced by the research findings of Industrial Economists, particularly those guided by the quantitative techniques of econometricians, for example, simulation of structural models, event studies using stock market data, and difference-in-differences, using ex-post before-and-after analysis relative to a control (the unaffected market).

In turn, this evaluation of the OFT has led to subsequent requests for advice and research on this subject from the OECD and the CAs of various other countries around the world (see below).

References to the research

Publications:
(numbers in curly braces are citations from Google Scholar, accessed 1 July 2013):

Research grants supporting this research and reflecting quality:

[A] ESRC award for c£8m (2004-14) for a Centre for Competition Policy (S. Davies, M. Hviid, B. Lyons and C. Waddams, 2004).

[B] Department of Trade and Industry (£24,000), "The Benefits from Competition: some Illustrative UK Cases" (assisted by four of his PhD students: H. Coles, M. Olczak, C. Pike and C. Wilson, 2004).

[C] Office of Fair Trading (£18,500) for the project "An evaluation of the impact upon productivity of ending resale price maintenance on books" (S. Davies, 2007).

Details of the impact

Because of Davies' established research on quantitative methodologies used in evaluation of competition, the OFT asked Davies to report on its self-evaluation methodologies. It found his report sufficiently important to publish a response, in which it states that it is implementing most of Davies' recommendations.

Davies' report, "A Review of OFT's Impact Estimation Methods" (2010) is publicly available:
http://www.oft.gov.uk/shared_oft/reports/Evaluating-OFTs-work/oft1164.pdf.

Its brief was to assess the methodologies employed by the OFT for its Annual Positive Impact report. This report is required as part of the Comprehensive Spending Review (CSR 07), under which the OFT agreed a performance target with HM Treasury (HMT) of delivering direct financial benefits to consumers of at least five times its cost to the taxpayer. The OFT estimates that it has saved consumers at least £326m per year over the period April 2008 to March 2011 (OFT, 2011), resulting from its interventions in mergers, cartels, abuse of market power and consumer protection. The OFT uses various advanced quantitative methods, e.g., econometric techniques, simulations, event studies, and difference-in-differences to measure the impacts of its many interventions. These impacts influence HMT's allocation of resources to the OFT and the allocation of resources across OFT activities.

Davies' academic research has direct implications for the OFT's evaluations. In particular, he has shown that: (i) policy interventions often have much larger (and unintended) long-term consequences; (ii) the choice of methodology is inextricably linked with what is the appropriate counterfactual; (iii) all methodologies have weaknesses which should be more explicitly acknowledged than they currently are. He makes various recommendations (pages 37-43) for improvement in OFT practices. In some cases, these concern the OFT's detailed assumptions and calibrations, for example on cartel duration and overcharge, and the costs of monopoly abuse; in others, they relate to the applicability of simulation methods and sensitivity analysis. He also argues for more flexibility in the broad methodology, for example, attaching statistical confidence intervals and assessing the likely impact of deterrence — a subject of current research interest.

The OFT's published response (OFT, 2010) states that some of Davies' recommended changes would be introduced immediately and the feasibility of others would be explored in coming months. Changes have indeed been made, as acknowledged in its current Positive Impact document: "We implemented some changes to our methods last year (such as for example revising our default price assumption when estimating impact of our competition enforcement cases). We intend to continue working on some of the other suggestions made by Professor Davies in his review — such as for example, testing our methodology to estimate impact of our consumer work over 2011-12." (OFT, 2011, p11).

Some of Davies' other recommendations have instigated further ongoing research in the OFT and will be acknowledged in future years' positive impacts.

These changes help improve existing estimates of the returns from competition policy in general and the relative contributions of different areas (e.g. mergers versus cartels).

Davies' academic research has also been influential in helping the OFT shape future policy. Its chief economist has outlined its impact on the methods that the organisation uses in evaluation saying: "He has played a particularly crucial role in respect of the OFT's evaluation programme, which he has audited for us on a regular basis, pushing us to be more innovative and ambitious. Steve Davies was commissioned by the OFT to do an independent evaluation of the impact on productivity of removing RPM (resale price maintenance) in the books industry. This evaluation has been valuable not only for the OFT's advocacy role, but also in discussions on the appropriate policy for RPM across Europe" [a], [d].

Davies' academic reputation granted him the keynote lecture at the International Conference on Ex-post Evaluation of Competition Policy, University of Mannheim (2009):
ftp://ftp.zew.de/pub/zew-docs/veranstaltungen/evalcomp2009/Programme_Evaluation2009.pdf

The 2010 the working paper of the publication in [4], prompted invitations to speak at key international conferences in Berlin (organised by the German Cartel Office, 2011) and Johannesburg (for African Competition Authorities, 2011), and to present technical seminars to national competition authorities in South Africa (2011) and Sweden (2012) [b].

In November 2011, the OECD invited Davies and Ormosi to advise them on a major new research programme "Evaluation of Competition Regimes". On the programme's launch, the OECD's chief economist circulated the 2010 working paper [4] to all participating countries [c].

Sources to corroborate the impact

The OFT's public corroborations are already referred to in the above response document and its current positive Impact:

I. OFT (2010), Response to the Review of Impact Estimation Methods, at
http://www.oft.gov.uk/shared_oft/reports/Evaluating-OFTs-work/oft1164r.pdf.
Note that the OFT publicly "thank[s] Professor Stephen Davies for the insightful points raised in his review of OFT's impact estimation methods".

II. OFT (2011), Positive impact 10/11: Consumer Benefits from OFT's Work,
http://www.oft.gov.uk/shared_oft/reports/Evaluating-OFTs-work/OFT1354.pdf.

III. A Guide to OFT's Impact Estimation Methods (Note reference to change in approach in para 3.3, 3.11 and 3.14):
http://www.oft.gov.uk/shared_oft/reports/Evaluating-OFTs-work/oft1250.pdf

IV. Written Evidence from the OFT to the House of Commons Protecting Consumers Committee mentioning a change in approach to estimating the impact of its enforcement work:
http://www.publications.parliament.uk/pa/cm201012/cmselect/cmpubacc/1468/1468we04.htm

V. Report on Evaluating the work of the Competition Council of Lithuania that directly relies on the methodology developed by S. Davies:
http://kt.gov.lt/en/info/doc/news_2011-10-06.pdf

VI. 2011/12 Annual Report from the Competition Commission in South Africa mentioning a seminar given by S. Davies relevant to develop the capacity to implement impact assessment framework, on p.113:
http://www.compcom.co.za/assets/Publications/Annual-Reports/COMPETITION-COMMISSION-AR11-12-LOW-RESWITH-HYPERLINKS.pdf

Extracts from email correspondences corroborating research impact (full text is available):

[a] Dr Amelia Fletcher (28/03/12), OFT's Chief Economist re the impact of Davies' academic research on the methods OFT uses in evaluation and when deciding whether or not to intervene in mergers, in which there are significant capacity constraints. She maintains "In its advocacy role, the OFT... still draws on work Steve (and others) did for BIS in 2004, looking at the benefits of competition through a number of case studies."

[b] Dr Simon Roberts (07/02/12), Chief Economist at the Competition Commission of South Africa corroborates the impact of Davies' work on the evaluation activities of his authority (especially with respect to cartels). He states: "The Working Paper of the Centre for Competition Policy you published together with Peter Ormosi in 2010 titled 'Assessing Competition Policy: Methodologies, Gaps and Agenda for Future Research' was very important and helpful to us in developing our own impact assessment framework at the South African Competition Commission."

[c] John Davies (no relation to Stephen Davies) (07/02/12), Chief Economist at OECD confirms the important role of the 2010 working paper [4] in helping instigate its new research programme. He explains that "Because I was familiar with Professor Davies's work in this area [...] I was delighted to find a 2010 discussion paper by Peter Ormosi and him on impact evaluation of competition policy that not only reviewed the literature and state of play very clearly, but also identified gaps and areas for future work. The thinking in this paper substantially informed our initial proposals to OECD Member States for work in this area."

[d] Natalie Timan (20/03/13), Senior Economist in the Economic Policy Team of the Office of the Chief Economist at OFT highlights the significant contribution of Davies to the work of OFT and competition policy. She writes "Steve has been influential in the decision making process for which pieces of economic research the OFT should carry forward."