Summary of the impact

Building on innovative statistical models developed at Oxford, initially conducted by Dr Clive Payne and Prof David Firth, Dr Stephen Fisher's research facilitates quick generation of predictions for the share of seats won by each party from exit poll data. As a consequence, Fisher has provided the BBC with prediction and analysis of election results from 1997 to the present.

At the 2010 General Election, the forecast was jointly commissioned by the BBC, ITV, and Sky News. This forecast reached a television audience of 17.7 million on the BBC alone. Exit poll forecasts are of benefit to television companies and journalists in providing accurate reporting on elections and in developing programmes that interest the public and help to inform them of voting patterns. This public information and engagement is beneficial to electoral politics more broadly.

Underpinning research

Oxford Sociology has a long tradition of statistical analyses of voting. Data from exit polls pose particular methodological problems, which Oxford researchers have gone some way to resolving. Many voters have no time or inclination to reveal their choice to interviewers waiting outside the polling station, generating bias when supporters of some parties are more reluctant than others. Such problems led to a famous error in the 1992 general election: exit polls conducted separately for the BBC and ITV misled forecasters into predicting a hung parliament, whereas in fact the Conservatives won by a significant margin.

This dramatic failure led Clive Payne (Research Lecturer, Nuffield College, Oxford (retired 2002)) and David Firth (Professor of Social Statistics at the University of Oxford (moved to Warwick 2003)) to develop more robust statistical models. Their key methodological insight, laid out in a path-breaking article [Section 3: R1], was that the analysis of change since the last election and a focus on differences in (rather than levels of) swing in different kinds of seats could both reduce bias and increase efficiency when using the exit poll to inform early results based forecasts. John Curtice (University of Strathclyde) and David Firth (University of Warwick since 2003) took this research a step further (2008) and demonstrated that it was preferable to measure change between two exit polls at the same polling stations in consecutive elections, cancelling out much of the bias. Stephen Fisher first became involved with this research in 1997, continuing after becoming Lecturer in Political Sociology at Oxford (in 2002) and since 2007 he has played the leading role in developing this statistical methodology.

Most recently for the 2010 election, as well as improving the speed, diagnostics and reporting functions of the forecasting process [R2, R4], Fisher developed and evaluated methods for predicting constituency victories from vote shares [R3, R4], for sampling new locations for exit polls [R4], for the selection of forecasting models [R2, R4], and for adjusting for postal voting and non-response bias [R4]. In particular, Fisher showed that probabilistic prediction of seat outcomes is relatively insensitive to distributional assumptions and tuning parameters [R3, R4], and also developed a stratification strategy and balance checking for the random selection of a very small number of new exit poll locations [R4]. While substantial numbers of electors are registered to vote by post, and so fall outside the sampling frame for the exit poll, the stability of the rates of postal voting between 2005 and 2010, together with the absence of a differential postal voting effect on change in vote choice, meant that the exit poll forecast was best left unadjusted in 2010 unlike in 2005 [R4]. This recent work shows that the accumulation of small methodological choices has a substantial aggregate effect on the forecast [R4].

Unusually in sociology, Fisher's methods have been tested under difficult time pressure and in the full glare of publicity via live international television, radio and online coverage. Accurate predictions were expected within 15 minutes of receiving the data [R2]. So the methodology developed had to be rigorously tested and capable of extremely fast implementation. Fisher's predictions proved to be remarkably accurate, demonstrating the successful deployment of this new approach to using these methods.

References to the research

NB Authors at Oxford at the time of the research are underlined.

Details of the impact

Fisher served as election forecaster for the BBC throughout the period (2008-2013). The BBC employed him for a total of seventy days to provide prediction and analysis for elections to the Westminster Parliament in 2010, the Scottish Parliament in 2011, the Welsh Assembly in 2011, the European Parliament in 2009, the Greater London Authority and London mayoralty in 2012, for the Alternative Vote referendum in 2011, and for local elections every year. As the BBC's Election Results Editor summarizes, Fisher `provides the statistical computing that produces the exit poll prediction on the day and also the constituency results based forecast overnight. His research is important not just in providing the data and figures, but also in informing the preparation work for how the exit poll should be conducted and what [can be said about it], and when it is possible to draw conclusions from it.' [C2].

The 2010 General Election is the focus of this case study. BBC, ITV, and Sky News placed such importance on forecasting the election that they unusually commissioned 18,000 interviews with voters at 130 polling sites. This must have cost well in excess of £100K, though the actual financial cost of the exercise is confidential. Based on these interviews, Fisher applied the methods developed through his research with Curtice and Kuha [R4] to produce a forecast as the polls closed at 10pm. As most of the data from exit polls arrived in the evening, this analysis had to be produced speedily, 15 minutes after the final batch of data arrived. After 10pm, the BBC deployed Fisher to provide results-based prediction and analysis.

The forecast was broadcast by all three news channels at the close of polls. The prediction that the Liberal Democrats would end up with fewer seats than in 2005 was immediately controversial. Most commentators dismissed it as implausible, given the party's momentum in the weeks before the election. Paddy Ashdown (former leader of the Liberal Democrats) referred to `fairyland figures' [C3], while Iain Dale (prominent political commentator) promised to `run naked down Whitehall if that turns out to be true' [C4]. These quotations reveal how strongly entrenched mistaken opinion-poll-based expectations were, and so how shocking the exit poll prediction was. The remarkable accuracy of the exit poll prediction was fully clear approaching breakfast time the next day: the number of Conservative seats was predicted perfectly, while overall the forecast was just three seats out.

The key beneficiaries of this research were:

• The media, who, as a result, were able to report in both an accurate and timely fashion on the outcomes of the election, improving the quality of their programmes and their reputation among their competitors.

According to the BBC's Head of Political Programmes, `the reputation of all three broadcasters, particularly the BBC, rests on producing an accurate exit poll result' [C1]. Predictions about the final outcome set the agenda for broadcasters to discuss the election and its political implications; this dominates programming after 10pm on election night. Thus, thanks to Fisher's exit poll analysis, interviewers were able to accurately query guests from the Liberal Democrats as to why they had not improved on their 2005 performance rather than mistakenly congratulating them in anticipation based on opinion-poll predictions. In addition, Fisher's statistical analysis enabled broadcasters to discuss the electorate's political choices in a more detailed and authoritative way. For example, the exit poll showed that Labour was particularly successful where it had an incumbent MP and where ethnic minorities were concentrated. Thanks to the nuances revealed by the research, broadcasters were able to describe these detailed patterns and invite various interviewees to explain them.

These detailed patterns are also important for understanding whether the early results, which mainly come from urban northern Labour constituencies, are in line with the exit poll predictions for that kind of seat despite being atypical of the average constituency. This enables the broadcasters to retain confidence in the exit poll prediction despite popular but naïve `swingometer' predictions suggesting a different outcome based on a biased set of early results. The exit poll continues to inform early results based forecasts until a wide variety of different kinds of seats have been declared, which happens as late as 4am the morning after. Thus the exit poll is valuable to the media not only before results are declared but also well after.

`One of the BBC's core public purposes is to sustain democracy and civil society', as the Head of Political Programmes emphasizes. `Reporting of politics and elections, and the election results programmes particularly, are a key part of fulfilling this public purpose.' [C1]. The Oxford-based research played a `vital' part in contributing to this mission [C2].

The election forecast reached an audience of several million. ITV1's election programme attracted a total of 1.3 million viewers, while Sky News averaged 0.4 million [C5]. BBC One's programme Election 2010 was watched by 6.6 million at its peak, and the BBC's overall television coverage of the election reached a total of 17.7 million [C6]. The election forecast was the headline news at 10pm on BBC Radio and on the BBC News website, which was visited by 4 million UK users on election day [C6]. Attention was also paid by still more people, internationally. These figures show widespread interest in the election outcome immediately after the close of polls, demonstrating that election night programmes are extremely important for broadcasters, highlighting their need for the provision of timely and accurate information, especially before the election results come in. They also show that the exit poll analysis reached large numbers of people who would not normally watch political programming.

• The Public, who, as a direct result of the programmes that were based on the research, became better informed about the mechanics of the electoral process and the contours of public opinion. This is especially significant for people who are not regular consumers of political news, as suggested by viewer numbers above. The validation of the exit poll forecast allowed people to see the power of social scientific methods, and may have helped them to establish a level of trust in evidence-based information. Those who rightly felt they could place a reasonable degree of trust in the exit poll could go to bed earlier! As the Research Strategy Manager from the Electoral Commission notes, "The nature of overnight counting of ballot papers (and, increasingly, the move to daytime counting) results in a delay in the announcement of any election result." [C7]. In 2010 most results were not declared until after 4am the morning after. So, "it is important [for the democratic process] not to lose the excitement of election night and the drama of the close of poll. Consequently, having the exit poll contributes enormously to the excitement of the evening ... the exit poll [is] a notable achievement and a valuable contribution to electoral politics." [C7]

Sources to corroborate the impact

[C1] Head of Political Programmes, BBC: letter of 17 May 2013

• explains the work done by Fisher and the importance of election night programming for the BBC's aim of informing the public

[C2] Election Results Editor, BBC: letter of 19 August 2013

• explains Fisher's statistical contributions to election night programming

[C3] Mark Blumenthal, `How The U.K. Polls Did', The National Journal, 10 May 2010

• American polling expert discusses the accuracy of election prediction; quotes Paddy Ashdown

[C4] `The Race for Westminster', The Times, 8 May 2010

• quotes Iain Dale

[C5] `TV Ratings', Guardian, 7 May 2010

• provides figures on the reach of election night programmes on television

[C6] BBC Press release http://www.bbc.co.uk/pressoffice/pressreleases/stories/2010/05_may/07/election_figures.shtml

• provides figures on the reach of election night programmes on BBC radio and website

[C7] Research Strategy Manager, Electoral Commission: letter of 16 September 2013

• explains how the exit poll is a valuable part of electoral politics and how impressive it was to be so accurate.