Summary of the impact

This case study describes the impact achieved by novel methodology for election exit-poll design and analysis, at the 2010 UK General Election. The context is that the underpinning research had already led to success in predicting the outcome of the May 2005 General Election for public broadcasts by BBC and ITV. Its direct impacts at the 2010 General Election were significant and far-reaching:

(i) The same statistical methods were used successfully in 2010 also by a third major UK and international broadcaster, Sky (in addition to BBC and ITV).

(ii) The TV, radio and internet audiences of the three channels combined totalled many millions of people who were thereby informed of the likely 2010 election outcome immediately after the close of polling stations.

(iii) The prediction that was produced and broadcast at close of polls in May 2010 was both surprising to political commentators and extremely accurate.

As a result of these successes, the statistical methods we developed are now the "industry standard" for electoral exit-poll design and analysis in the UK.

Underpinning research

The underpinning research was application-specific statistical methodology, developed by Professor David Firth at the University of Warwick between October 2004 and February 2006. The research work was carried out in preparation for the May 2005 General Election, and then continued over several months after that election with post-election analytical work to assess the performance of the methods in detail.

Firth, together with Professor John Curtice of the University of Strathclyde (Department of Government), had been engaged by the BBC and ITV jointly to design and analyse their election- day exit poll at the next General Election (expected to take place in 2005 or 2006). A full account of the methodological development was published in 2008 in the jointly authored paper [1], detailed below.

The Curtice & Firth (2008) paper [1] describes several strands of novel statistical methodology developed for use at the 2005 General Election, which would be the first time that BBC and ITV (and their respective opinion-polling partner companies, NOP and MORI) were to combine resources to run an exit poll and produce a single forecast that would be broadcast by both organisations at 10pm on polling day. The novel methods included, as the most important statistical ingredients:

(i) Design of the exit-poll via a panel of polling stations (drawn from those that had been used by NOP and MORI in their separate operations at the 2001 General Election).

(ii) Modelling of the exit-poll data through multivariate regressions of electoral change. This included the detailed development and testing of a completely new, coherent approach to the treatment of multi-party shares of the vote.

(iii) Accurate calibration of constituency-level probability forecasts through a new, non- standard method with tuning constants determined through extensive experimentation with data from previous elections.

The methodological (statistical) research was all carried out by Firth. Curtice's role was to provide essential and detailed political and polling knowledge.

Firth's involvement in election-night forecasting had begun in 1997 when he worked at the University of Oxford: at the 1997 and 2001 General Elections he served as assistant to the veteran psephologist Clive Payne, who retired as the BBC's statistical consultant on election forecasting after the 2001 election. Early developments in some parts of the new methods of Curtice and Firth [1] were trialled at the 2001 election; the bulk of the research work, though, was done after Firth moved to Warwick on 1 October 2003 and was subsequently appointed by the BBC as Payne's successor to work on the 2005 election.

References to the research

The paper that reports all of the underpinning research is:

Details of the impact

The impact occurred in 2010, and was the result of novel statistical methods developed and published over the period 2004-2008. The impact was made possible through:

(i) Firth's direct involvement as a consultant to the broadcasters BBC and ITV in connection with the previous General Election held in May 2005.

(ii) The fact that the methods developed for the 2005 election had received high-profile attention from pollsters and broadcasters, through their publication in 2007 (and ultimately in print in 2008) as one of the Royal Statistical Society's prestigious Ordinary Meeting papers.

(iii) The fact that Firth had also written fully-documented software to implement the methods — software that could readily be used by knowledgeable others at a later election (as indeed was the case at the May 2010 election).

The research done by Firth (with Curtice) for the 2005 General Election produced methods which have since become the industry standard. At the most recent General Election in May 2010 those methods were used not only by BBC and ITV, but also for the first time by the prominent satellite and online broadcaster, Sky. The reasons are clear:

(i) The combination of methods described in Curtice and Firth (2008) is substantially more sophisticated statistically than any of the relatively simple approaches that had been used prior to 2005.

(ii) On the day of the 2005 election the analysis performed by Curtice and Firth for the BBC and ITV networks had resulted in an unprecedented, perfect prediction of the resulting 66-seat Labour majority in the House of Commons. This accuracy was impressive in its own right. It put paid to the myth that exit-poll-based predictions are always unreliable (a myth due largely to a spectacular failure by both BBC and ITV forecasters at the 1992 election — see Payne(2003)[9]). Moreover, the prediction of a 66-seats majority on 5 May 2005 was substantially different from the 100-120 seats majority predictions that had been made in many of the national newspapers on the strength of the latest pre-election voting intention polls. The exit-poll analysis had provided BBC and ITV with exactly what they wanted, a story-changing prediction that strongly influenced their TV coverage from 10pm on election night.

By the time of the May 2010 election, Firth himself had moved on to other projects; he was not available at that time as a consultant to the broadcasters. He did, however, provide software (written in R, with a user-friendly interface) for use by the BBC's and ITV's new generation of statistical consultants (Dr Stephen Fisher of the University of Oxford, and Dr Jouni Kuha of the London School of Economics and Political Science). Fisher and Kuha, again with the political expertise of Professor John Curtice on hand, applied the methods of Curtice and Firth (2008) to the data from an exit poll conducted by market-research companies MORI and NOP for BBC, ITV and Sky. The result was, again, a perfect prediction of the all-important number of House of Commons seats for the largest party; in 2010, again, it allowed the broadcasters to talk about the "right" story from 10pm on election night, this time a hung parliament with the Conservatives as largest party (on 307 seats, 19 seats short of an overall majority).

The direct significance and reach of the impact, in connection with the 2010 General election, are clear and substantial:

(i) The methods of Firth and Curtice (2008) were adopted enthusiastically at the 2010 election by the major broadcasters (and by their new generation of statistical consultants).

(ii) The methods had proven to be so effective at the 2005 election that the consortium of broadcasters using them was enlarged to include also Sky for the first time in 2010 (in addition to BBC and ITV).

(iii) The broadcasts made from 10pm on election night 6 May 2010 were watched, heard and browsed (on TV, radio and the internet) by many millions of people in the UK and abroad [e.g., source [8] says "A peak audience of 6.6 million watched David Dimbleby anchor Election 2010 on BBC One, BBC HD, BBC News Channel and on BBC Two in the nations, with a 4.7 million average, 36.1 per cent share and 16.1 million reach."]. The exit-poll-based prediction of the final election outcome was central to those broadcasts (see, e.g., source [5]) for at least the first couple of hours — this being the peak viewing/listening/browsing period.

(iv) The fact that the methods delivered highly accurate predictions to millions of people was a major contribution to the immediate public understanding of a political outcome that was likely to be complex and to require negotiation in advance of the formation of a new Government.

(v) The exit-poll-based prediction was surprising to many, including some prominent political commentators. In particular, the prediction based on the exit poll was that the Liberal Democrats would lose some seats in the House of Commons, whereas the pre-election voting intention polls had quite consistently suggested an increased number of seats for the Liberal Democrats. In the event, the 57 seats actually won by the Liberal Democrats in 2010 (down from 62 seats in 2005) was very close to, and even slightly worse for that party than, the 59-seats prediction that was broadcast from the exit poll.

Sources [2], [6] and [7] confirm the role played by the methods of Curtice and Firth [1] at the 2010 election, including the inclusion for the first time of Sky among the broadcasters using the methods. The letter [6] to Firth from the BBC's Editor of Political Research includes this:
"The importance of [the accuracy of the prediction] to the BBC, and to the other broadcasters too, is hard to overstate. After the disaster of the 1992 election, predictions based on UK exit polls were for many years viewed with great scepticism. The methods you developed for the 2005 election have completely turned this around. Our election-night programming, on TV, radio and the internet, is some of the most high-profile and important work that the BBC ever does. To have authority restored to the election night broadcast is invaluable to us (and, we like to think, also to the viewing/listening/browsing public, in the UK and more widely); not least because in recent elections the exit poll has become even more crucial to our flagship election night programme. In 1992 — the last general election without local elections held on the same day — the exit poll prediction announced at 10pm had been displaced by 1am with a prediction based on 156 declared results. In the 2010 election only 11 seats had been declared by 1am. In 1992, by 2am some 464 seats had declared: in 2010, the equivalent figure was 62. As a result, the exit poll prediction carries our programme for much longer than in past elections and its accuracy is therefore more closely scrutinized than ever before. Without such effective exit-poll methodology, the broadcasts on 6-7 May 2010 would have been substantially less informative and less engaging to the audience(s).
Your methods are now certainly the `industry standard' for UK electoral exit polling and associated prediction, and it is almost inconceivable that they would not be used again at the next election due in 2015."
The letter [7] to Firth from the BBC's statistical consultants at the 2010 election includes:
"The statistical methods that you developed for the 2005 election, and which you reported and analysed in Curtice and Firth (2008), were absolutely crucial to the success of our work at the 2010 election. Your kind help in providing R software made it relatively easy for us to adopt the same methods in 2010, and we did so — with great success, as you know! The approach that you developed in Curtice and Firth (2008) is now the `gold standard' in this area of work, and will surely be used again at future elections."

Sources [4] and [5] exemplify the surprise, even disbelief, among political commentators. At 10pm on election night, the BBC TV presenter David Dimbleby himself [5] said of the prediction, on air:
"If that's right the Liberal Democrats despite all that noise and fury have actually dropped 3 seats, which could be one reason why we need to be sceptical about this exit poll."
After the event, John Rentoul in the Independent on Sunday (9 May 2010) wrote:
"The accurate prediction was so shocking, at 10pm on Thursday, that large numbers of Conservatives flooded the internet to scorn it as utterly implausible and to say that it could not possibly be right because it failed to accord with what they felt in their bones. Most incautious was Iain Dale, Tory blogger and would-be candidate, who said: 'It seems too incredible to be true that the Lib Dems are only predicted to get 59 seats. I'll run naked down Whitehall if that turns out to be true.' In the end, of course, the Lib Dems won even fewer seats, 57. Dale's streak is eagerly awaited."

Source [3] is a web-based public engagement resource that was made available by Firth soon after the 2010 election. It is designed to make this story accessible to non-specialists (e.g., to non- statistical journalists as well as to members of the general public). The main page there gives a non-technical description of the methods used, as well as some history, press quotes, references, etc. It is hoped that this very public success story has a substantial, indirect impact additional to the various direct impacts detailed above — namely, that it helps to inspire school-age students to consider mathematical sciences as a field of study and as a potentially interesting career choice.

Sources to corroborate the impact

[2] Curtice, J, Fisher, S D and Kuha, J (2011). Confounding the Commentators: How the 2010 Exit Poll Got it (More or Less) Right. Journal of Elections, Public Opinion & Parties A, 21, 211-235.
http://dx.doi.org/10.1080/17457289.2011.562612

[3] Firth, D. (2010) Exit polling explained. Public interest web-page and glossary,
http://warwick.ac.uk/dfirth/exit-poll-explainer. (Accessed more than 900 times from outside Warwick in the 180 days up to 16 October 2013.)

[4] Iain Dale's Diary, "Are the BBC/ITN/Sky About to Have Egg on Their Faces?",
http://iaindale.blogspot.co.uk/2010/05/are-bbcitnsky-about-to-have-egg-on.html

[5] YouTube clip of the opening few minutes of BBC TV election night coverage on 6 May 2010.
http://www.youtube.com/watch?v=hr2mojjmG1M

[6] Letter from the Editor of BBC Political Research.

[7] Letter from the BBC's statistical consultants for the May 2010 General Election.

[8] Official BBC viewing figures for election night 6 May 2010,
http://www.bbc.co.uk/pressoffice/pressreleases/stories/2010/05_may/07/election_figures.shtml

[9] Payne, C D (2003), Election Forecasting in the UK: The BBC's Experience. Euramerica 33, 193-234. http://www.ea.sinica.edu.tw/eu_file/12014268324.pdf