Summary of the impact

Since its launch in 2009, the mobile phone package price comparison tool Billmonitor has identified £35 million worth of savings available to the 110,000 users whose bills have been analysed. It was the first price comparison tool to be accredited by Ofcom and it has been widely praised in the media. Exploiting techniques that they had developed for applications in finance and genetics, University of Oxford researchers Chris Holmes and Nicolai Meinshausen developed the statistical algorithms underpinning the package, which uses simulation-based inference and careful statistical modelling to analyse users' phone bill data. It searches over 2.4 million available packages to identify the best mobile phone deal for each user's particular pattern of usage. Widely quoted in the press, reports in 2011 and 2012 from the Billmonitor team estimated that approximately three quarters of mobile phone customers are on the wrong tariff, with an overspend of around 40%.

Underpinning research

The identification of an effective statistical method for Billmonitor was itself a research problem in applied statistics, whose resolution rested on the insights gained from research in other, at first sight unconnected, application areas. Chris Holmes, Professor of Statistics at the University of Oxford since 2004, and Nicolai Meinshausen (University Lecturer at the University of Oxford, 2007-2013) were able to combine the insights that they had gained in overcoming parallel difficulties in their research in finance and genetics to build an effective forecasting tool for an individual's mobile phone usage.

Greatest costs are incurred when mobile phone users stray outside their monthly allowances and so, in order to reliably forecast the expected cost of a tariff, the model used must accurately approximate the tails of the user behaviour (the unusual months). Billmonitor rests on a bespoke bootstrap algorithm, developed by Holmes and Meinshausen, to make the predictions.

In any time series in which there is regime shifting, more recent observations are more reliable in predicting the future evolution than older ones. Bootstrap samples are therefore restricted to a time window. However, if this window is too short, sample size is reduced and prediction variance is inflated. If the window is too long, the predictive distribution is not adapted to regime-switching and bias is inflated. Holmes's work on Bayesian inference for regime switching in threshold models for time-series variability [1] underpinned his analysis of adaptation to user regime-switching in phone bills. Holmes and Meinshausen discussed notions of statistical uncertainty and translated this into a bootstrap setting for the application. The switch to a windowing approach was inspired by Holmes's experience working with data partitioning methods. A blocked bootstrap method is used to optimise the algorithm.

Another strand of Holmes's research at the time was developing loss-functions for recovering regime switching ("segmental classification") along genetic sequences with time-series-like structure [2]. Traditional approaches, such as reporting the most probable state sequence or the most probable set of marginal predictions, correspond to particular choices of loss function that may be inappropriate for segmental analysis of sequence data. The new class of Markov loss functions proposed in [2] penalises misclassification of both state occupancy and transitions. The sequence of minimum expected loss is then enumerated using dynamic programming methods.

Meinshausen completed the prototyping for Billmonitor using R.

References to the research

The two asterisked outputs best indicate the quality of the underpinning research. Both papers are in high quality internationally refereed journals.

Details of the impact

Not only has Billmonitor had a substantial economic impact, it has also affected public policy and had a significant impact on society. The main beneficiaries are the parent company Optimor and the general public.

Billmonitor is the sole product of the parent company Optimor, which was established specifically in order to bring Billmonitor to market. The resulting price comparison website, launched in 2009, was the first of its kind to be recognised by the independent regulator, Ofcom. The Mobile Phone Guide [A], published by Ofcom's Consumer Focus and Communications Consumer Panel, explains how Billmonitor can help consumers track down the best deal. Since the launch of Billmonitor, over 110,000 users have provided details of their mobile phone usage through the site. The savings available over the lifetime of a contract (typically 12-24 months) identified for those users total over £35 million [B]. Reports such as http://www.billmonitor.com/billmonitor-national-mobile-report-2012 by the Billmonitor team, which estimate a total overspend on UK mobile phone contracts totalling literally billions of pounds per annum, have received extensive press coverage.

How research underpins impact
The founder of Optimor discussed the idea of the company with Holmes and Meinshausen, who realised that their previous research provided just the insights required for building an accurate algorithm for predicting an individual's future mobile phone usage. He states [B] "The research done by Professor Chris Holmes and Professor Nicolai Meinshausen of the University of Oxford was an essential component which has enabled us to build an accurate algorithm for predicting an individual's future mobile phone usage and we have used this to develop Billmonitor."

Users provide access to their mobile phone call plan, call histories and billing and Billmonitor finds a telephone contract optimised to the individual's pattern of use (see figure). The user's call history is treated as a time series, and makes predictive inference for future calls. For each potential contract, the algorithm uses robust bootstrap methods for adaptive prediction to construct a distribution over future pseudo-bills. Two particular problems present themselves when trying to use this to predict an individual's future mobile phone usage. First, user behaviour evolves with time, resulting in regime shifting and consequently a bias-variance trade-off associated with window scale selection. The insights provided by Holmes's previous work were key to the development of the windowing approach taken in Billmonitor. The second problem is that users show occasional large deviations from their `usual' behaviour, exposing them to potentially heavy losses. The importance of the distribution of loss in the context of regime switching, which was so important in Holmes's work on segmental classification of genetic sequences [2], becomes the accurate estimation of the distribution of the upper tail of the `pseudo-bills' for each contract. The choice of window size in Billmonitor is optimised via a bootstrap study of the population exploiting a blocked bootstrap method. Prototyping was completed using R.

Once prototyping of the statistical tool was complete, the front-end user interface of the programme was designed by engineers at Optimor, with Holmes and Meinshausen advising on the graphical displays of information.

Nature and extent of impact
The parent company, Optimor, employs a managing director and four developers. Holmes and Meinshausen were directors of Optimor (Holmes from April 2008 until January 2012 and Meinshausen from April 2008 until September 2011) and continue to act as scientific advisers, and there are three further commercial advisers. The company's financial model allows it to provide a completely unbiased price comparison tool: Billmonitor receives a small affiliate commission if users purchase via the links on the site. However, they will pay no more than if they had bought direct. The links are non-biased and advice is given exclusively on the basis of which plan provides the best deal for the user's projected usage.

In 2009, Billmonitor was the first price comparison site to be awarded the price accreditation scheme logo from Ofcom, having met the terms of a rigorous independent audit, which tests whether information given to consumers is accessible, accurate, transparent, comprehensive and up to date. The head of company performance at Consumer Focus, the statutory organisation campaigning for consumer rights, explained at the time why this was such an important step: "Accreditation for comparison sites is vital. With tens of thousands of mobile phone tariffs on the market consumers need peace of mind that before they take up a new deal they [can] get comprehensive and unbiased advice" [C]. In 2010 Billmonitor featured in Ofcom's guide to obtaining the best mobile phone deal [A].

Since the launch of Billmonitor, a number of rival comparison sites have been launched. However, as the online technology magazine CNET puts it "There are enough mobile phone price comparison sites out there that someone should launch a price comparison comparison site. Except that they don't need to, because Bill Monitor is simply the best" [D]. TheBillmonitor site is also ranked top for accuracy by the online consumer advice service MoneySavingsExpert [E] who describe it as the "Best for any decent handset".

In 2011, the Billmonitor team published an analysis, based on 28,500 UK mobile phone bills, which revealed that three out of four users were paying an average of £195 too much for their mobile phone contracts, equating to almost half their annual spend. This report received widespread media coverage, with articles in The Times, The Guardian, The Telegraph, The Daily Mail, The Financial Times and The Wall Street Journal; and features on Sky News and Channel 5's Gadget show [F]. In a feature on BBC1's The One Show, Billmonitor was used to help the people of Bristol save £1,000 in one hour [G].

The 2012 Billmonitor National Mobile Report painted a similar picture, and recommended a "tariff diet" for Britons to help put an end to an estimated £6 billion pounds annually wasted on mobile phone bills. The team also published a separate smartphone data report, featured, for example, on the BBC news website [H].

The Founder and Managing Director of Optimor states [B]: "Since its inception, Billmonitor has analysed over 1.5 million phone bills, an average of 14 months worth for each of a total of over 110,000 users, and identified £35 million of savings. We estimate that the overspend on UK mobile phone contracts totals literally billions of pounds per annum." According to URLmetrics [I], the Billmonitor site receives an average of 725 visits per day. As one might expect for an internet service, it receives a huge amount of coverage on the internet. An internet search on `Billmonitor' reveals literally hundreds of blogs and websites recommending the service. Typical, is a piece on the personal finance website Money to the Masses, "Billmonitor I salute you as I will save over £360 a year" [J].

Sources to corroborate the impact

[A] Ofcom's "The Mobile Phone Guide", specifically recommends Billmonitor and demonstrates the reach and significance of the impact:
consumers.ofcom.org.uk/files/2010/03/mobiledeal_v2b.pdf

[B] Letter from Founder and Managing Director of Optimor, highlighting the pathway from research to impact and the significance of Billmonitor. Copy held by University of Oxford.

[C] Quote from Head of Company Performance at Consumer Focus, describing the importance of accreditation: http://www.guardian.co.uk/money/2009/may/21/mobile-phones-billmonitor

[D] CNET Magazine snippet comparing mobile phone comparison sites:
http://reviews.cnet.co.uk/gadgets/best-money-saving-tech-50004215/

[E] MoneySavingExpert website, demonstrating significance of Billmonitor:
http://www.moneysavingexpert.com/phones/mobile-phone-cost-cutting.

[F] Billmonitor in the press: http://www.billmonitor.com/what-others-are-saying.html?alt=false

[G] Billmonitor on One Show, BBC, demonstrating significance
www.youtube.com/watch?v=079u38-zCTA

[H] BBC News report on Billmonitor's smartphone data report www.bbc.co.uk/news/technology-21959032

[I] URL metrics analysis of the Billmonitor website http://urlm.co.uk/www.billmonitor.com#web

[J] Article on Money to the Masses website, indicating the significance of Billmonitor
http://moneytothemasses.com/how-i-cut-my-phone-bill-by-60-in-2-minutes