UOA10-12: Billmonitor: predicting the best mobile phone contract for users
Submitting InstitutionUniversity of Oxford
Unit of AssessmentMathematical Sciences
Summary Impact TypeEconomic
Research Subject Area(s)
Mathematical Sciences: Statistics
Summary of the impact
Since its launch in 2009, the mobile phone package price comparison tool
Billmonitor has identified £35 million worth of savings available
to the 110,000 users whose bills have been analysed. It was the first
price comparison tool to be accredited by Ofcom and it has been widely
praised in the media. Exploiting techniques that they had developed for
applications in finance and genetics, University of Oxford researchers
Chris Holmes and Nicolai Meinshausen developed the statistical algorithms
underpinning the package, which uses simulation-based inference and
careful statistical modelling to analyse users' phone bill data. It
searches over 2.4 million available packages to identify the best mobile
phone deal for each user's particular pattern of usage. Widely quoted in
the press, reports in 2011 and 2012 from the Billmonitor team
estimated that approximately three quarters of mobile phone customers are
on the wrong tariff, with an overspend of around 40%.
The identification of an effective statistical method for Billmonitor
was itself a research problem in applied statistics, whose resolution
rested on the insights gained from research in other, at first sight
unconnected, application areas. Chris Holmes, Professor of Statistics at
the University of Oxford since 2004, and Nicolai Meinshausen (University
Lecturer at the University of Oxford, 2007-2013) were able to combine the
insights that they had gained in overcoming parallel difficulties in their
research in finance and genetics to build an effective forecasting tool
for an individual's mobile phone usage.
Greatest costs are incurred when mobile phone users stray outside their
monthly allowances and so, in order to reliably forecast the expected cost
of a tariff, the model used must accurately approximate the tails of the
user behaviour (the unusual months). Billmonitor rests on a bespoke
bootstrap algorithm, developed by Holmes and Meinshausen, to make the
In any time series in which there is regime shifting, more recent
observations are more reliable in predicting the future evolution than
older ones. Bootstrap samples are therefore restricted to a time window.
However, if this window is too short, sample size is reduced and
prediction variance is inflated. If the window is too long, the predictive
distribution is not adapted to regime-switching and bias is inflated.
Holmes's work on Bayesian inference for regime switching in threshold
models for time-series variability  underpinned his analysis of
adaptation to user regime-switching in phone bills. Holmes and Meinshausen
discussed notions of statistical uncertainty and translated this into a
bootstrap setting for the application. The switch to a windowing approach
was inspired by Holmes's experience working with data partitioning
methods. A blocked bootstrap method is used to optimise the algorithm.
Another strand of Holmes's research at the time was developing
loss-functions for recovering regime switching ("segmental
classification") along genetic sequences with time-series-like structure
. Traditional approaches, such as reporting the most probable state
sequence or the most probable set of marginal predictions, correspond to
particular choices of loss function that may be inappropriate for
segmental analysis of sequence data. The new class of Markov loss
functions proposed in  penalises misclassification of both state
occupancy and transitions. The sequence of minimum expected loss is then
enumerated using dynamic programming methods.
Meinshausen completed the prototyping for Billmonitor using R.
References to the research
* Dellaportas P, Denison D and Holmes C (2007) "Flexible threshold
models for modelling interest rate volatility". Econometric Reviews.
Special Issue on Bayesian Dynamic Econometrics. 26(2). 419-437 DOI:
* Yau, C., Holmes, C. (2013), "A decision theoretic approach for
segmental classification using Hidden Markov models", Annals of
Applied Statistics, 7(3), 1814-1835. DOI: 10.1214/13-AOAS657
The two asterisked outputs best indicate the quality of the underpinning
research. Both papers are in high quality internationally refereed
Details of the impact
Not only has Billmonitor had a substantial economic impact, it
has also affected public policy and had a significant impact on society.
The main beneficiaries are the parent company Optimor and the
Billmonitor is the sole product of the parent company Optimor,
which was established specifically in order to bring Billmonitor
to market. The resulting price comparison website, launched in 2009, was
the first of its kind to be recognised by the independent regulator,
Ofcom. The Mobile Phone Guide [A], published by Ofcom's Consumer
Focus and Communications Consumer Panel, explains how Billmonitor
can help consumers track down the best deal. Since the launch of Billmonitor,
over 110,000 users have provided details of their mobile phone usage
through the site. The savings available over the lifetime of a contract
(typically 12-24 months) identified for those users total over £35 million
[B]. Reports such as http://www.billmonitor.com/billmonitor-national-mobile-report-2012
by the Billmonitor team, which estimate a total overspend on UK
mobile phone contracts totalling literally billions of pounds per annum,
have received extensive press coverage.
How research underpins impact
The founder of Optimor discussed the idea of the company with
Holmes and Meinshausen, who realised that their previous research provided
just the insights required for building an accurate algorithm for
predicting an individual's future mobile phone usage. He states [B] "The
research done by Professor Chris Holmes and Professor Nicolai
Meinshausen of the University of Oxford was an essential component which
has enabled us to build an accurate algorithm for predicting an
individual's future mobile phone usage and we have used this to develop
Users provide access to their mobile phone call plan, call histories and
billing and Billmonitor finds a telephone contract optimised to
the individual's pattern of use (see figure). The user's call history is
treated as a time series, and makes predictive inference for future calls.
For each potential contract, the algorithm uses robust bootstrap methods
for adaptive prediction to construct a distribution over future
pseudo-bills. Two particular problems present themselves when trying to
use this to predict an individual's future mobile phone usage. First, user
behaviour evolves with time, resulting in regime shifting and consequently
a bias-variance trade-off associated with window scale selection. The
insights provided by Holmes's previous work were key to the development of
the windowing approach taken in Billmonitor. The second problem is
that users show occasional large deviations from their `usual' behaviour,
exposing them to potentially heavy losses. The importance of the
distribution of loss in the context of regime switching, which was so
important in Holmes's work on segmental classification of genetic
sequences , becomes the accurate estimation of the distribution of the
upper tail of the `pseudo-bills' for each contract. The choice of window
size in Billmonitor is optimised via a bootstrap study of the
population exploiting a blocked bootstrap method. Prototyping was
completed using R.
Once prototyping of the statistical tool was complete, the front-end user
interface of the programme was designed by engineers at Optimor,
with Holmes and Meinshausen advising on the graphical displays of
Nature and extent of impact
The parent company, Optimor, employs a managing director and four
developers. Holmes and Meinshausen were directors of Optimor
(Holmes from April 2008 until January 2012 and Meinshausen from April 2008
until September 2011) and continue to act as scientific advisers, and
there are three further commercial advisers. The company's financial model
allows it to provide a completely unbiased price comparison tool: Billmonitor
receives a small affiliate commission if users purchase via the links on
the site. However, they will pay no more than if they had bought direct.
The links are non-biased and advice is given exclusively on the basis of
which plan provides the best deal for the user's projected usage.
In 2009, Billmonitor was the first price comparison site to be
awarded the price accreditation scheme logo from Ofcom, having met the
terms of a rigorous independent audit, which tests whether information
given to consumers is accessible, accurate, transparent, comprehensive and
up to date. The head of company performance at Consumer Focus, the
statutory organisation campaigning for consumer rights, explained at the
time why this was such an important step: "Accreditation for comparison
sites is vital. With tens of thousands of mobile phone tariffs on the
market consumers need peace of mind that before they take up a new deal
they [can] get comprehensive and unbiased advice" [C]. In
2010 Billmonitor featured in Ofcom's guide to obtaining the best
mobile phone deal [A].
Since the launch of Billmonitor, a number of rival comparison
sites have been launched. However, as the online technology magazine CNET
puts it "There are enough mobile phone price comparison sites out
there that someone should launch a price comparison comparison site.
Except that they don't need to, because Bill Monitor is simply the best"
[D]. TheBillmonitor site is also ranked top for accuracy by the
online consumer advice service MoneySavingsExpert [E] who describe it as
the "Best for any decent handset".
In 2011, the Billmonitor team published an analysis, based on
28,500 UK mobile phone bills, which revealed that three out of four users
were paying an average of £195 too much for their mobile phone contracts,
equating to almost half their annual spend. This report received
widespread media coverage, with articles in The Times, The Guardian, The
Telegraph, The Daily Mail, The Financial Times and The Wall Street
Journal; and features on Sky News and Channel 5's Gadget show [F]. In a
feature on BBC1's The One Show, Billmonitor was used to help the
people of Bristol save £1,000 in one hour [G].
The 2012 Billmonitor National Mobile Report painted a similar
picture, and recommended a "tariff diet" for Britons to help put an end to
an estimated £6 billion pounds annually wasted on mobile phone bills. The
team also published a separate smartphone data report, featured, for
example, on the BBC news website [H].
The Founder and Managing Director of Optimor states [B]: "Since its
inception, Billmonitor has analysed over 1.5 million phone bills, an
average of 14 months worth for each of a total of over 110,000 users,
and identified £35 million of savings. We estimate that the overspend on
UK mobile phone contracts totals literally billions of pounds per annum."
According to URLmetrics [I], the Billmonitor site receives an average of
725 visits per day. As one might expect for an internet service, it
receives a huge amount of coverage on the internet. An internet search on
`Billmonitor' reveals literally hundreds of blogs and websites
recommending the service. Typical, is a piece on the personal finance
website Money to the Masses, "Billmonitor I salute you as I will save
over £360 a year" [J].
Sources to corroborate the impact
[A] Ofcom's "The Mobile Phone Guide", specifically recommends Billmonitor
and demonstrates the reach and significance of the impact:
[B] Letter from Founder and Managing Director of Optimor, highlighting
the pathway from research to impact and the significance of Billmonitor.
Copy held by University of Oxford.
[C] Quote from Head of Company Performance at Consumer Focus, describing
the importance of accreditation: http://www.guardian.co.uk/money/2009/may/21/mobile-phones-billmonitor
[D] CNET Magazine snippet comparing mobile phone comparison sites:
[E] MoneySavingExpert website, demonstrating significance of Billmonitor:
[F] Billmonitor in the press: http://www.billmonitor.com/what-others-are-saying.html?alt=false
[G] Billmonitor on One Show, BBC, demonstrating significance
[H] BBC News report on Billmonitor's smartphone data report www.bbc.co.uk/news/technology-21959032
[I] URL metrics analysis of the Billmonitor website http://urlm.co.uk/www.billmonitor.com#web
[J] Article on Money to the Masses website, indicating the significance