21 - Improving Telephone and Internet Retail Financial Services
Submitting InstitutionsHeriot-Watt University,
University of Edinburgh
Unit of AssessmentGeneral Engineering
Summary Impact TypeTechnological
Research Subject Area(s)
Information and Computing Sciences: Artificial Intelligence and Image Processing, Information Systems
Economics: Applied Economics
Summary of the impact
Research in ERPE (1994-date) to measure customer reaction and attitude to
communication interfaces in consumer services has widely influenced the
design of customer services at Bank of Scotland, Halifax and Lloyds TSB
(now Lloyds Group). The ERPE metric and the use methodology it relies on
have been adopted by enterprises in several service industry sectors —
telephony, retail, travel and financial services. Since 2008 the use and
impact has been predominantly in the financial services sector and is
encountered on a daily basis by the millions of retail and commercial
banking customers who use internet banking, mobile phone banking and
telephone banking services that have been created based on the ERPE
ERPE has had intimate collaborations with Lloyds Banking Group, who have
now adopted our refined usability metric into their business on a
significant scale. Since 2008 their business benefits have been five times
their £7.1 M investment in the ERPE research programme.
This research team led by Professor Jack with PDRAs: Anderson; Gunson;
Marshall; and Roy (all throughout the period), PDRAs: Douglas (to 2011);
McInnes (to 2009); Peevers (to 2010); and Weir (to 2009); with former
PDRAs: Foster; Love; Nairn; Richardson; and Wilkie.
Key outputs of ERPE research, since 1994, in the field of Usability
- A practical and robust usability engineering metric [1, 2] based on
sound engineering principles with customer attitude data derived under
- An associated use methodology [3 - 6] to assess and measuring the
commercial impact when introducing new consumer-facing technologies to
telephone and internet based retail services. In the recent period this
has been predominantly deployed to assess the impact of new Lloyds
banking group services.
The foundations of the statistically-proven metric and its associated
robust use methodology can be traced back to a Science and Engineering
Research Council (SERC) grant which supported the basic work on "Usability
Engineering" research ["Dialogue Engineering In Automated Telephone
Services", 1994-1997, £0.6M] and to support from BT ["Strategic University
Research Initiative", £2.1M, 1993-2001, S1]. Lloyds TSB/Banking Group has
been the main sponsor of the research since 2004 ["Usability Engineering
for eBanking Services", £12.7M, 2004 to date].
The early usability engineering research identified the key attributes
involved in evaluating the usability of automated telephone dialogues,
which resulted in the development of a reliable usability metric for
telephony services [subsequently extended to Internet and mobile phone
services] to measure the contributions to usability made by each of these
key usability attributes . The usability metric uses a Likert-style
psychometric scale with respondents indicating on a 7-point response scale
the extent to which they agree or disagree with proposal statements
relating to each of the key usability attributes. The attributes in the
statistical metric focus on users' attitudes to new technologies and
consumer processes based on cognitive attributes — degree of
confusion experienced, concentration needed, stress and frustration; transparency
attributes — how well the process is structured, knowing what to do
next, feeling in control, perceived speed; quality attributes —
ease of understanding, readiness to use the process again, reliability,
efficiency, improvements; engagement attributes — friendliness and
attractiveness, security and reassurance, meeting user expectations.
The robustness of the usability engineering statistical assessment metric
is based on three factors: (a) ensuring that only users from the target
community of real-world users are selected, with equal numbers of men and
women; equal numbers of older and younger users; all being regular users
of the existing versions of the service for which the new technology is
being considered; (b) ensuring that each participant undertakes a set of
practical real world tasks with the technology in a realistic setting; and
(c) ensuring that each participant completes the metric data at
controlled, key points in their experience. In this way, the usability
engineering metric enables the successful collection of statistically
reliable data (customers' emotions, attitudes and perceptions of the
effectiveness, efficiency, user satisfaction) to assess the impact of how
future new technology roll-out will impact their business.
The usability engineering metric has been continually refined based on
data derived from several UK companies who have adopted the metric and its
underlying use methodology in their businesses . The metric has proven
to be particularly valuable in improving customer services in several
industry sectors: telephony; retail ; travel; and finance. The metric
has proven to be scalable, involving as many as 800 participants in some
cases; it has proven to be portable having been used in several UK cities
as well as in Europe and USA; and further it has proven to be reliable,
being validated with new technologies including Internet [IEEE
Transactions 3] automatic speech recognition and speech synthesis ,
mobile phone services  and biometrics, malware and phishing, and strong
multi-factor customer authentication technologies  using one-time
passcode generators etc.
References to the research
References identified with * are those which best indicate the quality of
the underpinning research.
 Love, S., Dutton, R.T., Foster, J.C., Jack M.A. and Stentiford, F.W.,
`Identifying Salient Usability Attributes for Automated
Telephone Services', in Proceedings of International Conference on
Spoken language Processing (ICSLP-94), pp. 1307-1310, 1994.
32 Google Scholar (GS) citations.
This paper broke new ground, identifying five factors central to the
design of a reliable metric for usability assessment of spoken telephone
dialogues. The metric served as the basis of several EU-funded projects
[e.g. Spotlight, IST-1999-10314, €2.2M] and has been adopted by several
research teams internationally [e.g. Universities of Nijmegen, Netherlands
and Aalborg, Denmark].
 Foster, J.C., McInnes, F.R., Jack, M.A., Love, S., Dutton, R.T.,
Nairn, I.A. and White, L.S., `An Experimental Evaluation of
Preferences for Data Entry Method in Automated Telephone Services',
Behaviour and Information Technology, Vol. 17, No. 2, pp. 82-92, 1998.
14 GS citations.
This paper is in one of the top international journals for studies in
Usability Engineering [Impact Factor 1.101] and the work was taken up and
extended by researchers at IBM Watson Labs. NY, USA.
* McBreen H.M. and Jack, M.A., `Evaluating Humanoid Synthetic
Agents in E-Retail Applications', IEEE Transactions on Systems, Man,
and Cybernetics, Part A, Systems and Humans, Vol. 31, No.5, pp. 394-405,
63 GS citations.
This paper is in the highly respected IEEE journal for studies in
Usability Engineering [Journal Impact Factor 2.123] and the paper
influenced further work at Stanford University, USA and Peking University,
* Wilkie, J., Jack M.A. and Littlewood, P., `System-Initiated
Digressive Proposals in Automated Human-Computer Telephone Dialogues:
The Use of Contrasting Politeness Strategies', International Journal
of Human Computer Studies, Vol. 62, No. 1, pp. 41-71, 2005. DOI:10.1016/j.ijhcs.2004.08.001.
16 GS citations.
This paper is in one of the top international journals for Usability
Engineering [Impact Factor 1.171] and was one of the first papers to
investigate human-human politeness theories applied to spoken
human-machine dialogues, and was a point of departure for other work at
the Universities of Tsing Hua, Taiwan, Santa Barbara, USA and North
* Peevers, G., Douglas, G., and Jack, M.A., `A Usability
Comparison of Three Alternative Message Formats for an SMS Banking
Service'. International Journal of Human Computer Studies, Vol. 66,
pp.113-123, February 2008. DOI:10.1016/j.ijhcs.2007.09.005.
31 GS citations.
This paper is in one of the top international journals for Usability
Engineering [Impact Factor 1.17] and the metric reported in it has been
further enhanced by researchers at Singapore University and at Cartagena
 Weir, C.S., Douglas, G., Richardson, T. and Jack, M.A., `Usable
Security: User Preferences for Authentication Methods in eBanking and
the Effects of Experience' May-2010, Interacting with Computers.
Vol. 22, No. 3, pp. 153-164, 2009. DOI:10.1016/j.intcom.2009.10.001.
20 GS citations.
This paper is in one of the leading international journals in the field of
Usability Engineering [Journal Impact Factor 1.23] and the work has been
developed further by researchers at NEC Labs. Germany.
Details of the impact
The ERPE usability metric has proven to be particularly significant for
improving customer services in several industry sectors: telephony (BT
[S1], Vodafone, Sky TV, Cable and Wireless); retail (lastminute.com) ;
travel (British Airways, British Midland); and finance (Barclays, RBS,
HSBC, Lloyds Group, Nationwide, Standard Life).
The key reportable impact since 2008 is evidenced by the millions of
retail and commercial customers using the Internet banking, mobile phone
banking and telephone banking enhanced services of Bank of Scotland,
Halifax and Lloyds TSB (now Lloyds Group) that have been created, based on
the commercially relevant research carried out in ERPE.
Lloyds Banking Group has embraced the engineering approach developed by
ERPE for all new technologies and business processes in advance of
commercial deployment. As a consequence of their early adoption of the
metric and its assessment methodology, Lloyds has made a distinct and
material contribution to enhancing their electronic banking services [S2].
Through their continuing ongoing commitment to embrace the research,
Lloyds has created innovative customer services that now successfully
handle 60 Million Internet banking customer log-ins per month via their
award-winning internet banking service (http://www.moneyfactsgroup.co.uk/awards/ca/winners/2011)
[S3] and 9 Million automated telephone banking customer log-ins per month
via their automated telephone banking service. Lloyds assess that the
method has provided a major change in the quality and capability of their
customer services, particularly for the self-service channels of Internet
and telephone banking. Lloyds has assessed that the impact in terms of
business benefits from its adoption of the metric and its use methodology
on this research programme (£7.1M since 2008) is "at least five times
the funding that the Group has invested in the research since 2008",
Senior Manager, Lloyds Banking Group [S2].
Lloyds started to introduce the ERPE usability metric, outlined in
Section 2, into developments of their customer-facing business operations
in 2008 and since 2008 some 70 large scale user experiments based on the
new usability engineering metric have been undertaken, involving some
9,000 Lloyds customers in Bridgend, Bristol, Edinburgh, Enfield and Leeds.
Specifically, Lloyds has identified four key impacts that they have
derived as a business —
Providing strategic guidance to the business.
The nature of the benefit centres on what Lloyds Banking Group recognises
as delivering a robust fact-based, metric data for Usability Engineering.
Results give the Group assurance about proceeding with adoption of a
particular technology, based on customers' attitudes to the core usability
of that technology. For example, the innovative design approach used by
Lloyds Banking Group in their mobile (smart) phone services for mBanking
(mobile banking) [5, 6] exploited our usability assessment metric to
identify optimal user interface designs for customer registration and
customer authentication login. These mBanking services handle over 30
Million customer log-ins per month.
Lloyds Banking Group has worked with ERPE to create fully realistic
facsimile copies of each of the Group's banking channels — a `customer
theatre'. These stand-alone services, with dummy customer data and stub
databases, faithfully reproduce the real-world banking experiences for
customers allowing use of the future technologies and business processes,
under controlled conditions. The huge costs associated with testing a
commercial system with customers are avoided by the design and
construction of prototype systems to replicate exactly the Group's
strategy that can deliver metric data on customer attitudes and
preferences that would otherwise be very expensive for the Group to
derive. The usability engineering metric has been used extensively
throughout the Lloyds Banking Group covering all aspects of the Group's
operations — ATMs, branch offices, Internet banking, automated telephone
banking [2, 4] and mobile phone banking , and has been particularly
valuable in the work of the Group to integrate the services of the three
Banks: Bank of Scotland; Halifax; and Lloyds TSB, after the Lloyds Banking
Group was formed through the acquisition of HBOS by Lloyds TSB in 2009.
The metric and its use methodology also allow staff to become trained up
in usability engineering. This addresses the Group's stated policy of
taking customer views seriously. Three members of senior staff from Lloyds
Banking Group have successfully competed MSc degrees in Usability
Engineering at Edinburgh, and have succeeded in refining aspects of the
metric and its underlying use methodology in their studies.
Allowing Go/No-Go decisions.
The Group can measure customer reactions and likelihood of acceptance —
and importantly, customer resistance. This puts the Group in a stronger
position to make decisions about making expensive investments in adopting
novel technology and thereby reduces the likelihood of poor customer
reaction, bad publicity and serious financial losses. As an example, a
planned early deployment  of voice biometric technology for customer
authentication in telephone banking was cancelled based on usability
engineering data from research by ERPE, avoiding serious financial and
reputational loss for the Group.
Sources to corroborate the impact
[S1] Formerly Senior Manager BT, who funded and assessed their strategic
research initiative, now at UCL can further corroborate the wider impact
of this research.
[S2] Senior Manager, Lloyds Banking Group, Bristol, see comments included
in Section 4.
shows the internet banking award to Lloyds Group following on from their
adoption research results of ERPE.