Summary of the impact

Research in ERPE (1994-date) to measure customer reaction and attitude to communication interfaces in consumer services has widely influenced the design of customer services at Bank of Scotland, Halifax and Lloyds TSB (now Lloyds Group). The ERPE metric and the use methodology it relies on have been adopted by enterprises in several service industry sectors — telephony, retail, travel and financial services. Since 2008 the use and impact has been predominantly in the financial services sector and is encountered on a daily basis by the millions of retail and commercial banking customers who use internet banking, mobile phone banking and telephone banking services that have been created based on the ERPE metric.

ERPE has had intimate collaborations with Lloyds Banking Group, who have now adopted our refined usability metric into their business on a significant scale. Since 2008 their business benefits have been five times their £7.1 M investment in the ERPE research programme.

Underpinning research

This research team led by Professor Jack with PDRAs: Anderson; Gunson; Marshall; and Roy (all throughout the period), PDRAs: Douglas (to 2011); McInnes (to 2009); Peevers (to 2010); and Weir (to 2009); with former PDRAs: Foster; Love; Nairn; Richardson; and Wilkie.

Key outputs of ERPE research, since 1994, in the field of Usability Engineering are:

• A practical and robust usability engineering metric [1, 2] based on sound engineering principles with customer attitude data derived under controlled conditions.
• An associated use methodology [3 - 6] to assess and measuring the commercial impact when introducing new consumer-facing technologies to telephone and internet based retail services. In the recent period this has been predominantly deployed to assess the impact of new Lloyds banking group services.

The foundations of the statistically-proven metric and its associated robust use methodology can be traced back to a Science and Engineering Research Council (SERC) grant which supported the basic work on "Usability Engineering" research ["Dialogue Engineering In Automated Telephone Services", 1994-1997, £0.6M] and to support from BT ["Strategic University Research Initiative", £2.1M, 1993-2001, S1]. Lloyds TSB/Banking Group has been the main sponsor of the research since 2004 ["Usability Engineering for eBanking Services", £12.7M, 2004 to date].

The early usability engineering research identified the key attributes involved in evaluating the usability of automated telephone dialogues, which resulted in the development of a reliable usability metric for telephony services [subsequently extended to Internet and mobile phone services] to measure the contributions to usability made by each of these key usability attributes [1]. The usability metric uses a Likert-style psychometric scale with respondents indicating on a 7-point response scale the extent to which they agree or disagree with proposal statements relating to each of the key usability attributes. The attributes in the statistical metric focus on users' attitudes to new technologies and consumer processes based on cognitive attributes — degree of confusion experienced, concentration needed, stress and frustration; transparency attributes — how well the process is structured, knowing what to do next, feeling in control, perceived speed; quality attributes — ease of understanding, readiness to use the process again, reliability, efficiency, improvements; engagement attributes — friendliness and attractiveness, security and reassurance, meeting user expectations.

The robustness of the usability engineering statistical assessment metric is based on three factors: (a) ensuring that only users from the target community of real-world users are selected, with equal numbers of men and women; equal numbers of older and younger users; all being regular users of the existing versions of the service for which the new technology is being considered; (b) ensuring that each participant undertakes a set of practical real world tasks with the technology in a realistic setting; and (c) ensuring that each participant completes the metric data at controlled, key points in their experience. In this way, the usability engineering metric enables the successful collection of statistically reliable data (customers' emotions, attitudes and perceptions of the effectiveness, efficiency, user satisfaction) to assess the impact of how future new technology roll-out will impact their business.

The usability engineering metric has been continually refined based on data derived from several UK companies who have adopted the metric and its underlying use methodology in their businesses [2]. The metric has proven to be particularly valuable in improving customer services in several industry sectors: telephony; retail [3]; travel; and finance. The metric has proven to be scalable, involving as many as 800 participants in some cases; it has proven to be portable having been used in several UK cities as well as in Europe and USA; and further it has proven to be reliable, being validated with new technologies including Internet [IEEE Transactions 3] automatic speech recognition and speech synthesis [4], mobile phone services [5] and biometrics, malware and phishing, and strong multi-factor customer authentication technologies [6] using one-time passcode generators etc.

References to the research

References identified with * are those which best indicate the quality of the underpinning research.

Details of the impact

The ERPE usability metric has proven to be particularly significant for improving customer services in several industry sectors: telephony (BT [S1], Vodafone, Sky TV, Cable and Wireless); retail (lastminute.com) [3]; travel (British Airways, British Midland); and finance (Barclays, RBS, HSBC, Lloyds Group, Nationwide, Standard Life).

The key reportable impact since 2008 is evidenced by the millions of retail and commercial customers using the Internet banking, mobile phone banking and telephone banking enhanced services of Bank of Scotland, Halifax and Lloyds TSB (now Lloyds Group) that have been created, based on the commercially relevant research carried out in ERPE.

Lloyds Banking Group has embraced the engineering approach developed by ERPE for all new technologies and business processes in advance of commercial deployment. As a consequence of their early adoption of the metric and its assessment methodology, Lloyds has made a distinct and material contribution to enhancing their electronic banking services [S2]. Through their continuing ongoing commitment to embrace the research, Lloyds has created innovative customer services that now successfully handle 60 Million Internet banking customer log-ins per month via their award-winning internet banking service (http://www.moneyfactsgroup.co.uk/awards/ca/winners/2011) [S3] and 9 Million automated telephone banking customer log-ins per month via their automated telephone banking service. Lloyds assess that the method has provided a major change in the quality and capability of their customer services, particularly for the self-service channels of Internet and telephone banking. Lloyds has assessed that the impact in terms of business benefits from its adoption of the metric and its use methodology on this research programme (£7.1M since 2008) is "at least five times the funding that the Group has invested in the research since 2008", Senior Manager, Lloyds Banking Group [S2].

Lloyds started to introduce the ERPE usability metric, outlined in Section 2, into developments of their customer-facing business operations in 2008 and since 2008 some 70 large scale user experiments based on the new usability engineering metric have been undertaken, involving some 9,000 Lloyds customers in Bridgend, Bristol, Edinburgh, Enfield and Leeds. Specifically, Lloyds has identified four key impacts that they have derived as a business —

Providing strategic guidance to the business.

The nature of the benefit centres on what Lloyds Banking Group recognises as delivering a robust fact-based, metric data for Usability Engineering. Results give the Group assurance about proceeding with adoption of a particular technology, based on customers' attitudes to the core usability of that technology. For example, the innovative design approach used by Lloyds Banking Group in their mobile (smart) phone services for mBanking (mobile banking) [5, 6] exploited our usability assessment metric to identify optimal user interface designs for customer registration and customer authentication login. These mBanking services handle over 30 Million customer log-ins per month.

Service creation.

Lloyds Banking Group has worked with ERPE to create fully realistic facsimile copies of each of the Group's banking channels — a `customer theatre'. These stand-alone services, with dummy customer data and stub databases, faithfully reproduce the real-world banking experiences for customers allowing use of the future technologies and business processes, under controlled conditions. The huge costs associated with testing a commercial system with customers are avoided by the design and construction of prototype systems to replicate exactly the Group's strategy that can deliver metric data on customer attitudes and preferences that would otherwise be very expensive for the Group to derive. The usability engineering metric has been used extensively throughout the Lloyds Banking Group covering all aspects of the Group's operations — ATMs, branch offices, Internet banking, automated telephone banking [2, 4] and mobile phone banking [5], and has been particularly valuable in the work of the Group to integrate the services of the three Banks: Bank of Scotland; Halifax; and Lloyds TSB, after the Lloyds Banking Group was formed through the acquisition of HBOS by Lloyds TSB in 2009.

Staff training.

The metric and its use methodology also allow staff to become trained up in usability engineering. This addresses the Group's stated policy of taking customer views seriously. Three members of senior staff from Lloyds Banking Group have successfully competed MSc degrees in Usability Engineering at Edinburgh, and have succeeded in refining aspects of the metric and its underlying use methodology in their studies.

Allowing Go/No-Go decisions.

The Group can measure customer reactions and likelihood of acceptance — and importantly, customer resistance. This puts the Group in a stronger position to make decisions about making expensive investments in adopting novel technology and thereby reduces the likelihood of poor customer reaction, bad publicity and serious financial losses. As an example, a planned early deployment [2009] of voice biometric technology for customer authentication in telephone banking was cancelled based on usability engineering data from research by ERPE, avoiding serious financial and reputational loss for the Group.

Sources to corroborate the impact

[S1] Formerly Senior Manager BT, who funded and assessed their strategic research initiative, now at UCL can further corroborate the wider impact of this research.

[S2] Senior Manager, Lloyds Banking Group, Bristol, see comments included in Section 4.

[S3] http://www.moneyfactsgroup.co.uk/awards/ca/winners/2011 shows the internet banking award to Lloyds Group following on from their adoption research results of ERPE.