Summary of the impact

A new design methodology which enhances the quality and accuracy of information derived from non-market valuation surveys is now the instrument of choice for many consultancy projects worldwide. The approach — known as Sequential Efficient Bayesian (SEB) design — enhances the decision-making process and the efficiency of service providers and is now available in major commercial software such as JMP from SAS (the integrated statistical software package created by SAS Corp and used by many businesses and academic institutions around the world). The methodology is routinely used in the design of surveys to analyse consumer and public choices in relation to willingness to pay for health, transport and environmental services. The research team's external affiliation since 2012 with the business consultancy CENSOC Sydney helps to provide tailored SEB designs to commercial clients, including several international, blue-chip corporations listed on the CENSOC site:http://www.censoc.uts.edu.au/about/members/external.html

Underpinning research

The impetus for this work arose in 2003 when a research team led by Professor George Hutchinson of Queen's University undertook a project to value the non-market benefits produced by EU agri-environmental schemes in the Republic of Ireland, in terms of rural landscape improvements. The project involved the design of a nationwide survey. The researchers faced the challenge of deriving nationally representative estimates from a relatively small sample size. In response, the researchers developed the Sequential Efficient Bayesian (SEB) Design and applied it for the first time in a non-market valuation survey.

The design, based on a technique named after the 18^th century mathematician Thomas Bayes, uses probability calculations and knowledge gained from pilot surveys to produce statistically robust final estimates. The published results show that the SEB approach can produce the same level of accuracy with a sample size one third smaller than non Bayesian designs (Scarpa, Campbell and Hutchinson 2007). Improved accuracy means that smaller sample surveys are now viable, thus extending the market for commercial surveys of this type.

With the market for such surveys approaching hundreds annually, many costing over £100,000, the financial savings of SEB designs are in the order of millions of pounds globally. Using the design in the area of non-market valuation provides decision-makers with estimates for the value of intangible outputs such as time saved in transportation, value of health improvements in clinical care, and the benefits of improved air quality or drinking water in environmental management.

The work was originally disseminated in Scarpa, Campbell and Hutchinson 2007. This paper reports the first SEB Survey in the world and suggests SEB design as a major tool in non-market valuation using Choice Experiments, staking the team's claim as the original inventors of this now widely used method. The research on SEB design was prepared for market application by larger internationally well-known research institutes with closer links to consultancy and business than the team in Queen's, all of whom have cited this paper as the earliest examples of the use of SEB design in a non-market valuation survey.

The approach was further developed and refined in a number of subsequent papers.

The second paper (Campbell, D., Hutchinson, W.G., Scarpa, R.), published by the team in 2009, extends SEB design to provide a GIS based spatial mapping of willingness to pay using statistical interpolation techniques. The third (Marco Boeri, Alberto Longo, José M. Grisolía, W. George Hutchinson, Frank Kee [2013]) explores random regret minimisation as a potentially better fit for choice data than the usual random utility maximisation model, when applied to health-related choices in SEB design surveys. Paper 4 (Scarpa, Riccardo; Zanoli, Raffaele; Bruschi, Viola; Naspetti, Simona) explores the problem issue of respondents not attending to all attributes when making food choices using SEB design surveys.

References to the research

Publications:

Grants:

The ongoing work on SEB design in Choice Modelling has attracted high quality grant income into Queen's, amounting to £1.1million since 2002. Major funding bodies include the ESRC (Centre of Excellence for Public Health NI), the Medical Research Council, Department for Employment and Learning, (research excellence) and EU funding.

Since 2002 five Postdoctoral Researchers and 11 PhD students have been involved in this work. Since 2012 Professor Scarpa spends three months per year as an affiliate of the leading Choice Modelling Consultancy CENSOC ensuring the transfer of academic ideas such as SEB design into commercial and industrial practice.

Details of the impact

The impact of this research can be demonstrated in a number of ways, but in particular by the influence of this approach across the survey design and operational research community worldwide. The SEB approach is now the standard method for experimental design of stated choice experiments, especially in non-market valuation where often only small samples are viable and affordable.

The pioneering development of the earliest SEB design survey has been translated into a framework with global reach, through the following stages in which the Queen's team has engaged.

1. Larger research institutes, which span the middle ground between university research and industry application, further developed and introduced the SEB design method to transportation modelling engineers. These groups include the Institute for Transport and Logistics, University of Sydney and the Institute of Transport, Leeds (UK). Similarly, the SEB design has been developed and promoted to the consultancy and business sector by the Operational Research and Business Statistics practitioners at the University of Leuven, Belgium and the Centre for Study of Choice, CENSOC University of Sydney. The Queen's team has worked with these institutes through joint publications, and has a formal link as an external affiliate of CENSOC Sydney http://www.censoc.uts.edu.au/about/members/external.html and an honorary research fellowship advising Rand Europe Cambridge http://www.rand.org/randeurope.html on Queen's developments in design and modelling.
2. Since 2008 the Institute for Transport and Logistics, University of Sydney has included SEB design in industry training programmes for survey consultants. These short courses are run annually in both Australia and Europe. Queen's staff have been guest lecturers at several of these, in Bologna (2008 and 2009), Venice 2010 and Venice and Trento in 2011.
3. The recent International Conferences of Choice Modelling (Sydney 2013 and Leeds 2011) and the launch of the Journal of Choice Modelling by Elsevier http://www.journals.elsevier.com/journal-of-choice-modelling/ are industry-led initiatives to transfer academic-led innovation into industry practice. The team actively disseminated and promoted SEB design by presenting seven papers at Leeds 2011. Queen's is on the Editorial Committee of the Journal and, through CENSOC, had input into the organisation of Sydney 2013. Three Queen's papers have been accepted for the Sydney conference.
   http://www.icmconference.org.uk/index.php/icmc/ICMC2013
4. Specific software packages, which permit consultants to use the SEB design method without advanced programming expertise, are now commercially available . These include specialist choice modelling, software Ngene by Choicemetrics Ltd Sydney. The routine for SEB design is on page 220 of the manual http://www.mediafire.com/?2bljnvc67djr9vs The subsidiary of the well-known statistical package SAS known as JMP now provides routines to conduct the SEB design, an acknowledgement of our contribution by the Director of Choicemetrics has been uploaded separately.
5. The SEB experimental design techniques have been implemented in stated preference surveys deployed to assess "willingness to pay" for water services in the UK by market research firms including Perceptive Insight, Belfast; EFTEC, London; Accent, London; and Rand Europe, Cambridge. . The method was used in studies for Yorkshire Waters, United Utilities, Thames Waters and Anglia Waters, among others. It was also utilised in the preparation of the OFWAT (the Water Services Regulation Authority in England and Wales) documents for tariff negotiation rounds in 2007 and 2012. Queen's is advising Perceptive Insight Belfast on a Customer Survey it is conducting locally on behalf of the utility company Northern Ireland Water. Riccardo Scarpa is also leading a team advising the Australian Energy Market Operator in the use of SEB design in the largest non-market survey of this kind into the value of lost electricity load across 6 of the 8 Commonwealth States. The results will be used to plan investments to improve the national electricity grid.
6. The Director of the Centre for Study of Choice in Sydney, a leading world consultancy on Choice Modelling, , has endorsed the SEB design as follows: "Since 2007 our centre has made frequent use of the principles of efficient design for stated choice....Designs of this type were found to greatly improve efficiency and hence require smaller sample sizes ......in applied contract work that we perform for a variety of clients, such as Bose (USA), Motorola (USA and Europe), Deloitte Data Analytics (Australia), the NSW Department of Transport, The Victorian Department of Health, Lend-Lease (Australia) and many more."

Sources to corroborate the impact

The Director (Research), CENSOC, Sydney Australia. This person is one of the world's leading practitioners and lifelong developer of Choice Modelling. He has the academic, consultancy and advisory expertise to confirm the significance of our introduction of SEB design into non-market valuation in Choice Modelling. He can corroborate the sample size savings that result and the increased precision in estimates.

The Director, Institute of Transport and Logistic Studies (ITLS), University of Sydney This international expert can confirm that our SEB design method is used by his consultancy organisation and many other expert consultants in the field of transport modelling to estimate, among other factors, willingness to pay for time reductions in transportation.

The Managing Director, Accent Market Research, London, Edinburgh and Bristol. Accent is a medium sized full services market research agency. Clients include British Gas, CAA, Consumer Focus, East Coast Mainline, NHS, Legal and General, Ofgem, Royal Mail, Scottish Water and TfL. The company has expertise in large scale Choice Modelling and Non Market Valuation. The Managing Director can confirm that, in surveys of this type, Accent uses the SEB design which offers cost advantages in terms of sample size and also improves the accuracy and precision of estimates.

The Director Choice Metrics, Sydney Australia and partner in the Institute of Transport and Logistic Studies (ITLS), University of Sydney. This person can confirm that SEB design is now incorporated in the Choice Metrics statistical package Ngene. He can also confirm the use of SEB design in consultancy projects for clients of ITLS.

The Senior Human Resource Manager of Rand Europe, Cambridge UK can confirm that Professor Scarpa has been offered an Honorary Research Fellowship to advise Rand Europe in developments in Choice Modelling made by our research group in Queen's University.