Summary of the impact

A programme of methodological research undertaken by the University of York has shaped the economic evaluation methods used by the National Institute for Health and Care Excellence (NICE) since its inception, to assess the cost-effectiveness of healthcare interventions. Therefore the methods developed by York underpin many of the healthcare decisions by NICE on which new and existing interventions and programmes should be funded and used in the NHS. The methods used by NICE have also had a major influence on the approach taken to technology assessment internationally and so York's underpinning research has had wide impact beyond the UK.

Underpinning research

The University of York has been providing the evidence to support the NICE technology appraisal process since its inception in 1999. The process entails assessment of the benefits, harms and cost-effectiveness of healthcare interventions. Given the nature of NICE's decision-making responsibilities, the methods used for cost effectiveness analysis (CEA) required further development. York research addressed important methodological issues and has therefore shaped the methods adopted by NICE in key areas (listed below). Developing methods for evaluation of healthcare technologies brings together a range of research skills from different institutions. Much of the work described was led by York whose researchers have made a sustained intellectual contribution throughout the period, centred on methods of cost effectiveness analysis to support health system decision making; characterising and reflecting evidential uncertainty and heterogeneity; and value of information analysis to prioritise and design research. Much of the research was led by York or, where undertaken in collaboration with other institutions, York researchers made significant intellectual contributions focusing on the economic issues.

(1) Approaches to dealing with decision uncertainty
We demonstrated: the sensitivity of evaluation results to key parameter estimates and the centrality of handling uncertainty appropriately in decision making; the importance of dealing with uncertainty in all sources of evidence simultaneously using probabilistic methods; the need to link parameter uncertainty to decision uncertainty and to the value of additional research; the potential need to tie access to new technologies to requirements for more data collection (1-3).

(2) Methods for evidence synthesis for use in economic evaluation
Research with the Bristol University (York co-investigators) considered appropriate methods for evidence synthesis to support decisions by NICE and similar organisations. We demonstrated how to estimate the effectiveness of an intervention relative to policy relevant comparators where no head-to-head comparison was available (4), building on earlier York research (5).

(3) Approaches to dealing with heterogeneity
We showed the importance of exploring variation in both the effects and the costs of health technologies according to patient clinical and socio-demographic characteristics. The research can inform decisions on which interventions should be recommended for which patients. We also explored the methodological issues raised by undertaking sub-group analysis in CEA (6).

(4) Methods to estimate the cost-effectiveness of increasing NHS uptake of innovation
A major challenge for most healthcare systems is how to ensure the adoption of cost-effective interventions. Value of implementation analysis, which estimates the level of investment justified for implementation activities was developed in York (7). With clinical input from East Anglia, we proposed that the Quality and Outcomes Framework performance indicators should use value of implementation analysis, applying the same evidential standards as those used by NICE (8).

(5) Estimating the cost-effectiveness threshold
CEA assesses whether the expected health gain from the use of a new medical technology exceeds the health likely to be forgone as other NHS activities are displaced to cover the additional costs of the new technology. The cost-effectiveness threshold represents an estimate of the health forgone as services are displaced. York researchers developed appropriate methods for threshold estimation to be used in decisions about the use of NHS resources (9).

Researchers: Mark Sculpher (Prof Nov 1997-); Susan Griffin (Research Fellow (RF) & Senior Research Fellow (SRF) Oct 2002-) Richard Cookson (Reader Aug 2006-); Simon Walker (RF Oct 2006-); Karl Claxton (SRF & Prof Oct 1989-); Anne Mason (RF & SRF June 1998-); Stephen Palmer (SRF & Prof April 1995-); Mike Drummond (Prof May 1990-); Fuijan Song (SRF July 1993-Oct 2000); Alison Eastwood (RF & SRF Sept 1994-); Anne-Marie Glenny (RF Oct 1995-April 1999); Elizabeth Fenwick (RF Oct 1999-Dec 2005); Marta Soares (RF 2007); Seb Hind (RF Oct 2010-); Eldon Spackman (RF Feb 2010-)

References to the research

Most of the research has been published in high quality peer reviewed journals and was supported primarily by competitively awarded grants.

Grants supporting the research
Sculpher MJ, Claxton K. Methodological issues relating to decision analysis for resource allocation in healthcare. MRC HSRC Programme Grant 1.4.04-31.3.09. £437,281 (York element)
Sculpher MJ. National Public Health Career Scientist. 1.7.01-30.6.06. NHSR&D £304,826. Manca A. Issues in generalisibility by location of results in economic evaluation. Wellcome Fellowship 1.9.04-31.8.07. The Wellcome Trust £131,552.
Ginnelly L. Use of decision analysis for evidence synthesis 1.10.04-30.9.07. NCCRCD £150,446.
Sculpher MJ, Claxton K, et al. Economic Evaluation of Healthcare Interventions DH Policy Research Programme (with Sheffield) 2011-2016. £1,891,841 (York element).
Sculpher MJ, Claxton K, Palmer S. NICE Decision Support Unit (with Sheffield) 1.4.03-31.3.12 £116,615; and Palmer SJ 1.4.12-31.3.17 £87,615.
Glenny AM, Altman DG, Song F, et al. Indirect comparisons of competing interventions 1999.
NIHR HTA £37,385 (with Oxford).
Sculpher MJ. Are the quality & outcomes framework (QOF) indicators a cost-effective use of NHS
resources? 1.1.07-30.6.07. Department of Health Policy Research Programme £69,536.

Details of the impact

The research undertaken at York underpins NICE technology appraisal methods. NICE's 2008 "Guide to the Methods of Technology Appraisal" (source 1) describes the key principles of appraisal methodology which must be adhered to by all organisations submitting evidence to the technology appraisal programme. The guidance was shaped by the York research and York researchers involved in: the NICE Decision Support Unit (Palmer); the Methodology Working Party (Sculpher); the Methods Guide Review Workshops (Sculpher, Griffin); producing 3 briefing papers (source 2). The Guide drew on York research showing the need to consider the uncertainty and limitations of the evidence base and a requirement for rigorous methods "to assess the implications of uncertainty, including the uncertainty around the appropriate structure of the economic model, the choice of sources and analyses to inform the estimates of costs and health effects, and the precision with which these are known." (para 5.1.11).

This Guide (and the 2013 version) has governed the methods used to generate each of the 161 pieces of Technology Guidance issued by NICE since 2008. These are relevant to clinical practice in conditions that are major causes of mortality and morbidity which also place significant demands on NHS resources. Over half of NICE guidance has been in the areas of cancer, cardiovascular and respiratory heath, which, in 2008, were responsible for almost 350,000 deaths (associated with over 2m years of life lost) and accounted for over £16bn of NHS spending. York research on methods for evidence synthesis was cited extensively in four NICE Decision Support Unit Technical guidance documents in 2011 (source 3). NICE has reviewed the Methods Guide (2013) and York staff have been heavily involved as members and facilitators of the Methods Guide Review Workshops (Griffin, McKenna, Palmer, Sculpher, Spackman, Walker) and have produced a briefing paper on the appropriate cost perspective for NICE appraisals to adopt (ie, whether evaluation of healthcare interventions should include costs that fall on non-health sectors) (source 4). Sculpher participated in the decision-making meetings on perspective and the revised Guide follows York recommendations in terms of limiting the perspective to the NHS and personal social services, unless NICE specifically requests a broader scope (source 5).

The Department of Health's consultation document on value based pricing referred to the need to establish the evidence base for an appropriate cost-effectiveness threshold, saying that "Work is already in hand with external experts to achieve this", citing Sculpher's research as the source of that evidence (source 6). The value and appropriateness of NICE processes were reviewed by the House of Commons Health Select Committee. Oral and written evidence was presented by York on the cost effectiveness threshold used by NICE and expenditure decisions by Primary Care Trusts. The Select Committee advocated that further research - similar to that undertaken by York - takes place on the cost effectiveness thresholds used by NICE. The Government endorsed the Committee's expression of support for NICE's programmes and commended the processes by which it discharges its responsibilities (source 7).

The York contribution to economic evaluation methods used by NICE is internationally influential. Countries that have developed/are developing decision-making institutions like NICE are listed by NICE International and the methods adopted are based on those developed by York (source 8). York researchers have been involved in directly advising other health systems about their methods and processes of technology assessment, for example, in Ireland: "The guidelines for evaluating clinical effectiveness have been developed in consultation with the Scientific Advisory Group of the Authority... this group includes methodological experts from the field of HTA." Drummond and Sculpher are listed in the Scientific Group as the only non-Irish members (source 9). The Belgian Guidelines external expert group lists York's Claxton and Gravelle as the only UK experts (source 10) and Sculpher is the only non-North American on the USA Cost-effectiveness Panel (source 11). The establishment and/or development of approaches to technology appraisal in countries such as Brazil, USA, China, Colombia, Estonia, Serbia, Thailand and Turkey are influenced by the methods and processes developed by York and employed by NICE: eg, the Brazilian guidelines: "The entire methodology pointed out in the following topics is based on internationally published methodological guidelines" — and cites NICE as one of the sources (source 12); the US Wellpoint HTA Guidelines: "The US Panel on Cost-Effectiveness and the latest NICE Guide to the Methods of Technology Appraisal argue for the pivotal role of a reference case in driving resource allocation decisions within health care systems. This position is also taken by WP ..." (source 13).

Our research has impacted on major NHS policy initiatives and has had a wider impact on the health care system. York research on the Quality and Outcomes Framework (QOF), whereby GPs are paid for achieving quality, has informed the approach taken by NICE. Since 2009 NICE has been responsible for producing an annual `menu' of new, evidence-based cost-effective clinical and health improvement indicators. The methods used by NICE to assess the value for money of potential QOF indicators are taken directly from York's research: "this appendix presents the underpinning assumptions and data requirements that will be used to determine cost effectiveness by the York Health Economics Consortium. The work has been informed by research undertaken by York University and University of East Anglia ..." (source 14). By ensuring that QOF payments reflect the value of the indicators in terms of the health benefit produced, our research helps to ensure maximum health gain is achieved for money spent on rewarding high quality care. The QOF indicators cover a range of conditions and activities for the population of 54 million registered patients in England served by over 8,000 GP practices, thus the impact on quality is considerable.

Professor Sir Michael Rawlins, chair of NICE until 2012, supporting our successful application for a Queen's Anniversary Prize, described York's impact: "The centre [CHE, Uni of York] has made extraordinary contributions both nationally and internationally to the development of health economics. It has done so from both a theoretical and practical standpoint. In particular, its contributions to the work of NICE have been so very important that I doubt if we would have achieved anything without the rigour and expertise provided by the Centre and many of its staff."

Sources to corroborate the impact

1. National Institute for Health and Clinical Excellence (NICE). Guide to the Methods of Technology Appraisal. London: NICE, 2008.
   http://www.nice.org.uk/media/B52/A7/TAMethodsGuideUpdatedJune2008.pdf
2. A Special issue of Pharmacoeconomics 2008 volume 26, issue 9 published all the briefing papers produced as part of the development of the NICE Methods Guidance, including 4 involving authors from York. http://link.springer.com/journal/40273/26/9/page/1
3. NICE Decision Support Unit Technical Documents 1-4. Dias S et al 2011
   http://www.nicedsu.org.uk/Technical-Support-Documents(1985314).htm
4. Centre for Health Economics Research paper 54 http://www.york.ac.uk/media/che/documents/papers/researchpapers/rp54_appropriate_perspectives_for_health_care_decisions.pdf
5. Guide to the methods of technology appraisal 2013. NICE, April 2013
   http://publications.nice.org.uk/guide-to-the-methods-of-technology-appraisal-2013-pmg9
6. Dept of Health A new value-based approach to the pricing of branded medicines, 2010
   http://webarchive.nationalarchives.gov.uk/20110322002013/http://www.dh.gov.uk/prod_consum_dh/groups/dh_digitalassets/@dh/@en/documents/digitalasset/dh_122793.pdf
7. House of Commons Health Committee. National Institute for Health and Clinical Excellence: First report of the Health Committee 2007-08. HC550. London: Stationery Office, 2008.
   http://www.publications.parliament.uk/pa/cm200708/cmselect/cmhealth/27/27.pdf

House of Commons Health Committee. National Institute for Health and Clinical Excellence: NICE's response to the Health Select Committee's first report of session 2007-08. HC550; 2008.
http://www.publications.parliament.uk/pa/cm200708/cmselect/cmhealth/550/550.pdf

The Government's Response to the Health Select Committee's first report of session 2007-08 on the NICE. CM7331; 2008.
http://webarchive.nationalarchives.gov.uk/20080817152455/http://www.dh.gov.uk/en/Publicationsandstatistics/Legislation/DH_083348?IdcService=GET_FILE&dID=161359&Rendition=Web

8. NICE International Review 2011
   http://www.nice.org.uk/media/5F8/F8/NICEInternationalReview2011.pdf
9. Health Information and Quality Authority. Guidelines for Evaluating the Clinical Effectiveness of Health Technologies in Ireland. Dublin: Health Information and Quality Authority; 2011.
   http://www.hiqa.ie/healthcare/health-technology-assessment/guidelines
10. Guidelines for pharmacoeconomic evaluations in Belgium. KCE Report 78C. Belgium Healthcare Knowledge Centre, 2008. Updated in 2011 (KCE report 103) with similar citation.
    https://kce.fgov.be/sites/default/files/page_documents/d20081027327.pdf
11. Second panel on cost-effectiveness in health and medicine, USA http://2ndcep.hsrc.ucsd.edu/
12. Methodological Guidelines for Appraisals on HTA for the Ministry of Health of Brazil. Department of Science and Technology, 2007. Section 2.2:
    http://200.214.130.94/rebrats/publicacoes/diretrizes_ptc.pdf
13. Wellpoint HTA Guidelines The Wellpoint Outcomes based formulary 2008
    http://www.elsevierbi.com/~/media/Images/Publications/Archive/The%20Pink%20Sheet/70/043/00700430001/wellpoint_formulary_guidelines.pdf
14. NICE Developing clinical and health improvement indicators for the Quality and Outcomes Framework: Interim process guide. May 2009, Appendix A.
    http://www.nice.org.uk/media/742/32/QOFProcessGuide.pdf