Summary of the impact

Research at York has had a direct impact on national guidance about the use of health technologies in the NHS. It provided methods that are used to assess whether a technology is expected to be a cost-effective use of NHS resources, how uncertain this assessment is likely to be and whether additional evidence is sufficiently valuable to recommend further research to support its widespread use. It has had an impact on the technologies available in the NHS and the evidence available to support their use: improving patient outcomes; saving NHS resources and strengthening the evidence base for clinical practice. It gives an explicit signal and incentive to manufacturers; informing development decisions and the type of evidence collected. It has had an international impact on how the adequacy of evidence is judged and research is prioritised; particularly in recent reforms in the United States (US) where the principles of this value of information (VOI) analysis are informing the prioritisation of $3.8bn for `comparative effectiveness research'. It has also informed the methods used in low and middle income countries, especially national agencies in health care systems in South East Asia and South America, as well as global funding bodies.

Underpinning research

The key contribution [see references 1 and 2 in Section 3] was to change the way uncertainty about the performance of health technologies (drugs, devices, diagnostics and public health interventions) is understood, quantified and interpreted. It rejected traditional statistical methods (whether based on frequentist hypothesis testing or Bayesian error probabilities and credible intervals) in favour of a more general framework, based on the principles of Bayesian decision theory. This identifies when: i) a health technology should be approved based on existing evidence; ii) whether the value of additional evidence would justify further research to inform these decisions in the future; and iii) what type of research is needed and how it should be designed. Subsequent development of this framework identified when the approval of a technology, that is expected to be cost-effective based on current evidence should be withheld until research findings or other sources of uncertainty resolve [3], e.g., when widespread use reduces the prospects of conducting the type of research that would be valuable, and/or commits (opportunity) costs that cannot be recovered should approval be withdrawn once research reports or other sources of uncertainty resolve over time. Work at York has also established the implications that these considerations have for research design [4] and drug pricing [8].

Research at York has demonstrated how this framework for decision making can be implemented using probabilistic decision analytic models (using simulation methods) and VOI analysis. It has changed the way uncertainty about the cost-effectiveness of health technologies is characterised [5], represented [6] and used to inform health care decisions. Researchers at York have demonstrated how decisions can be informed during the appraisal of health care technologies, including when access to a technology should be restricted until further research is completed [8]. This body of underpinning research also provides a means to identify research priorities and the efficient design of subsequent research [7]. It has provided the methodological foundations for a range of diverse applications and further development of methods, most of which has been undertaken at York.

This programme of methodological research at York has strong international and multidisciplinary recognition with significant impacts on methods within related and international fields. It has been funded from a number of external sources, including the Medical Research Council (MRC) and National Institute of Health Research (NIHR) methodology programme, which demonstrates its originality and rigour [8, 9]. The two externally funded pilot studies [10, 11] and the funding from National Institute for Health and Clinical Excellence (NICE) to form the NICE Decision Support Unit, as well as a number of other applications funded by MRC and the NHS HTA programme, demonstrates its significance and relevance. Its international reach is evidenced by the commissioning of research at York, which details how these principles and methods can be used to prioritise `comparative effectiveness research' in the US [12].

Researchers at York: Claxton K (Professor, Oct 1989-); Sculpher M (Professor, Nov 1997-); Palmer S (Professor; April 1995-); Griffin S (Senior RF; Oct 2002-) ; McKenna C (RF, Oct 2006-); Ginnelly (now Bojke) L (Senior RF; Oct 1999-) and Fenwick E (RF; Oct 1999-2005).

References to the research

Grants supporting the research

[8] Claxton K. Palmer S and Longworth L. Informing a decision framework for when NICE should recommend the use of health technologies only in the context of an appropriately designed programme of evidence development. Medical Research Council and National Institute for Health Research Methodology Programme, 2010 to 2011, £285,000 Research report published by NIHR in peer reviewed monograph series, Health Technology, 2012, Volume 16, number 46.

[9] Sculpher M., Claxton, K. and Ades, T. Methodological issues relating to decision analysis for resource allocation in health care. Medical Research Council Programme Health Services Research Collaboration, 2004 to 2009, £437,281 (York element).

[10] Claxton, K., Ginnelly L, and Sculpher. A pilot study of using value of information analysis to set priorities in research and development. National Coordinating Centre for Health Technology Assessment, 2003, £30,000.

[11] Claxton K. and McCabe C. A pilot study of using value of information analysis to support research recommendations for the National Institute for Clinical Excellence. National Institute for Clinical Excellence, 2004, £15,000 (York element).

[12] Claxton K. Expected health benefits of additional evidence: Principles, methods and application. Patient Centred Outcomes Research Institute, 2012, £25,000. Published by PCORI and as CHE Research Paper 83, 2013, (374 downloads).

Details of the impact

Research at York has shaped the analysis required during the appraisal of health technologies undertaken by NICE and the (mandatory) guidance that it issues to the NHS. Since 2008 NICE has issued 161 pieces of Technology Guidance relevant to clinical practice in conditions that are major causes of mortality and morbidity which also place significant demands on NHS resources, e.g., over half of NICE guidance has been in the areas of cancer, cardiovascular and respiratory heath, which, in 2008, were responsible for almost 350,000 deaths associated with over 2m years of life lost and accounted for over £16bn of NHS spending.

The NICE Guide to the Methods of Technology Appraisal specifies the type of analysis required in submissions made by manufacturers and independent assessments by academic units (the `Reference Case'). It also specifies how such analysis will be used by the Appraisal Committees in developing guidance about the use of health technologies in the NHS. In 2008 NICE published the updated the NICE Guide to the Methods of Technology Appraisal. The briefing paper for the Methods Review Workshop on exploring uncertainty drew heavily on the underpinning research [see sources 1 and 2 in Section 5]. Claxton was invited to present this briefing at the Methods Review Workshop (2007) and participated in the Methods Review Working Party (2007) responsible for updating the Guide. The updated Guide required the use of probabilistic analysis to characterise decision uncertainty as part of the Reference Case. It recommended the use of VOI analysis to understand the consequences of uncertainty and the need for additional evidence to inform the research recommendations made when NICE issues guidance [3, 15]. The Guide informed all subsequent appraisals and the guidance issued by NICE.

The NICE Guide to the Methods of Technology Appraisal was updated during 2011-12 and a new Guide was published in 2013. The research funded by MRC/NIHR, which built on other underpinning research, had an important impact on the 2013 Guide. It was presented and evaluated at two stakeholder workshops hosted by NICE (2010 and 2011) which included members of NICE and its advisory committees, Department of Health, clinicians, patient representatives, and manufacturers. Claxton was a Specialist Advisor to the Methodology Working Party, responsible for updating the Guide, and the finding of this research formed the basis of the briefing paper on uncertainty and only in research recommendations (Nov 2011). The 2013 Guide maintained a requirement to use probabilistic analysis to characterise decision uncertainty and continued to recommend the use of VOI to understand its consequences. Importantly, the guidance on when `only in research' recommendations will be made also reflects the principles, considerations and type of assessments set out in the York research [4, 16]. This Guide continues to inform all appraisals undertaken by NICE. The significance and relevance of this research is also evidenced by requests to present the key finding to the NICE Technical Forum (Oct 2011), Diagnostic Technologies Appraisal Committee (May 2012) and the Medical Technologies Appraisal Committee (June 2012). Work at York has also informed how NICE clinical guidelines are developed by the Royal Collages [5], technical guidance on use of methods [6] and the House of Commons investigation of NICE in 2012 [7]

The research has also had an impact on how publically funded evaluative research is prioritised and commissioned. For example, the MRC and NIHR funded research has included the use of VOI analysis in initial grant funding before committing to funding large scale research proposals. More recently Claxton was invited to present work developed for PCORI to the HTA Commissioning Board (2013) and researchers at York are currently working with the NIHR Evaluation, Trials and Studies Coordinating Centre (NETSCC) to use these methods to routinely prioritise all research topics considered by NETSCC.

Research at York has also had an international impact on how uncertainty should be characterised and the adequacy of evidence judged. The international reach of this research is evidenced by the recommendations to use these methods in international guidelines on evaluation of health technologies [8-10], as well as citation and use of these methods in policy documents in Europe and the US [9-11]. It is also evidenced by invitations to present and advise institutions and policy makers (e.g., Swedish Council on Health Technology Assessment, 2011; Canadian Agency for Drugs and Technologies in Health (CADTH), 2009; US Agency for Healthcare Research and Quality, 2008) and a European policy network which includes manufacturers, regulators, reimbursement authorities from across Europe (European Healthcare Innovation Leadership Network, 2011, and European working group on post launch research, 2012).

The research has had an impact on the methods of analysis used in low and middle income countries to assess the cost-effectiveness and health technologies and the adequacy of evidence through guidance from WHO [10, 12] and the guides to methods of evaluation adopted by national agencies in health care systems in South East Asia (e.g., HTAsiaLink region) and in South America (Chile, Columbia and Brazil) [10, 16]. It is also influencing policy formation in global health and the type of analysis required by global funding bodies, e.g., Bill and Melinda Gates Foundation (BMGF), World Bank and Department for International Development (DFID) [16]. Claxton was invited to the Bill and Melinda Gates Foundation (BMGF, 2011) to present and advise how these methods of analysis can inform the research, development and funding decisions made by the foundation. In 2013 BMGF commissioned development of reference case guide for the methods used in funded research. Researchers at York are playing a key role in developing this guide, including how uncertainty and the need for evidence should be assessed [16].

This research has had an impact on research prioritisation in the United States where the Patient Centred Outcomes Research Institute (PCORI) is using the principles of VOI analysis to inform the prioritisation of the initial $3.8bn allocated by the US Congress for `comparative effectiveness research' as part of the recent health care reforms. Researchers at York have worked with PCORI in developing methods which can be routinely applied in ways that are consistent with the remit of this federally funded body. Claxton was an expert advisor at the PCORI Methodology Committee meeting (Baltimore March 2012) to present the principles of VOI analysis and act as discussant for the `white papers' presented, which also cited work at York [13, 18] . Claxton was commissioned to develop a `white paper' on how these methods could be used by PCORI consistent with its particular remit and demonstrate their application through a number of case studies. This work was presented to the PCORI methodology workshop in Washington in December 2012 [17]. The principles of VOI analysis outlined in this research are being used to inform prioritisation of the many research topics that have been suggested to it [14, 17, 18].

The impact was achieved by demonstrating the feasibility and benefits of the application of these methods [10, 11] and long term engagement with decision makers (Claxton was a founding member of: the NICE Technology Appraisal Committee, 1999 to 2010; the NICE Medical Technologies Appraisal Committee 2010-12; and the NICE DSU, 2003-). It also required development of methods relevant to a range of different contexts [12] and communicating insights in a way that was accessible to a wide policy audience, through workshops, seminars and invited lectures around the world; as well publications in more general clinical and policy journals. It also required a number of short courses to be developed and delivered to train analysts from the pharmaceutical sector, public agencies and ministries of health in how to use these methods. The core 3-day course has been running at least twice each year since 2003 and has been delivered to almost 1000 participants in 5 countries across 4 continents.

Sources to corroborate the impact

[1] National Institute for Health and Clinical Excellence. Briefing paper for methods review workshop on exploring uncertainty. London, Manchester: NICE; 2008. Cites [refs 1 and 3]

[2] National Institute for Health and Clinical Excellence. Briefing paper for methods review workshop on identifying sub-groups and exploring heterogeneity. London, Manchester: NICE; 2007. Cites [ref 1]

[3] National Institute for Health and Clinical Excellence (NICE). Guide to the Methods of Technology Appraisal. London: NICE, 2008.

[4] Guide to the methods of technology appraisal 2013. NICE, April 2013

[5] National Institute for Health and Care Excellence. The guidelines manual 2012. London: NICE; 2012. Cites [ref 7]

[6] NICE DSU technical support documents 6. Sheffield: University of Sheffield; 2012. Cites [ref 5]

[7] House of Commons Health Select Committee. National Institute for Health and Clinical Evidence. Volume II Additional written evidence. London: The Stationary Office; 2013. Cites [ref 7]

[8] Pharmaceutical Management Agency. Prescription for pharmacoeconomic analysis: methods for cost utility analysis. Wellington NZ: PHARMAC; 2007. Cites [ref 5]

[9] Guidelines for pharmacoeconomic evaluations in Belgium. KCE Report 78C. Cites [source 1] and Claxton and Gravelle (from York) as the only UK expert advisors. Updated in 2011 (KCE report 103) with similar citation.

[10] NICE International Review, 2011. Cites where [source 1] has been used/influenced policy internationally.

[11] Hoomans T, et al. Systematizing the use of value of information analysis in prioritizing systematic reviews. AHRQ Publication No. 12-EHC109-EF. Rockville, MD: Agency for Healthcare Research and Quality; 2012. Cites [ref 7]

[12] World Health Organization. WHO guide to cost-effectiveness analysis. Geneva: WHO; 2003. Cites [ref 6]; World Health Organization. WHO guide for standardization of economic evaluations of immunization programmes. Geneva: WHO; 2008. Cites [ref 6]

[13] Myers E, et al. Value-of-information analysis for patient-centered outcomes research prioritization. Washington DC: Patient-Centered Outcomes Research Institute; 2012. Cites [ref 4 & 7]

[14]. Pilot testing PCORI's Process for prioritizing Research Topics. Washington DC: Patient-Centered Outcomes Research Institute; 2013.

Factual statements:

[15] Director of the Centre for Health Technology Evaluation at NICE; , Chair of the Appraisal Committee and Chair of the Methods Working Party at NICE; and Programme Director, Technology Appraisals Centre for Health Technology Evaluation at NICE.

[16] Director, NICE International.

[17] Director, CER Methods and Infrastructure Program.

[18] Chair, Methodology Committee of the Patient Centered Outcomes Research Institute (PCORI)