ECO05 - Characterising uncertainty and value of information in health care decisions
Submitting InstitutionUniversity of York
Unit of AssessmentEconomics and Econometrics
Summary Impact TypePolitical
Research Subject Area(s)
Mathematical Sciences: Statistics
Economics: Applied Economics
Summary of the impact
Research at York has had a direct impact on national guidance about the
use of health technologies in the NHS. It provided methods that are used
to assess whether a technology is expected to be a cost-effective use of
NHS resources, how uncertain this assessment is likely to be and whether
additional evidence is sufficiently valuable to recommend further research
to support its widespread use. It has had an impact on the technologies
available in the NHS and the evidence available to support their use:
improving patient outcomes; saving NHS resources and strengthening the
evidence base for clinical practice. It gives an explicit signal and
incentive to manufacturers; informing development decisions and the type
of evidence collected. It has had an international impact on how the
adequacy of evidence is judged and research is prioritised; particularly
in recent reforms in the United States (US) where the principles of this
value of information (VOI) analysis are informing the prioritisation of
$3.8bn for `comparative effectiveness research'. It has also informed the
methods used in low and middle income countries, especially national
agencies in health care systems in South East Asia and South America, as
well as global funding bodies.
The key contribution [see references 1 and 2 in Section 3] was to change
the way uncertainty about the performance of health technologies (drugs,
devices, diagnostics and public health interventions) is understood,
quantified and interpreted. It rejected traditional statistical methods
(whether based on frequentist hypothesis testing or Bayesian error
probabilities and credible intervals) in favour of a more general
framework, based on the principles of Bayesian decision theory. This
identifies when: i) a health technology should be approved based on
existing evidence; ii) whether the value of additional evidence would
justify further research to inform these decisions in the future; and iii)
what type of research is needed and how it should be designed. Subsequent
development of this framework identified when the approval of a
technology, that is expected to be cost-effective based on current
evidence should be withheld until research findings or other sources of
uncertainty resolve , e.g., when widespread use reduces the prospects
of conducting the type of research that would be valuable, and/or commits
(opportunity) costs that cannot be recovered should approval be withdrawn
once research reports or other sources of uncertainty resolve over time.
Work at York has also established the implications that these
considerations have for research design  and drug pricing .
Research at York has demonstrated how this framework for decision making
can be implemented using probabilistic decision analytic models (using
simulation methods) and VOI analysis. It has changed the way uncertainty
about the cost-effectiveness of health technologies is characterised ,
represented  and used to inform health care decisions. Researchers at
York have demonstrated how decisions can be informed during the appraisal
of health care technologies, including when access to a technology should
be restricted until further research is completed . This body of
underpinning research also provides a means to identify research
priorities and the efficient design of subsequent research . It has
provided the methodological foundations for a range of diverse
applications and further development of methods, most of which has been
undertaken at York.
This programme of methodological research at York has strong
international and multidisciplinary recognition with significant impacts
on methods within related and international fields. It has been funded
from a number of external sources, including the Medical Research Council
(MRC) and National Institute of Health Research (NIHR) methodology
programme, which demonstrates its originality and rigour [8, 9]. The two
externally funded pilot studies [10, 11] and the funding from National
Institute for Health and Clinical Excellence (NICE) to form the NICE
Decision Support Unit, as well as a number of other applications funded by
MRC and the NHS HTA programme, demonstrates its significance and
relevance. Its international reach is evidenced by the commissioning of
research at York, which details how these principles and methods can be
used to prioritise `comparative effectiveness research' in the US .
Researchers at York: Claxton K (Professor, Oct 1989-); Sculpher M
(Professor, Nov 1997-); Palmer S (Professor; April 1995-); Griffin S
(Senior RF; Oct 2002-) ; McKenna C (RF, Oct 2006-); Ginnelly (now Bojke) L
(Senior RF; Oct 1999-) and Fenwick E (RF; Oct 1999-2005).
References to the research
 Claxton K. The irrelevance of inference: a decision making approach
to the stochastic evaluation of health care technologies. Journal of
Health Economics 1999, 18: 341-364. DOI:
10.1016/S0167-6296(98)00039-3 Citations: 344 (Scopus citations at
 Claxton K., Walker S. and Lacey L. Selecting treatments: a decision
theoretic approach. Journal of the Royal Statistical Society A
2000, 163: 211-225. DOI:10.1111/1467-985X.00166 Citations: 35
 Griffin S, Claxton K, Palmer S, Sculpher M. Dangerous omissions: the
consequences of ignoring decision uncertainty. Health Economics
2011, 20: 212-24. DOI:10.1002/hec.1586 Citations: 22 (ISPOR Excellence
Award for Methodology Excellence 2012).
 McKenna C. and Claxton K.. Addressing Adoption and Research Design
Decisions Simultaneously: The Role of Value of Sample Information
Analysis. Medical Decision Making 2011, 316: 853-865. DOI:
10.1177/0272989X11399921 Citations: 4.
 Claxton K., Sculpher MJ., McCabe C., Briggs A., Akehurst R., Buxton
M., Brazier J. and O'Hagan A. Probabilistic sensitivity analysis for NICE
technology assessment: not an optional extra. Health Economics
2005, 14: 339-347. DOI: 10.1002/hec.985 Citations: 156 (submitted to RAE
2008 where 96.6% of Departmental outputs were rated 2* or higher).
 Fenwick E., Claxton K. and Sculpher, MJ. Representing uncertainty:
the role of cost-effectiveness acceptability curves. Health Economics
2001, 10: 779-89. DOI: 10.1002/hec.635 Citations: 416.
 Claxton K. and Sculpher M. Using Value of Information Analysis to
Prioritise Health Research: Some Lessons from Recent UK Experience. Pharmacoeconomics
2006, 24:1055-1068. DOI: 10.2165/00019053-200624110-00003 Citations: 80.
Grants supporting the research
 Claxton K. Palmer S and Longworth L. Informing a decision framework
for when NICE should recommend the use of health technologies only in the
context of an appropriately designed programme of evidence development. Medical
Research Council and National Institute for Health Research
Methodology Programme, 2010 to 2011, £285,000 Research report published by
NIHR in peer reviewed monograph series, Health Technology, 2012, Volume
16, number 46.
 Sculpher M., Claxton, K. and Ades, T. Methodological issues relating
to decision analysis for resource allocation in health care. Medical
Research Council Programme Health Services Research Collaboration,
2004 to 2009, £437,281 (York element).
 Claxton, K., Ginnelly L, and Sculpher. A pilot study of using value
of information analysis to set priorities in research and development. National
Coordinating Centre for Health Technology Assessment, 2003, £30,000.
 Claxton K. and McCabe C. A pilot study of using value of information
analysis to support research recommendations for the National Institute
for Clinical Excellence. National Institute for Clinical Excellence,
2004, £15,000 (York element).
 Claxton K. Expected health benefits of additional evidence:
Principles, methods and application. Patient Centred Outcomes Research
Institute, 2012, £25,000. Published by PCORI and as CHE Research
Paper 83, 2013, (374 downloads).
Details of the impact
Research at York has shaped the analysis required during the appraisal of
health technologies undertaken by NICE and the (mandatory) guidance that
it issues to the NHS. Since 2008 NICE has issued 161 pieces of Technology
Guidance relevant to clinical practice in conditions that are major causes
of mortality and morbidity which also place significant demands on NHS
resources, e.g., over half of NICE guidance has been in the areas of
cancer, cardiovascular and respiratory heath, which, in 2008, were
responsible for almost 350,000 deaths associated with over 2m years of
life lost and accounted for over £16bn of NHS spending.
The NICE Guide to the Methods of Technology Appraisal specifies the type
of analysis required in submissions made by manufacturers and independent
assessments by academic units (the `Reference Case'). It also specifies
how such analysis will be used by the Appraisal Committees in developing
guidance about the use of health technologies in the NHS. In 2008 NICE
published the updated the NICE Guide to the Methods of Technology
Appraisal. The briefing paper for the Methods Review Workshop on exploring
uncertainty drew heavily on the underpinning research [see sources 1 and 2
in Section 5]. Claxton was invited to present this briefing at the Methods
Review Workshop (2007) and participated in the Methods Review Working
Party (2007) responsible for updating the Guide. The updated Guide
required the use of probabilistic analysis to characterise decision
uncertainty as part of the Reference Case. It recommended the use of VOI
analysis to understand the consequences of uncertainty and the need for
additional evidence to inform the research recommendations made when NICE
issues guidance [3, 15]. The Guide informed all subsequent appraisals and
the guidance issued by NICE.
The NICE Guide to the Methods of Technology Appraisal was updated during
2011-12 and a new Guide was published in 2013. The research funded by
MRC/NIHR, which built on other underpinning research, had an important
impact on the 2013 Guide. It was presented and evaluated at two
stakeholder workshops hosted by NICE (2010 and 2011) which included
members of NICE and its advisory committees, Department of Health,
clinicians, patient representatives, and manufacturers. Claxton was a
Specialist Advisor to the Methodology Working Party, responsible for
updating the Guide, and the finding of this research formed the basis of
the briefing paper on uncertainty and only in research recommendations
(Nov 2011). The 2013 Guide maintained a requirement to use probabilistic
analysis to characterise decision uncertainty and continued to recommend
the use of VOI to understand its consequences. Importantly, the guidance
on when `only in research' recommendations will be made also reflects the
principles, considerations and type of assessments set out in the York
research [4, 16]. This Guide continues to inform all appraisals undertaken
by NICE. The significance and relevance of this research is also evidenced
by requests to present the key finding to the NICE Technical Forum (Oct
2011), Diagnostic Technologies Appraisal Committee (May 2012) and the
Medical Technologies Appraisal Committee (June 2012). Work at York has
also informed how NICE clinical guidelines are developed by the Royal
Collages , technical guidance on use of methods  and the House of
Commons investigation of NICE in 2012 
The research has also had an impact on how publically funded evaluative
research is prioritised and commissioned. For example, the MRC and NIHR
funded research has included the use of VOI analysis in initial grant
funding before committing to funding large scale research proposals. More
recently Claxton was invited to present work developed for PCORI to the
HTA Commissioning Board (2013) and researchers at York are currently
working with the NIHR Evaluation, Trials and Studies Coordinating Centre
(NETSCC) to use these methods to routinely prioritise all research topics
considered by NETSCC.
Research at York has also had an international impact on how uncertainty
should be characterised and the adequacy of evidence judged. The
international reach of this research is evidenced by the recommendations
to use these methods in international guidelines on evaluation of health
technologies [8-10], as well as citation and use of these methods in
policy documents in Europe and the US [9-11]. It is also evidenced by
invitations to present and advise institutions and policy makers (e.g.,
Swedish Council on Health Technology Assessment, 2011; Canadian Agency for
Drugs and Technologies in Health (CADTH), 2009; US Agency for Healthcare
Research and Quality, 2008) and a European policy network which includes
manufacturers, regulators, reimbursement authorities from across Europe
(European Healthcare Innovation Leadership Network, 2011, and European
working group on post launch research, 2012).
The research has had an impact on the methods of analysis used in low and
middle income countries to assess the cost-effectiveness and health
technologies and the adequacy of evidence through guidance from WHO [10,
12] and the guides to methods of evaluation adopted by national agencies
in health care systems in South East Asia (e.g., HTAsiaLink region) and in
South America (Chile, Columbia and Brazil) [10, 16]. It is also
influencing policy formation in global health and the type of analysis
required by global funding bodies, e.g., Bill and Melinda Gates Foundation
(BMGF), World Bank and Department for International Development (DFID)
. Claxton was invited to the Bill and Melinda Gates Foundation (BMGF,
2011) to present and advise how these methods of analysis can inform the
research, development and funding decisions made by the foundation. In
2013 BMGF commissioned development of reference case guide for the methods
used in funded research. Researchers at York are playing a key role in
developing this guide, including how uncertainty and the need for evidence
should be assessed .
This research has had an impact on research prioritisation in the United
States where the Patient Centred Outcomes Research Institute (PCORI) is
using the principles of VOI analysis to inform the prioritisation of the
initial $3.8bn allocated by the US Congress for `comparative effectiveness
research' as part of the recent health care reforms. Researchers at York
have worked with PCORI in developing methods which can be routinely
applied in ways that are consistent with the remit of this federally
funded body. Claxton was an expert advisor at the PCORI Methodology
Committee meeting (Baltimore March 2012) to present the principles of VOI
analysis and act as discussant for the `white papers' presented, which
also cited work at York [13, 18] . Claxton was commissioned to develop a
`white paper' on how these methods could be used by PCORI consistent with
its particular remit and demonstrate their application through a number of
case studies. This work was presented to the PCORI methodology workshop in
Washington in December 2012 . The principles of VOI analysis outlined
in this research are being used to inform prioritisation of the many
research topics that have been suggested to it [14, 17, 18].
The impact was achieved by demonstrating the feasibility and benefits of
the application of these methods [10, 11] and long term engagement with
decision makers (Claxton was a founding member of: the NICE Technology
Appraisal Committee, 1999 to 2010; the NICE Medical Technologies Appraisal
Committee 2010-12; and the NICE DSU, 2003-). It also required development
of methods relevant to a range of different contexts  and
communicating insights in a way that was accessible to a wide policy
audience, through workshops, seminars and invited lectures around the
world; as well publications in more general clinical and policy journals.
It also required a number of short courses to be developed and delivered
to train analysts from the pharmaceutical sector, public agencies and
ministries of health in how to use these methods. The core 3-day course
has been running at least twice each year since 2003 and has been
delivered to almost 1000 participants in 5 countries across 4 continents.
Sources to corroborate the impact
 National Institute for Health and Clinical Excellence. Briefing paper
for methods review workshop on exploring uncertainty. London, Manchester:
NICE; 2008. Cites [refs 1 and 3]
 National Institute for Health and Clinical Excellence. Briefing paper
for methods review workshop on identifying sub-groups and exploring
heterogeneity. London, Manchester: NICE; 2007. Cites [ref 1]
 National Institute for Health and Clinical Excellence (NICE). Guide
to the Methods of Technology Appraisal. London: NICE, 2008.
 Guide to the methods of technology appraisal 2013. NICE, April 2013
 National Institute for Health and Care Excellence. The guidelines
manual 2012. London: NICE; 2012. Cites [ref 7]
 NICE DSU technical support documents 6. Sheffield: University of
Sheffield; 2012. Cites [ref 5]
 House of Commons Health Select Committee. National Institute for
Health and Clinical Evidence. Volume II Additional written evidence.
London: The Stationary Office; 2013. Cites [ref 7]
 Pharmaceutical Management Agency. Prescription for pharmacoeconomic
analysis: methods for cost utility analysis. Wellington NZ: PHARMAC; 2007.
Cites [ref 5]
 Guidelines for pharmacoeconomic evaluations in Belgium. KCE Report
78C. Cites [source 1] and Claxton and Gravelle (from York) as the only UK
expert advisors. Updated in 2011 (KCE report 103) with similar citation.
 NICE International Review, 2011. Cites where [source 1] has been
used/influenced policy internationally.
 Hoomans T, et al. Systematizing the use of value of information
analysis in prioritizing systematic reviews. AHRQ Publication No.
12-EHC109-EF. Rockville, MD: Agency for Healthcare Research and Quality;
2012. Cites [ref 7]
 World Health Organization. WHO guide to cost-effectiveness analysis.
Geneva: WHO; 2003. Cites [ref 6]; World Health Organization. WHO guide for
standardization of economic evaluations of immunization programmes.
Geneva: WHO; 2008. Cites [ref 6]
 Myers E, et al. Value-of-information analysis for patient-centered
outcomes research prioritization. Washington DC: Patient-Centered Outcomes
Research Institute; 2012. Cites [ref 4 & 7]
. Pilot testing PCORI's Process for prioritizing Research Topics.
Washington DC: Patient-Centered Outcomes Research Institute; 2013.
 Director of the Centre for Health Technology Evaluation at NICE; ,
Chair of the Appraisal Committee and Chair of the Methods Working Party at
NICE; and Programme Director, Technology Appraisals Centre for Health
Technology Evaluation at NICE.
 Director, NICE International.
 Director, CER Methods and Infrastructure Program.
 Chair, Methodology Committee of the Patient Centered Outcomes
Research Institute (PCORI)