Summary of the impact

Imperial College researchers have developed methods and indicators for highlighting potential variations in healthcare performance and safety using routinely collected health data. Analytical tools based on our methodological research are used by managers and clinicians in over two thirds of NHS hospital trusts, and hospitals throughout the world. The results of our analyses helped detect problems at Mid Staffordshire NHS Foundation Trust and triggered the initial investigation and subsequent public inquiry with wide ranging recommendations based on the recognition of their value and their use in enhancing the safety of healthcare.

Underpinning research

Key Imperial College London researchers:
Dr Paul Aylin, Clinical Reader in Epidemiology and Public Health (1997 to date)
Dr Alex Bottle, Senior Lecturer in Statistics (1998 to date)
Professor Sir Brian Jarman (1984-1998, Emeritus since 1998)

We have used routinely collected clinical and administrative data to examine variations in quality and safety in healthcare. The research has increased the use of data in the management and monitoring of healthcare in the UK and internationally. Our work has led to the development of innovative statistical and computational methods for processing large data sets derived from electronic medical records and NHS databases.

Work by Professor Jarman and colleagues at Imperial published in 1999 on hospital standardised mortality ratios (HSMRs) established that there was substantial variation in mortality between hospitals in England which was not accounted for by a range of explanatory variables (1). In work examining paediatric cardiac surgical outcomes for the Bristol Royal Infirmary Inquiry, we confirmed serious concerns around the surgical outcomes at Bristol, and established the usefulness of routine administrative data (Hospital Episode Statistics) in helping to identify quality of care issues (2). In further research commissioned by the Shipman Inquiry and published in 2003, we established the role that statistical process control charts (specifically log-likelihood CUSUM charts), and other routinely collected data (from death certificates) could play in the continuous surveillance of healthcare outcomes, and in this specific case, the detection of unusual patterns of patient mortality within General Practices (3). Further research using routinely collected hospital data have demonstrated the comparable (or better) coverage and completeness of routine data compared to clinical audit data (4). We have also demonstrated the strength of risk prediction models based on hospital administrative data compared to clinical data (5).

We have developed indicators of healthcare performance, some of which were aimed at the general public and were first published in national newspapers in 2001 based on hospital mortality, patient safety indicators, and more recently stroke care and returns to theatre. We have also developed a national surveillance tool, the Real-Time Monitoring System (RTM as it is known), designed to monitor hospital outcomes across a range of diagnosis and procedure groups in near real time with data updated monthly (6). More recent research carried out by the unit since 2007 has refined this system, setting thresholds based on false alarm rates within CUSUM charts for multiple institutions, the automation of multiple risk adjustment models, the incorporation of hierarchical modelling techniques, the refinement of co-morbidity indices and the development of new indicators with potentially greater sensitivity than mortality.

References to the research

Key funding:

• Bristol Royal Infirmary Inquiry (1999-2000; £72,080), Principal Investigator (PI) P. Aylin, Analysis of HES data.

• The Shipman Inquiry (2001-2002; £96,190), PI P. Aylin, Monitoring of mortality rates in Primary Care, The Shipman Inquiry.

• Dr Foster Intelligence (2002-2006; £988,830), PI P. Aylin, Explanatory variables for regression analysis to explain variations in mortality rates in medium and large acute hospital trusts across England.

• Dr Foster Intelligence (2006-2010; £2,034,235), PI P. Aylin, Explaining variations in outcome in healthcare across England.

• National Institute of Health Research (NIHR; 2007-2012; £4,499,500), Co-Principal Investigators (Co-PIs) C. Vincent and P. Aylin, Research Centres for NHS Patient Safety and Service Quality.

• Rx Foundation (2008-2012; £550,248), Co-PIs B. Jarman and P. Aylin, The Rx Foundation proposal.

• Dr Foster Intelligence (2010-2015; £2,485,273), PI P. Aylin, Explaining variations in outcome in healthcare across England.

• NIHR (2010-2014; £372,061), Co-PIs A. Bottle and P. Aylin, Can valid and practical risk-prediction or casemix adjustment models, including adjustment for co-morbidity, be generated from English hospital administrative data (Hospital Episodes Statistics)?

• NIHR (2012-2017; £7.5M), Co-PIs C. Vincent and P. Aylin, Patient Safety Translational Research Centre.

Details of the impact

Impacts include: health and welfare, public policy and services, society, economy Main beneficiaries include: NHS, patients, Care Quality Commission, Department of Health

Our methodological research forms the basis of a near Real-Time Monitoring System (RTM as it is known) produced by Dr Foster Intelligence and is currently used by 70% of English NHS acute trusts to assist them in monitoring a variety of casemix adjusted outcomes at the level of diagnosis group and procedure group [1]. Dr Foster Intelligence is an independent healthcare information company and joint venture with the UK Department of Health. It provides a research grant to the unit to develop indicators and methodologies to assist in the analysis of healthcare performance.

We work with the Care Quality Commission, contributing to its surveillance remit using the same methods and data to generate mortality alerts from within our unit since 2007, based on more extreme thresholds [2]. This mortality alerting system, which looks at all acute non-specialist NHS trusts in England, was pivotal in alerting the then Healthcare Commission to problems (between July and November 2007) at the Mid Staffordshire NHS Foundation Trust (investigation in 2009) [3]. The resulting Public Inquiry recognised the role that our work on HSMRs and our surveillance system of mortality alerts had to play in identifying Mid Staffs as an outlier [4]. Key recommendations made in 2013 reflecting our unit's work, are that all healthcare provider organisations should develop and maintain systems which give effective real-time information on the performance of each of their services, specialist teams and consultants in relation to mortality, patient safety and minimum quality standards [5]. A further recommendation is that summary hospital-level mortality indicators should be recognised as official statistics [6].

As a result of our leading role in the development of hospital mortality measures, in 2010 we were invited to contribute to a Department of Health Commissioned expert panel (Steering Group for the National Review of the Hospital Standardised Mortality Ratio) to develop a national indicator of hospital mortality [7]. The resultant Summary-level Hospital Mortality Indicator (SHMI; based in part on HSMR methods) is now a public indicator used by all acute trusts and guidance from Professor Sir Bruce Keogh suggests that a relatively "poor" SHMI should trigger further analysis or investigation by the hospital Board [8]. The recent review (published in July 2013) into the quality of care and treatment provided by 14 hospital trusts with consistently high mortality in either measure (with Professor Jarman on the Advisory Group), led to 11 out of the 14 trusts identified being immediately placed on special measures. The other three hospital trusts were also mandated to make improvements. Actions required included: immediate closure of operating theatres; rapid improvements to out of hours stroke services; instigating changes to staffing levels and deployment; and dealing with backlogs of complaints from patients. The review also informs the way in which hospital reviews and inspections are to be carried out with the recommendation that mortality should be used as part of a broad set of triggers for conducting future inspections [9].

An international system for comparing benchmarks for individual diagnoses and procedures based on our methods and developed with the unit is also used by Academic Health Science Centres in the USA, Australia, Holland, Italy and Belgium to stimulate international comparisons of treatment pathways and more detailed methods to compare systems. As an example, University Hospitals Coventry & Warwickshire NHS Trust (UHCW) collaborated with Yale-New Haven Hospital (YNHH) to reduce delays in treatment of acute myocardial infarction, leading to improved outcomes [10].

Sources to corroborate the impact

[1] Real Time Monitoring (RTM). Enabling providers and commissioners to benchmark and monitor clinical outcomes. http://drfosterintelligence.co.uk/solutions/nhs-hospitals/real-time-monitoring-rtm/. Archived on 24^th October 2013.

[2] Care Quality Commission Quarterly publication of individual outlier alerts for high mortality: Explanatory text (URL available at: http://www.cqc.org.uk/public/about-us/monitoring-mortality-trends). Archived at 24^th October 2013.

[3] Investigation into Mid Staffordshire NHS Foundation trust. Healthcare Commission 2009. Outcomes for patients and mortality rates. Pages 20 - 25
http://www.midstaffspublicinquiry.com/sites/default/files/Healthcare_Commission_report_on_Mid_Staffs.pdf. Archived on 24^th October 2013.

[4] Report of the Mid Staffordshire NHS Foundation Trust Public Inquiry 2013. Volume 1. Pages 458 - 466 http://www.midstaffspublicinquiry.com/report. Archived on 24^th October 2013.

[5] Report of the Mid Staffordshire NHS Foundation Trust Public Inquiry 2013. Volume 3, page 1671. Recommendation 262. Archived on 24^th October 2013.

[6] Report of the Mid Staffordshire NHS Foundation Trust Public Inquiry 2013. Volume 3, page 1673. Recommendation 271. Archived on 24^th October 2013.

[7] Development of the new Summary Hospital-level Mortality Indicator. Department of Health Website. http://www.dh.gov.uk/health/2011/10/shmi-update/. Archived on 24^th October 2013.

[8] Indicator Specification: Summary Hospital-level Mortality Indicator. http://www.ic.nhs.uk/SHMI. Archived on 24^th October 2013.

[9] Review into the quality of care and treatment provided by 14 hospital trusts in England: overview report Professor Sir Bruce Keogh KBE. http://www.nhs.uk/NHSEngland/bruce-keogh-review/Documents/outcomes/keogh-review-final-report.pdf. Archived on 24^th October 2013.

[10] Global Comparators brochure 2013.