Summary of the impact

Statistical analysis and methodological development carried out by Imperial College London on data from the Bristol Royal Infirmary Inquiry and the Shipman Inquiry have led to new monitoring systems in healthcare. Using routinely collected healthcare information, we have highlighted variations in performance and safety, impacting the NHS through direct interventions and/or policy change. For example: (i) findings and recommendations arising from our research for the Bristol Inquiry were reflected in the final inquiry outputs, which highlighted the importance of routinely collected hospital data to be used to undertake the monitoring of a range of healthcare outcomes, (ii) a range of monitoring recommendations have arisen as a direct result of the research on data from the Shipman Inquiry, (iii) analytical tools based on our methodological research are used by managers and clinicians in over two thirds of NHS hospital trusts, (iv) Imperial's monthly mortality alerts to the Care Quality Commission were major triggers leading to the Healthcare Commission investigation into the Mid Staffordshire NHS Trust.

Underpinning research

We have used routinely collected clinical and administrative data to examine variations in quality and safety in healthcare.

Work by Jarman et al published in 1999 first established that there was substantial variation in mortality between hospitals in England. Starting from work to look at paediatric cardiac surgical outcomes commissioned by the Bristol Royal Infirmary Inquiry in 1999 [G1], we confirmed serious concerns around the surgical outcomes at Bristol, and established the usefulness of routine administrative data (Hospital Episode Statistics) in helping to identify quality of care issues [1]. Three levels of analysis of increasing sophistication were carried out. The reasonable consistency of the results arising from different sources of data, together with a number of sensitivity analyses, led to conclusion that there had been excess mortality in Bristol in open heart operations on children under one year of age [1]. Paper [2] developed the underlying statistical methodology used for the Bristol Royal Infirmary work, including techniques to identify `divergent' as opposed to just `extreme' performance, and estimation of uncertainty intervals on hospital ranks. The potential statistical role in future programmes for monitoring clinical performance was also highlighted in this paper, including use of cumulative sums risk adjusted outcomes and the need for appropriate statistical adjustment when a large number of comparisons are made, to avoid the danger of excessive false positive results arising from the naive use of significance tests[2].

In further research commissioned by the Shipman Inquiry in 2001 [G2], we established the role that statistical process control (SPC) charts (specifically log-likelihood CUSUM, or cumulative sum control charts), and other routinely collected data (from death certificates) could play in the continuous surveillance of healthcare outcomes, and in this specific case, the detection of unusual patterns of patient mortality within General Practices [3]. This work required developing underlying statistical methodology for detecting unusual patterns of mortality [4]. We considered some of the methodological and practical aspects that surround the routine surveillance of health outcomes and, in particular, we focussed on two important methodological issues that arise when attempting to extend SPC charts to monitor outcomes at more than one unit simultaneously: the need to acknowledge the inevitable between-unit variation in `acceptable' performance outcomes due to the net effect of many small unmeasured sources of variation (e.g. unmeasured case mix and data errors) and the problem of multiple testing over units as well as time. We addressed the former by using quasi-likelihood estimates of over dispersion, and the latter by using methods based on estimation of false discovery rates. An application of this approach to annual monitoring `all-cause' mortality data between 1995 and 2000 from 169 National Health Service hospital trusts in England and Wales was presented [4].

Building on the statistical foundations established in [4], we have also developed a national surveillance tool, the Real-Time Monitoring System (RTM as it is known), designed to monitor hospital outcomes across a range of diagnosis and procedure groups in near real time with data updated monthly [5][G3]. RTM implements statistical procedures for setting alarm thresholds based on false alarm rates within CUSUM charts for multiple institutions, with automated multiple risk adjustment methods.

Key personnel:

• Professor N Best, Professor of Statistics and Epidemiology, Faculty of Medicine, School of Public Health, Imperial College London, 1996-present.
• Dr P Aylin, Clinical Reader in Epidemiology & Public Health, Faculty of Medicine, School of Public Health, Imperial College London, 1997-present
• Dr Alex Bottle, Senior Lecturer in Statistics, Faculty of Medicine, School of Public Health, Imperial College London, 1998-present

References to the research

(* References that best indicate quality of underpinning research)

Grants:

[G1] Bristol Royal Infirmary Inquiry (1999-2000; £72,080), Principal Investigator (PI) P. Aylin, "Analysis of HES data".

[G2] The Shipman Inquiry (2001-2002; £96,190), PI P. Aylin, "Monitoring of mortality rates in Primary Care, The Shipman Inquiry."

[G3] Dr Foster Intelligence (2006-2010; £2,034,235), PI P. Aylin, "Explaining variations in outcome in healthcare across England"

Details of the impact

The research described in section 2 has increased the use of data and statistics in the management and monitoring of healthcare in the UK. Imperial's work has led to the development of innovative statistical and computational methods for processing large data sets derived from electronic medical records and NHS databases.

Findings and recommendations arising from our research for the Bristol Inquiry were reflected in the inquiry outputs, with the importance of routinely collected hospital data highlighted in Ian Kennedy's final Bristol Royal Infirmary Inquiry Report in 2001, `The Report of the Public Inquiry into children's heart surgery at the Bristol Royal Infirmary 1984-1995' [A]:

"From the start of the 1990s a national database existed at the Department of Health (the Hospital Episode Statistics database) which among other things held information about deaths in hospital. It was not recognised as a valuable tool for analysing the performance of hospitals. It is now, belatedly."

As a direct consequence of our work final recommendations of the Inquiry included:

"Steps should be taken nationally and locally to build the confidence of clinicians in the data recorded in the Patient Administration Systems in trusts (which is subsequently aggregated nationally to form the Hospital Episode Statistics). Such steps should include the establishment by trusts of closer working arrangements between clinicians and clinical coding staff."

"The Hospital Episode Statistics database should be supported as a major national resource which can be used reliably, with care, to undertake the monitoring of a range of healthcare outcomes." [A]

The Commission for Healthcare Improvement (CHI, now called the Care Quality Commission or CQC) took forward the recommendations of the Bristol Inquiry from July 2001, which to this day uses Hospital Episode Statistics to monitor healthcare performance (for example, `CQC indicators for mortality and emergency readmissions using Hospital Episode Statistics (HES)', May 2013 [B]).

Our contribution to the Shipman Inquiry was recognised in the final report by Dame Janet Smith [C, paragraph 14.27]:

"I am most grateful to Dr Aylin and his colleagues for the work that they have done for the Inquiry. It is innovative and, as I had hoped, it has made a real contribution to the debate about the feasibility and the value of setting up a system for the routine monitoring of mortality rates among the patients of GPs."

A number of recommendations arose as a direct result of this work [C, Recommendations, paragraphs 22-44, page 53] :

The Department of Health (DoH) must take the lead in developing a national system for monitoring GP patient mortality rates. The system should be supported by a well-organised, consistent and objective means of investigating those cases where a GP's patient mortality rates signal as being above the norm.

Every GP practice should keep a death register in which particulars of the deaths of patients of the practice should be recorded for use in audit and for other purposes.

PCTs should undertake reviews of the medical records of deceased patients, either on a routine periodic basis (if resources permit) or on a targeted basis limited to those GPs whose performance gives rise to concern.

The above recommendations are detailed further in Chapter 27, `Proposals for Change — The Use of Mortality Data as a Clinical Governance Tool: A National System of Monitoring' (27.105-27.107, [C]), and Chapter 14, `The Monitoring of Mortality Rates among the Patients of General Practitioners' [C]. Chapter 14 details the exact contribution of Aylin, Best, Bottle, Marshall and the Imperial team to the Inquiry (14.23-14.71, [C]). Chapter 14 described how "CUSUM charts could be used to monitor patient mortality rates at GP level and that they would have been capable of detecting Shipman if they had been in use at the relevant time" (14.65, [C]).

The methodology developed and published in [4] and [5] is the methodology that now underpins both (i) our Real Time Monitoring System which is currently used by 70% of English NHS acute trusts to assist them in monitoring a variety of casemix adjusted outcomes at the level of diagnosis group and procedure group [D], and (ii) the CQC mortality outliers programme that looks at patterns of death rates within NHS trusts and is used to generate the quarterly alerts of trusts with high mortality [E]. The process involves analysing data that suggests concerning trends in the death rate for specific conditions or operations, with the trends being calculated using SPC charts [F]. All of the outliers are calculated using patient-level data from hospitals which become part of a national HES system. Some outliers are calculated by Imperial College (Dr Foster Unit), while others are calculated by the CQC [F].

Imperial's mortality alerting system has also been pivotal in alerting the then Healthcare Commission (HCC) to problems at the Mid Staffordshire NHS Foundation Trust [G], which has been the centre of a number of investigations and national inquiries. Through the HCC's programme to analyse mortality rates in England, it received an unprecedented 11 alerts about high mortality at the trust, four of which were after the investigation was launched. Of the seven alerts that were received prior to the launch of the investigation, four came from the Dr Foster Research Unit at Imperial College as part of its analysis of data [G, Appendix E].

Sources to corroborate the impact

[A] `The Report of the Public Inquiry into children's heart surgery at the Bristol Royal Infirmary 1984-1995: learning from Bristol',
http://www.tsoshop.co.uk/bookstore.asp?Action=Book&ProductId=9780101520720 (report available here)

[B] CQC guidance document, `CQC indicators for mortality and emergency readmissions using Hospital Episode Statistics (HES)', May 2013,
http://www.cqc.org.uk/sites/default/files/media/documents/nhs_hes_qrp_data_item_guidancefo r_publication.pdf (archived here)

[C] Dame Janet Smith, `The Shipman Inquiry. Fifth Report — Safeguarding Patients: Lessons from the Past — Proposals for the Future', 9/12/04,
http://webarchive.nationalarchives.gov.uk/20090808154959/http://www.the-shipman-inquiry.org.uk/fifthreport.asp (PDF archived here). See Recommendations (pp49-65), Chapter 14 (The Monitoring of Mortality Rates among the Patients of General Practitioners, The Inquiry's Approach: The Commissioning of Work from Dr Paul Aylin and His Team, pp411-423), and Chapter 27 (Proposals for Change, The Use of Mortality Data as a Clinical Governance Tool: A National System of Monitoring, pp1123-1178)

[D] Real Time Monitoring (RTM). Enabling providers and commissioners to benchmark and monitor clinical outcomes. http://drfosterintelligence.co.uk/solutions/nhs-hospitals/real-time-monitoring-rtm/ (archived here on 26/11/13)

[E] CQC quarterly mortality outlier reports, http://www.cqc.org.uk/public/reports-surveys-and-reviews/reports/mortality-outlier-reports (archived at
https://www.imperial.ac.uk/ref/webarchive/z1f on 30/10/13).

[F] CQC mortality outliers programme (archived here) and explanatory text (June 2012, archived
here).

[G] Investigation into Mid Staffordshire NHS Foundation trust, Healthcare Commission, March 2009,
http://www.midstaffspublicinquiry.com/sites/default/files/Healthcare_Commission_report_on_Mid_Staffs.pdf (archived here).