Summary of the impact

This case study describes a significant new index used to monitor death rates in hospitals. The Summary Hospital Mortality Index (SHMI) was developed as a direct result of research carried out at the School of Health and Related Research (ScHARR). This was implemented nationally in October 2011 and the SHMI is now the main mortality indicator used by the NHS. Following publication of the high profile Francis Inquiry on Mid Staffordshire in February 2013, set up to investigate excess mortality in the Trust, the Government has used the SHMI to identify and target 8 further hospitals for investigation.

Underpinning research

Around 60% of all deaths occur in hospital and preventing avoidable deaths is an essential objective for health services. Since 2001, the Department of Health (DoH) used the Hospital Standardised Mortality Ratio, developed by commercial company Dr Foster. In 2010 the DoH decided it needed its own index and after a competitive tendering round, the University of Sheffield was commissioned to develop this. During the early part of 2011 the ScHARR team had regular meetings with the DoH in London and Sheffield as they worked on the project.

The research team was led by Mike Campbell (Professor of Medical Statistics) and all the team were based in ScHARR.

The research question was this: When explainable factors such as the age of the patient and method of admission are taken into account, is there unexplainable variation remaining between hospitals in mortality in hospital and 30 days after discharge?

The project was a major data processing and statistical exercise [R1]. It led to two papers in mainstream journals [R2,R4] and an international speaking invitation [R3]. It involved linking hospital and national mortality data from 2005 to 2010 from the Office of National Statistics (92 million records). There were considerable logistical problems to be overcome in fitting statistical models to such large data sets. The analysis used data from all deaths both in hospital and within 30 days of discharge. The 30 day post discharge period is to try and discourage hospitals from discharging patients who are likely to die within 30 days so that the deaths no longer count in their mortality figures. The key insight was that certain hospitals had mortality rates well above that which could be accounted for by chance, even when known risk factors were accounted for.

The key innovations are:

• The SHMI uses all deaths in hospital and deaths 30 days after discharge rather than only 80% of in-hospital deaths used by the Dr Foster model
• It is robust to practices such as coding patients as `palliative care'
• The limits are calculated using a `random effects' model
• It is updated every three months, but with the weights calculated from 12 months of data, which means that seasonality is not a problem.
• The weights are available on the NHS Information Centre Website and so are open to scrutiny (unlike other competing models).

Our report's specification was accepted by the DoH and was implemented nationally in October 2011.

References to the research

Details of the impact

The impact of this research is on national policy and public services

Pathway to impact
A report was produced in April 2011 [R1] and this was presented to the DoH Hospital Mortality committee chaired by Sir Bruce Keogh (Medical Director of the NHS) in May 2011.The key insights were that a relatively simple combination of age, sex, diagnosis on admission, method of admission (emergency or elective) and comorbidity score produced a useful and stable model which could be used to calculate how many deaths would have been expected in a hospital, given its particular case-mix. Using random effects model and funnel plots to identify unexplained variation, limits were drawn for the ratio of the observed to expected number of deaths, and above or below these limits a hospital was deemed to be an `outlier'. A number of hospitals were thus identified.

We recommended that mortality indicators should not be used on their own at one point in time, but rather over a period of time, and with due consideration of the changes to both the observed mortality and the expected mortality. We also advised that the index should be used in conjunction with other indicators, and that avoidable mortality was only a small proportion of the total deaths in a hospital.

The hospital morbidity indicator developed by Sheffield has been rolled out across the NHS. Our SHMI enables robust and transparent monitoring of hospital performance and enables early identification of possibly failing hospitals so that investigations and remedial action can be taken. In the long run patients will benefit from better hospital care.

ScHARR's contribution is highlighted by the following statement on the NHS Information Centre website:

"The Department of Health are committed to implementing the SHMI as the single hospital-level indicator for the NHS in England and have commissioned the Health and Social Care Information Centre to produce it. This decision was based on the recommendations from the national review of the Hospital Standardised Mortality Ratios (HSMR) with independent statistical modelling work commissioned by the Department of Health and carried out by the School of Health and Related Research (ScHARR) at the University of Sheffield." (http://www.hscic.gov.uk/SHMI .)

Dissemination
The SHMI was disseminated via a major conference in London in May 2012:
http://www.healthcareconferencesuk.co.uk/reducing-measuring-avoidable-mortality attended by managers and clinicians from Hospital Trusts across England and by the Department of Health.

Beneficiaries:
DoH — accurate and independent monitoring
The Francis Report on the Mid-Staffordshire Hospital Trust in Feb 2013 showed that excess mortality for the Trust was associated with poor care. Subsequently 14 hospitals (many of which were also identified in our report of 2011) have been identified by the Department of Health as having unacceptably high mortality, over two years using the Sheffield SHMI, amongst other measures. The consequence of this is that the Care Quality Commission has sent teams in to investigate the care of patients at these hospitals and this has been reported in the Keogh report (2013) which will ultimately impact on staff, patients and hospital systems with the aim of improving patient outcomes.

NHS Trusts — can verify their results from published weightings
Previous hospital mortality indicators have been strongly criticised because of a lack of transparency as to how different case-mix variables were weighted. This has been solved because the weights for the SHMI are published on the NHS Information Centre's website and can be downloaded and used by individual Trusts to verify their SHMI.

International beneficiaries — learning from our research

As authors of the SHMI we were in a unique position of being able to advise on its use. In 2012 we gave a presentation on the use of the SHMI to the Catalonian Department of Health in Barcelona. In April 2013 five representatives from The Netherlands National Statistics Group came see us to discuss comparisons with Dutch methods and we have also had a visitor from Milan to discuss Italian methods. This may lead to attempts to standardise these measure in parts of Europe.

Sources to corroborate the impact

S1. The SHMI and the HSMR were used in the Keogh report, to identify 14 `at risk' hospitals
http://www.nhs.uk/NHSEngland/bruce-keogh-review/Documents/outcomes/keogh-review-final-report.pdf (p.3, 13, 16)

S2. The Francis Report into the mid Staffordhsire Hospital Trust states (p.457) "There is now a consensus that significantly high HSMR/SHMI results should trigger a serious consideration of whether poor care is an explanation for them"
http://www.midstaffspublicinquiry.com/sites/default/files/report/Volume%201.pdf (p.461)

S3. Its development was widely reported, for example News Medical `University of Sheffield's ScHARR team develops new index to measure hospital mortality rates'
http://www.news-medical.net/news/20110610/University-of-Sheffields-ScHARR-team-develops-new-index-to-measure-hospital-mortality-rates.aspx

S4. Medicalxpress `New hospital mortality rate index to be used across UK'
http://medicalxpress.com/news/2011-06-hospital-mortality-index-uk.html

S5. The fact that it has been used to identify failing hospitals was widely reported for example by the Daily Telegraph, which named eight hospitals identified by the SHMI as having high death rates.
http://www.telegraph.co.uk/health/healthnews/9824260/Persistently-high-death-rates-at-eight-large-hospitals.html