Summary of the impact

A team led by a Manchester Metropolitan University researcher developed the PROBE (Project Review, Objective Evaluation) methodology for evaluating healthcare information technology systems. This became a core part of the National Health Service (NHS) information strategy; in 1996, PROBE was accepted by the NHS Executive and HM Treasury as a central method for project review. PROBE has therefore brought significant benefits to clinical users, patients and NHS decision makers. Impacts described here relate to the use of PROBE to evaluate an electronic blood transfusion system. Specifically, through enabling the effective evaluation of the system, decisions by a health authority have been informed by research, clinical practices have improved, and levels of patient safety and confidentiality have risen.

Underpinning research

From the late 1980's onwards, healthcare providers, commercial system developers and academic researchers began to focus on the potential benefits of IT systems in health. There followed a period of intense activity, during which many hundreds of health-related systems were developed and piloted in the UK, notably decision support systems (DSS) and electronic patient record systems (EPRS). However, systems often failed to provide the anticipated benefits, and, in some cases, were actively obstructive. Common problems included clinicians shunning systems in favour of paper-based practice, or systems actually introducing new errors into clinical practice.

By the early 1990's, Dr Heathfield Heathfield had been developing DSS and EPRS for some years, firstly as part of her Ph.D. studies at the University of Brighton and later as an IBM Research Fellow. Frustrated by the lack of success of healthcare IT systems, Dr Heathfield turned her attention to evaluation methodologies. She found that existing methodologies were not suited to healthcare IT systems intended to operate (a) in complex working environments, and (b) to high standards of safety and security.

Dr Heathfield subsequently began to study the implementation of healthcare IT systems, identifying the key barriers, incentives and lessons learned for successful systems. This research led to several high profile publications [1-4]. The National Health Service Information Authority (NHSIA) subsequently commissioned Dr Heathfield and colleagues to conduct specific research in evaluation methodologies for EPRS, followed by a project to develop a practical toolkit (PROBE: Project Review and Objective Evaluation) for evaluating EPRS [5-6]. The PROBE methodology makes knowledge and best practice in evaluation widely available in an easily accessible and usable format. It supports evaluations that are systematic, rigorous and consistent across projects, and therefore enables comparison and further learning. PROBE was recommended by the NHSIA for all IT projects, particularly those involving EPR and EHRS, and therefore formed a central part of NHS information strategy and policy [5].

Dr Heathfield undertook the research leading to the development of PROBE during her time at as a Senior Lecturer at MMU (September 1994 - 2002). She led the work in collaboration with colleagues from the Yorkshire Institute for Health Informatics (YICHI) at the University of Leeds, and Secta Ltd, a healthcare consultancy company. In 1996 she was funded by Cambridge University to spend a year as a Senior Visiting Fellow, directing the evaluation of a large-scale pilot of a General Practice DSS for a large pharmaceutical company.

Between 1998 and 2005, the NHS ran several large-scale national pilot projects of healthcare IT systems. Foremost among these was the Electronic Record Development and Implementation Project (ERDIP), a multi-million pound project which piloted EPRS in 12 NHS communities, and involved many thousands of clinicians and patients. Dr Heathfield and colleagues implemented several of the evaluation work packages using the PROBE methodology. The evaluation outputs played a fundamental role in the further development of EPRS, both as part of ERDIP and more widely. The resulting outputs further underlined the importance of good evaluation in healthcare IT, and (more importantly, for the purposes of this case study) emphasised the significant and central role of PROBE as an NHS-endorsed methodology.

References to the research

Work underpinning, motivating and complementing the development of PROBE is available as:

The NHS Executive Information Management group give an early summary of PROBE in:

The final PROBE methodology is described in the formal release document:

Details of the impact

Since the publication of PROBE in 2001, the NHS has included evaluation as a central element of major IT projects. Furthermore, as recommended in PROBE, evaluation is undertaken formatively in parallel with system implementation, rather than as afterthought when a system fails to deliver benefits. PROBE is seen as a popular evaluation methodology, and is widely used in the UK. A recent example of its application (which forms the basis of the claimed impact) is the Blood Safety Tracking System.

The PROBE methodology was used between 2007-2010 to design the evaluation of a pilot project intended to reduce the number of deaths due to inappropriate blood transfusions. Between 1996 and 2004, five patients died as a result of receiving incompatible blood during transfusions, and incompatibility contributed to the deaths of a further nine patients, and caused major illness in 54 patients [C]. In November 2006, the National Patient Safety Agency (NPSA) issued a national specification, giving guidance on how to ensure that patients receive blood that is correctly cross-matched for them, and titled "Electronic Clinical Transfusion Management System (ECTMS)". The NPSA set a goal of reducing the number of incompatible transfusions by 50% over 3-5 years (from January 1 2005) [A].

The NHS Connecting for Health Evaluation Programme (CFHEP) funded a pilot study at Croydon Health Services (Mayday) Healthcare NHS Trust, the main objective of which was the deployment of an Electronic Blood Tracking System that complied with the ECTMS specification.

PROBE was used (in conjunction with another methodology) to design the evaluation of this pilot study [B], which ran from 2007 until 2010. This evaluation process covered aspects such as the implementation and installation of the system, project management, changes in clinical and other working practices required to implement the system, the response of clinical and other staff to the system, patient perception, the efficiency, effectiveness and reliability of the system (particularly in terms of ensuring the correct matching of patients and blood) and any problems identified with the system or its use.

PROBE was used in [C] to "explore, guide and provide context to the quantitative analysis." This gave a detailed picture of stakeholder views and organisational processes involved in the pilot, and highlighted important factors such as the need for effective change management (almost 2000 people were involved in the new system), and the need for adequate supplier representation at high-level project meetings.

The pilot study overall final report, "Right Patient Right Blood", was published in September 2011 [D]. Specific benefits of the system identified in the report included improved patient safety (through identification during bedside checks), better monitoring of blood movement, and better management of blood stocks. A significant finding of the formal evaluation process [C] was that the time required to resolve "blood alerts" (corresponding to possible problems with transfusion) fell from 123 hours in the initial period of live system usage to around 30 minutes per alert by August 2010.

Head of Patient Safety at the NPSA, stated that "The NPSA welcomes this report on the pilot of the ECTMS ... The lessons learned will be useful for all NHS organisations implementing systems to improve the safety of blood transfusion." [E]

The precise impacts we claim for PROBE, therefore, include changes to the delivery of a public service, benefits from public service improvements, adoption of new processes, and improvements to patient health outcomes. The direct policy impacts of the work are still ongoing, as evidenced below:

Testimonial by The Head of Patient and Public Partnerships, NHS Connecting for Health (supplied in Autumn 2012):

"Evaluation of public sector Information and Communication Technology projects is often overlooked or done without due recognition of the complexity of the scheme. The work of Heather Heathfield developed the multifactorial evaluation approach PROBE (Project Review, Objective Evaluation) in two ways. Firstly, by focusing upon evaluation questions which are important to electronic health record projects. Secondly, by providing more detailed information about how to evaluate, including a review of the various tools and techniques available, with information to assist the reader in applying these appropriately. This greatly improved understanding of ICT projects. The approach is easily understood by lay people, and ensures that more equitable comparisons can be made. PROBE was used systematically by most of the 17 different projects that were part of the Electronic Record Development and Implementation Programme. This evaluation data is still being used, informing the development of the current Information Strategy (The Power of Information)" [F]. This information strategy from the Department of Health, published in 2012, sets a ten-year framework for transforming information for the NHS, public health and social care.

Sources to corroborate the impact

All public sources are linked from http://www.scmdt.mmu.ac.uk/cir/REF

[A] National Patient Safety Agency (November 2006). Electronic Clinical Transfusion Management System: Right Patient Right Blood.
http://www.connectingforhealth.nhs.uk/systemsandservices/clinsafety/projects/bloodpilot/mayday/ectms.pdf/at_download/file

[B] Information on the use of PROBE as part of the Blood Safety Tracking Pilot evaluation.
http://www.connectingforhealth.nhs.uk/systemsandservices/clinsafety/projects/bloodpilot/mayday/evaluation

[C] Kay, J. and Roudsari, A. (2011) NHS CFHEP 003 - Evaluation of the pilot implementation of an IT specification for a blood tracking systems.
http://www.birmingham.ac.uk/research/activity/mds/projects/HaPS/PHEB/CFHEP/reports/projects/003.aspx

[D] Croydon Health Services NHS Trust (2011) Right Patient Right Blood Pilot. Final Report. Report available at: http://www.scmdt.mmu.ac.uk/cir/REF

[E] National Patient Safety Agency response to ECTMS pilot.
http://www.connectingforhealth.nhs.uk/systemsandservices/clinsafety/projects/bloodpilot/about

[F] Department of Health (2012). The power of information: putting all of us in control of the health and care information we need. Published May 21 2012.
http://www.dh.gov.uk/en/Publicationsandstatistics/Publications/PublicationsPolicyAndGuidance/DH_134181

Personal statement available to view from:

[G] The Head of Patient and Public Partnerships, NHS Connecting for Health corroborating ongoing impacts of research on improvements to the NHS, including the development of ICT evaluations and the creation of information policy.