Summary of the impact

A freely available, open access, decision-training web technology, for occupational therapists in community mental health services, has been developed and adopted internationally in professional training and practice. This decision-training aid, approved and advocated by the College of Occupational Therapy, has been extensively used in the UK and has commenced application internationally. NHS practice policies have been enhanced through their adoption of the research findings, facilitating service providers to target the most needy and improving the quality and efficiency of this profession's service. Increased workforce capacity has been demonstrated through the improved referral prioritisation skills of novice occupational therapists from numerous NHS Trusts and 27 pre-registration training programmes.

Underpinning research

Dr Priscilla Harries led the research, in collaboration with Dr Kenneth Gilhooly; they are both current academic staff at Brunel University. Dr Harries has been at Brunel for the duration of the research (2000- 2013) and impact period (2008-2013). Other contributing academics: Weiss and Shanteau (USA) assisted with the expert validation phase; Tomlinson (Imperial College London) developed the web based software and data storage system; Davies and Notley (RAs,Brunel) assisted with recruitment and analysis during the final randomised control trial (RCT). The overall aim of the research was to improve novice occupational therapists' decision-making skills in referral prioritisation.

Finite NHS resources require services to be targeted at the most needy. In addition, occupational therapy professional directives, on service delivery, require the majority of services be focussed on meeting service users' occupational needs as opposed to generic needs (College of Occupational Therapists, 1996); the purpose of services being to enhance health through active engagement in the occupations of self care, work and leisure. In order to identify how experienced occupational therapists' identify the most needy referrals for profession specific occupational therapy services, our research identified how experienced occupational therapist prioritised referrals and how those judgement policies differed between those carrying professional specific versus generic ways of working. Early stages of research were supported by two competitively awarded grants from Brunel University (£9k & £15k). Research was undertaken, using a judgement analysis methodology, to capture 40 experienced UK occupational therapists decisions on 90 referrals¹. Their decisions were used to statistically model how they used referral content when prioritising referrals. In order to identify the most expert policies amongst experienced therapists, repeats were used to measure discrimination and consistency of individual judgement policies. This phase of expert validation was the first application of an empirical expertise index on occupational therapists' clinical judgments². The policies were statistically clustered and caseloads examined to determine the nature of the service provided. These were compared with professional directives on service delivery, in order to identify the optimal referral prioritisation policies for best practice³.

The consensus standard, representing the optimal judgement approach, was then used to create an educational intervention. The intervention was initially tested for efficacy on a paper based task. Novices completed the prioritisation of 52 referrals, participated in the educational training, and then prioritised a second set of 36 referrals⁴. Correlation (r_p) between novices and experts' decisions of the same referrals showed an improvement from 0.26 to 0.7. The complete training package was then incorporated into a website for ease of access. Its final stage of testing, was a randomised controlled trial⁵, this was funded through following a successful application to a nationally competitive call from the Higher Education Academy (£5k). The research demonstrated a large effect (d = 1.19, r = 0.69): only novices in the intervention group were able to improve their decisions. The optimal judgement approach has been regularly updated through consultations with current practitioners as well as a 2012 replication of the 2001 expert data collection study. The research has now extended internationally; the web-training tool has been used to train novice practitioners internationally, for example 50 novices accessed the site in April 2013 in Queensland, Australia. New Zealand's clinicians have recently started collaborations with Harries to develop a bespoke training package for their novice practitioners; the first stage of data collection of experts' decisions on referrals commenced in Spring 2013.

References to the research

Details of the impact

Impact 1

A freely available, open access, decision-training web technology, for occupational therapists in community mental health services, has been developed and adopted in clinical training and practice. The web technology is highly effective therefore it has had impressive significance as a decision training aid for novice occupational therapists. Reach has been extensive in the UK and is a growing in terms of international impact. Clinicians regularly request to access the training website to support Continuing Practice Development NHS programmes e.g. Practice Development Lead for Occupational therapy in Glasgow and Clyde Hospitals. Pre-registration occupational therapists have used the training website across 95% of clinical training organisations in the UK. In terms of international use, over 100 novices in Australia and New Zealand have used the website. New Zealand have now commenced a replication study of the expert prioritisation policies capturing project, in collaboration with Harries, in order to identify expert practice in order that an equivalent tool for New Zealand's novice occupational therapists can be developed. The College of Occupational Therapists have hosted a link to the web based training tool on their website and approved and advocated the web based training tool for use.

Evidence is demonstrated through the following:

Large data usage statistics held by Imperial College IT on use of clinical decision aid Screenshot of website.

Emails from NHS Trusts, clinicians: "I have just used your training tool. It is very useful especially to highlight the basis of decisions made and goes to the heart of the generic/profession specific debate that has taxed us all in recent years" from experienced clinician; "Thanks for letting me use your web tool. It's a brilliant idea and I am very glad to have the chance to use it. I have learnt a lot and it has given me so much to think about" from novice clinician."

Users from 27 pre-registration NHS funded occupational therapy training organisations.
Emails from NZ proposing replicated study and from Australia stating requests to use the web based training with pre-registration occupational therapists College of Occupational Therapists (COT) website advocating use of site.

Impact 2

NHS and professional policies have been enhanced by the research.

The identification of differences in generic and profession specific ways of case working, as identified through the referral prioritisation study of experienced occupational therapists, was used to develop the College of Occupational Therapists mental health strategy entitled `Recovering Ordinary Lives: The Strategy for Occupational Therapy in Mental Health Services 2007-2017: Literature Review'. This review was subsequently used to underpin the professional strategic vision entitled `Recovering Ordinary Lives: The Strategy for Occupational Therapy in Mental Health Services 2007-2017: A vision for the next ten years', which has become a key practice driver for clinicians.

Findings that demonstrated the optimal approach to occupational therapy referral prioiritisation, have been used to develop several NHS Trust referral prioritisation policies e.g. Occupational Therapy Priorities for Community Treatment Teams, Northumberland, Tyne and Wear NHS Trust, (2012) and Referral Prioritisation Policy Dublin/Mid-Leinster (2010).

The workforce has continued to support the resource eg 18 experts and two NHS occupational therapy managers contributed to the further development and updating of the expert consensus. Their were involvement was supported by the College of Occupational Therapists -Specialist Section- Mental Health.

Evidence is demonstrated through the following:

COT document — College of Occupational Therapists `Recovering Ordinary Lives'; Literature review.
NHS Trusts requesting permission and advice on how to use the research findings in policy developments- e.g. Dublin, Gateshead.
COT Specialist Section recruitment email to request participants for the updating of the expert consensus policy capturing exercise.

Impact 3

Improved workforce capacity has been demonstrated. The quality of a professional service has improved through enhanced decision-making workforce capacity within the health professions. The randomised controlled trial demonstrated a large positive effect on referral prioritisation with average correlation with the average experts' prioritisation judgements moving from 0.23 to 0.58 (mean difference 0.35). Cohen's d (r) was 1.91 (0.69).

Evidence is demonstrated through the following:
Four of the research papers (2, 4, 5 & 6), as listed in references section, all of which demonstrated a large effect of the decision training on referral prioritisation capacity.

Impact 4

The quality and efficiency of the profession's services have been improved, which will facilitate services being targeted at the most needy and support staff retention. This impact is proposed as a future impact, which based on the impacts 2 & 3 above.

"I wanted to let you know that I used the knowledge gained from [your website training] to answer several hard questions [at interview] about prioritising caseloads. I got the job and I am convinced I wouldn't have been able to answer those questions without [your website]. Out of all the candidates, senior and junior, they had interviewed that day they said I was the best candidate! Again I am sure I wouldn't have been able to answer many of the questions without your [decision training] task."

There are over 34,000 practising occupational therapists in the UK. Occupational therapists in community mental health teams, which constitute approximately a fifth of the professional workforce (n6800), are receiving approximately 500 referrals a year, therefore 17,000,000 referrals would have been received in total over the years 2008 and 2012. The evidence based training improves occupational therapists' capacity to prioritise referrals by approximately 35% therefore potentially ensuring that the needs of 5,950,000 UK patients can be more successfully prioritised. The potential benefit to patients is apparent.

In addition to enhancing the referral prioritisation service for patients, well managed workloads and the effective use of services also enhances work satisfaction, which is necessary to promote staff retention and optimise the health of the NHS workforce. As training is funded by the NHS it is important that staff are retained in the workforce and sustained in working effectively by being satisfied that they are delivering an efficient and effective service.

Evidence is demonstrated through the following:
Continued use of the website.
Web users and experts in the field supporting value of training.

Sources to corroborate the impact

Impact 1

A freely available, open access, decision-training web technology, for occupational therapists in community mental health services, has been developed and adopted in clinical training and practice. Sources:

i) Website hosting domain for www.priscillaharries.com which is also embedded in www.referralprioriitsation.com

ii) Corroborating contact — Research and Development Office, College of Occupational Therapists agreeing to advocate the training website for member use. UK College of Occupational Therapists advocated use of the training tool for 34,000 members through online link.

iii) Corroborating contact — Web developer Senior Software Developer, Imperial College London to confirm development, testing and extensive use of the website.

iv) Corroborating contact — Education Projects Manger -effective Practitioner Initiative, NHS Scotland — used resources in CPD training programmes to supports level 5/6 NMAHP's to deliver person centred, safe and effective care.

v) Corroborating contact — Principal Lecturer, Otago, NZ re evidence of use with New Zealand occupational therapy students as well as commencement of project to develop a New Zealand `policy capturing' replication study to be used in developing a dedicated New Zealand decision training aid training site.

Impact 2
NHS policies have been enhanced by the research. Sources:

vi) COT document College of Occupational Therapists `Recovering Ordinary Lives' citing research.

vii) Corroborating contact- Lead AHP Urgent Care Adults re research used to develop referral priorities for the Community Treatment teams in both Gateshead.

Impact 3 Improved workforce capacity has been demonstrated. Sources:

vii) 4 research papers 2013.

Impact 4
The quality and efficiency of the profession's services have been improved, which will facilitate services being targeted at the most needy and support staff retention. Source:

viii) Corroborating contact — Web developer Senior Software Developer, Imperial College London to confirm website development, testing, effect of training and extensive use of the website.