Summary of the impact

Research carried out at the University of Aberdeen has addressed important clinical needs in neuropsychology/ clinical psychology. The work has provided large sample normative data for psychological tests which clinicians can use for comparison with a patient's test score. It has also resulted in methods enabling them to draw inferences on individual patient performance, and methods to use existing published data for reference purposes. All of these have been made available to clinical practitioners in user-friendly computer programs. Together, they have impacted directly on the quality of care for people with neurological and psychological conditions in the UK and worldwide, as well as the training of clinical neuropsychologists.

The resulting claimed impact has been on healthcare professional guidelines and training. Practitioners have used these research findings and tools in the conduct of their work.

Underpinning research

For psychologists and other clinicians, comparing a patient's score on some form of test (for example a measure of cognitive ability or mood) against a reference sample (often a healthy normative sample) is a fundamental process when conducting an assessment or working towards a diagnosis. There are three essential requirements to this process: first, adequate normative (reference sample) data must be available; secondly, sound inferential methods are required to compare the patient against this reference sample; and thirdly, the methods need to be made available to clinicians in user-friendly form.

It is on these three aspects that John Crawford, Professor of Psychology at the University of Aberdeen since 1996, has focused a systematic programme of research in collaboration with, primarily, Professors Paul Garthwaite from the Open University and David Howell from the University of Vermont.

In research projects carried out between 1996 and 2011, Crawford et al gathered normative data for mood scales (notably anxiety and depression) from large, broadly representative samples of the general adult UK population. The studies showed that demographic variables had only very modest influences on anxiety and depression scores, and that anxiety and depression scores are moderately correlated. Crawford et al concluded that the normative data gathered were suitable for clinicians to assess the rarity of a given anxiety/depression score [3.1, 3.2]. Further research (1996-2013) focused on the large, but largely untapped, reservoir of published data that can be used for reference purposes when assessing individual cases: Crawford et al developed methods (mainly based on regression, a statistical technique for estimating the relationships among variables) that allow clinicians to make use of such data [3.3].

Other aspects of the research (1996-2013) focused on the development of sound inferential methods. Crawford et al developed a large suite of novel, statistically sound and convenient quantitative methods to enable psychologists and medical practitioners to draw inferences over the performance of an individual patient [3.4-3.6]. These methods successfully address a number of problems clinicians may encounter, including the often small size of reference samples and resulting systematic bias in favour of overestimating the abnormality of scores [3.4]; the need to compare a case's profile of test results rather than just single scores [3.5, 3.6]; and the increasing awareness that clinicians should complement point estimates of a patient's standing on a given scale with estimates of the interval or stretch on the scale which the patient falls into [3.3].

Many of the quantitative methods Crawford et al developed are complex, and there was a risk that they would not actually be used in practice. To overcome this problem, Crawford made it an integral feature of his research programme to create tailored computer programs that implement a variety of psychometric and statistical methods for use in clinical research and practice. These programs are specially designed to be user-friendly for practitioners and made freely available to them [e.g., 3.3 to 3.6]. The URL of the main web page for these programmes is http://homepages.abdn.ac.uk/j.crawford/pages/dept/psychom.htm.

References to the research

This paper (and the next) is one of a series of four papers published in BJCP on assessment of anxiety and depression. Of more than 550 papers published in BJCP since 2000, these papers are ranked 1, 2, 3, & 4 for citation impact in Web of Science (WoS). The above was named the most influential article published in the period 2000-2005.

One of a series of four papers (1998-2012; three of which are published in APA journals) developing inferential methods for the use of regression in the individual case.

A foundational paper for this approach: the method has now largely replaced the previously widespread practice of using z scores when comparing a case to a reference sample (cited > 325 times in WoS as of June 2013).

This paper provided a sound classical statistical solution to the problem of testing for a difference between a patient's scores on two measures (e.g, verbal versus spatial memory scores etc).

This paper addressed the problem of estimating base rates of low scores and large differences when using multiple tests. It has since been used by the group and independent researchers to provide base rate data for a wide number of psychological test batteries including the HVLT-R, WJ-III, WAIS-IV, WISC-IV, D-KEFS, and RBANS. Moreover a number of independent studies have evaluated the method for use in clinical practice and report that it is sound.

Details of the impact

The research described has had direct impacts on clinical practice in the UK and abroad, the development of commercially available test batteries, and the training of neuropsychologists. More than 70 user-friendly computer programs for clinicians developed by Crawford on the basis of his research [e.g., 3.3 -3.6] have been made available free of charge on the author's website. Crawford received at least 800 emails from users between 2008 and July 2013, enquiring about his quantitative methods and computer programs, including numerous unsolicited comments testifying to their clinical use, such as "Re-visiting your site to download .exe's to my new computer. Thanks for all the stunning work. Most helpful"; and "I have been finding your website and resources absolutely fantastic, and as a clinician have recommended them to others. Thanks for all your wonderful work, we appreciate it down under!"; and "I just wanted to take a minute to tell you how much I appreciate the contribution you have made to the field with your work on statistical analyses of psychometric change, especially as it applies to neuropsychology" (representative email log available on request — [5.7]).

The methods developed have also been used in clinical practice in the following ways. Since 2008, the generic methods have been used in at least 500 published clinical case studies, primarily in North America and Europe. As only a very small fraction of cases seen clinically warrant a subsequent write-up as a case report, it is safe to assume that the methods have been used with many more clinical cases. Furthermore, many of the methods have been tailored to specific psychological test batteries which would not routinely find their way into single case studies, but are very widely used in clinical practice. These include the D-KEFS (Delis-Kaplan Executive Function System); the WAIS-IV (Wechsler Adult Intelligence Scale, Fourth Edition); and the RBANS (Repeatable Battery for the Assessment of Neuropsychological Status). The take-up of the normative data provided by the research is hard to quantify precisely, but again, the widespread use of these data in clinically oriented research papers is a strong indication that it is widely used. Google Scholar records over 1900 citations (as of June 2013) to the normative data for self-report mood scales.

Major clinical textbooks / training manuals on assessment in clinical neuropsychology [e.g., 5.a] provide extensive coverage of this group's methods. Given the central role of these textbooks in clinical neuropsychology training and practice, and their practical orientation, these endorsements will have led to the use of the methods in clinical practice.

As a result of his research expertise, Crawford has been called upon to act as statistical/clinical consultant and/or author for most of the major psychological tests used routinely in the UK. Recent examples include the above-mentioned WAIS-IV^UK, 2009; the Wechsler Memory Scale — Fourth Edition (2009); the Rivermead Behavioural Memory Test — 3^rd Edition (2009); the Test of Premorbid Functioning (2011); The Functional Living Scales (2012); and the Spot-the-Word Test Second Edition (2012). These tests are used daily in clinical practice, and as they are commercially available, the work carried out by Crawford et al has also led to commercial impact.

Eminent clinicians in the UK and abroad have confirmed the significance of Crawford's work for clinical practice. From the US, Gordon Chelune, Professor of Neurology at the University of Utah, reports (March 2013) [5.8] that the pioneering emphasis of Crawford's work has helped change neuropsychological practice as well as the way neuropsychology is taught. He highlights the fact that a publisher has asked him to write a book for clinicians on the application of Crawford's methods. Jonathan Evans, Professor of Applied Neuropsychology at the University of Glasgow, confirms that Crawford's work has had a "very significant impact" on the clinical neuropsychology profession and has "impacted directly on quality of care for people with neurological and psychological conditions in the UK and around the world" (March 2013) [ 5.9].

Since 2008, Crawford has led a number of training and continuous professional development (CPD) events for clinical neuropsychologists and similar practitioners based on his research expertise. These have included annual CPD workshops on psychological assessment for the British Psychological Society, which attract around 25 participants each year. Feedback collected since 2009 has been extremely positive, with one delegate speaking for many when s/he wrote, "I'm not good at stats and quantitative issues but I actually understood this. Found it really useful and thought provoking in terms of rethinking how I use tests and interpret results." Another delegate wrote "Superb! I wish I had been taught this in my clinical training". Other CPD events since 2008 have included an annual post-qualification course for clinical neuropsychologists and psychologists at the University of Glasgow. Delegates since 2010 (around 20 per course) have rated the event very highly at 4.7 out of a possible 5. Crawford also ran CPD workshops for the International Neuropsychological Society (Oslo, June 2012), and the Australian Society for the Study of Brain Impairment (Hobart, May 2013). These attracted over 75 delegates.

Claimed impact as defined by REF guidance: clinical guidelines have changed; professional standards, guidelines and training have been influenced by research; professionals have used research findings in conducting their research.

Sources to corroborate the impact

[5.1] Lezak, MD, Howieson, DB, Bigler, ED, & Tranel, D (2012). Neuropsychological Assessment (5th ed.). New York: Oxford University Press.

This is the standard reference work for clinical neuropsychological assessment. Reference is made to 21 of Crawford's first author papers. This count exceeds that of any other European psychologist.

[5.2] Atzeni, T. (2009). Statistiques appliquées aux études de cas unique: méthodes usuelles et alternatives. Revue de Neuropsychologie Neurosciences Cognitives et Cliniques, 1, 343-351.

This (French) review of how to make inferences concerning the performance of a single case is, in essence, solely concerned with Crawford and colleagues' methods; eight of the ten equations presented are those developed by Crawford and colleagues.

[5.3] Balboni, G, & Cubelli, R. (2011). How to use psychological tests for functional diagnosis: the case of assessment of learning disabilities. Advances in Learning and Behavioral Disabilities, 24, 79-92.

This (Italian) guide to assessment shows that Crawford's methods are now also having an impact in the area of learning disabilities. It recommends five of Crawford and colleagues' methods.

[5.4] Brooks, BL, Strauss, E, Sherman, EMS, Iverson, GL, & Slick, DJ. (2009). Developments in neuropsychological assessment: Refining psychometric and clinical interpretive methods. Canadian Psychology, 50, 196-209.

This review provides further evidence of the impact of Crawford's work on assessment in clinical practice. It recommends (and illustrates the use of) five of Crawford and colleagues' methods.

[5.5] Hanson, RK, Lloyd, CD, Helmus, L, & Thornton, D. (2012). Developing non-arbitrary metrics for risk communication: percentile ranks for the Static-99/R and Static-2002/R sexual offender risk tools. International Journal of Forensic Mental Health, 11, 9-23.

This recent (Canadian) paper illustrates that Crawford and colleagues' methods are now also having an impact in the forensic area (the methods were used with risk assessment tools).

[5.6] McIntosh, RD, & Brooks, JL. (2011). Current tests and trends in single-case neuropsychology. Cortex, 47, 1151-1159.

This review is focused almost exclusively on reviewing and recommending Crawford and colleagues' single case methods; it notes that "current practice has been shaped considerably by Crawford and colleagues' statistical refinements over the past 12 years" (p.1151) and notes that "Crawford and colleagues' tests are now the tests of choice for single-case comparisons" (p. 1155). It cites 15 of Crawford's papers.

[5.7] Representative log of unsolicited feedback and comments on the freely available computer programmes developed by Professor Crawford.

[5.8] Testimonial provided by Professor of Neurology, Department of Neurology, University of Utah School of Medicine.

[5.9] Testimonial provided by Professor of Applied Neuropsychology, University of Glasgow.

[5.10] Testimonial provided by Director of Critical Care and Neurosciences Research, Murdoch Childrens Research Institute, Australia.