Summary of the impact

UCL research shows that UK ethnic minority (EM) medical students and doctors frequently underperform in both undergraduate and postgraduate examinations. These findings have been used to help ensure the safety of medical healthcare, particularly via contributions to debate and decision-making among policy-makers and medical education professionals. This has led to: the development of new or amended guidelines; changes to the ways in which international examinations are run; greater transparency in the sector's analyses of how ethnicity impacts on key outcomes; and subsequently greater transparency in the public dissemination of information relating to medical education's successes and shortcomings. The use of the research to inform relevant media discourse has likewise improved transparency about these shortcomings, as well as engaging a broad public audience with these and important related issues.

Underpinning research

Research into ethnicity and performance among British medical students has been conducted at UCL by Chris McManus (Professor of Psychology and Medical Education) since 1996, as part of a broader investigation of undergraduate and postgraduate medical training. Ethnic differences in medical school attainment had previously been demonstrated in the USA (where the historical, legal and political context is very different) and in UK compulsory school education, but had not been considered at all by UK medical schools. Research led by McManus and published in 1996 [1] used data from two large longitudinal studies (n=691) to demonstrate that, whilst non-UK ethnic minority (EM) medical students performed better than White students, UK EM students were 2.09 time more likely to fail machine-marked multiple-choice medical examinations. This publication challenged the attribution by medical schools (particularly that at Manchester University) that poor performance among EM students was due to ethnic discrimination by examiners.

Subsequent research within the Unit has built on these findings to address the question of why UK EM medical students perform less well, and to enhance understanding of broader issues pertaining to ethnicity in relation to medical school training, including the examination performance of qualified doctors. Since 2004 McManus has worked closely with Dr Katherine Woolf (joined UCL 2004), whose PhD research (co-supervised by McManus with Professor Jane Dacre, Director of UCL Medical School) included two cohort studies of students (n=729) entering UCL Medical School between 2001 and 2003. The results, published in 2011 [2], showed that none of a wide range of social, cognitive and personality measures could explain EM student underperformance. Crucially, McManus and Dacre had shown in 2007 that the ethnic gap persists even after doctors qualify and enter clinical practice [3]; in 2013 they proved that this it is not due to bias on the part of individual examiners [4]. UK-trained EM doctors underperformed overall in the Membership of the Royal Colleges of Physicians (UK) — MRCP (UK) — postgraduate examination for physicians, taken by some 24,000 doctors at 32 centres world-wide each year. Detailed analyses of candidate and examiner sex and ethnicity found only tiny aggregate effects, reconfirming their hypothesis that underperformance cannot be explained by examiner discrimination.

The research team's (2009) analyses of national data from the Youth Cohort Study of England and Wales and UCAS showed that EM school students applied for medical school despite slightly lower school achievement, which partially explained underperformance [5]. A widely invoked explanation of American EM underperformance is `Stereotype threat', and Woolf's 2008 qualitative study of UCL medical students and teachers did find evidence of negative stereotyping of EM students [6]. However, a subsequent (2009) randomised controlled trial by Woolf et al. of Geoffrey Cohen's much-cited anti-stereotype threat intervention did not improve students' grades [7]. Woolf and McManus responded to these findings by studying UCL medical students' social networks, where they discovered ethnically homogenous clusters of students and showed that closeness in social networks predicted similarity in examination performance [8]. Importantly, randomisation of students into tutorial groups by the medical school reduced ethnic clustering significantly. A major implication is that EM underachievement may be attributable to differences in social networks, with random allocation presenting an effective intervention strategy. The key findings of these and other studies were brought together and made more robust in Woolf and McManus's high-profile and influential 2011 British Medical Journal meta-analysis (n=23,742; Cohen's d= -0.42) [9].

References to the research

Funding: Earlier work on the cohort studies was Leverhulme Trust and Department of Health funded. More recent grants to McManus include: £42,661 from the DEPARTMENT OF HEALTH (02-FEB-98 to 31-JAN-99); two grants between 01-FEB-99 and 30-SEP-02 with a total value of £90,214 from THAMES POSTGRADUATE MEDICAL & DENTAL EDUCATION; £49,904 from the LONDON POSTGRADUATE MEDICAL AND DENTAL EDUCATION (01-NOV-02 to 31-MAR-04); 7 grants since 2005 totalling £143,450 from the LONDON DEANERY. Woolf received a £4,135 BRITISH ACADEMY grant (01-10-2011 to 01-10-2012) for the social network analysis work.

Details of the impact

The research findings outlined above have cohered into a vital critique that influenced the development of national and international medical education policy and practice. In the UK, it has had an important influence on medical and other higher education professionals, with some of its most significant impacts arising from the use of key research findings by the General Medical Council (GMC) — the most powerful body in UK medicine — and Royal College of Physicians (RCP), whose membership examination MRCP(UK) is taken by about 24,000 doctors worldwide every year and is mandatory for those specialising in internal medicine. In many cases, however, its beneficial effects have resulted more generalised use to compel enhanced transparency in the operation of UK medical education professionals.

Postgraduate institutions such as the MRCP(UK) were, for many years, extremely reluctant to publish data about EM student performance because they believed that this would impact negatively on candidates' confidence in the fairness of the examination and, in turn, on their own reputations. The publication in 2007 of [3] showed, on the contrary, that transparency only underlined institutions' commitment to equality. In January 2008 the MRCP(UK) decided to routinely and systematically collect data about the ethnicity of all candidates, examiners and boards, and to publish outcomes by ethnicity [a]. As well as allowing the UCL team's subsequent statistical analysis disproving bias in individual clinical examiners [4], this shift marked the influence of the research on the running of the MRCP(UK) Examination. Since 2011, the same approach has been replicated by other Royal Colleges: data on EM performance are now routinely published by the Royal College of General Practitioners (RCGP) — whose first publication of exam results in 2008 attracted considerable media attention and who explicitly cite our research in their FAQs [b] — and by the Royal College of Psychiatrists (RCPsych), the latter in relation to its membership examination. The Royal College of Physicians agreed in 2011 to publish a publically-available annual report on how gender and ethnicity relate to pass rates on MRCP(UK) and Specialty Certificate Examinations, again citing the UCL work in so doing [c]. On 8^th August 2012, in a document that was "the first of its kind", the GMC published various Royal College postgraduate examination results by country of training and ethnicity where available, with the specific aim of "promoting transparency" and "ensuring training is fair and based on principles of equality". In that document, the GMC responded to the incapacity of many Royal Colleges to provide ethnicity data by taking the unprecedented step of stating its intention to work with them to ensure its inclusion in future to facilitate analysis of ethnicity amongst UK-trained doctors [d].

Beyond its specific influence on examination practice and the publication of examination results, the research has achieved significant impacts by stimulating and informing debate among key stakeholders and policy-makers in fields related to the provision and regulation of medical education. The importance of the meta-analysis of research on EM underperformance [9] to the medical regulator was recognised explicitly by GMC Chief Executive Niall Dickson in a BMJ Careers article in March 2011. Here, he wrote: "we will be looking closely at [the study's] findings and the implications for our role as the regulator for all stages of medical education and training" [e]. The study was subsequently included as a key reference for the GMC's inaugural Being Fair conference on 12^th September 2012, where it was shared with approximately 50 influential stakeholders from policy-making and practitioner backgrounds. In addition to the inclusion of a summary of the paper in all delegates' information packs, it was cited explicitly by the GMC Director of Education and Standards Paul Buckley, and discussed in detail by participants in one of the event's three workshop sessions [f, g]. Attendees at the conference, which was chaired by the Parliamentary and Health Service Ombudsman Dame Judie Mellor, included the heads of the GMC, Chair of the British Association of Physicians of Indian Origin, Director of NHS Employers, Chair of the Legal Services Consumer Panel, former Chair of the BMA Equality and Diversity Committee, postgraduate Deans, and medical directors of Primary Care Trusts. Underscoring the study's relevance and utility to these diverse stakeholder parties, the GMC Deputy Chief Executive Peter Phillips drew on the findings summarised in delegates' packs to warn that the GMC was "losing the battle for hearts and minds" in delivering its statutory purpose, partly because of "the differences in outcomes and attainment rates for some groups of trainees and doctors (men compared with women, BME compared with their counterparts) in examinations" [f]. The GMC Chief Executive meanwhile, emphasised the need for "more analysis and sharing of data to understand what's happening, and to prevent problems occurring in the first place"; he pledged to analyse ethnicity data from the National Trainee Survey and Annual Review of Competence Progression (a summative assessment of all UK junior doctors) and publish the results in 2013 [e].

Through these and other media, the research has been used to transfer expert knowledge to professional standards bodies and thereby contribute to improving standards of professional practise. In September 2011 the GMC drew explicitly on the underpinning research in publishing its first report on `The state of medical education and practice in the UK', which acknowledges "unexplained differences in exam performance across all domains, including ethnicity", and recommends "further research to identify and address the causes of these differences" [h]. In the same month, the GMC's Undergraduate Board met to discuss three major proposals by the Medical Schools Council (the body representing the interests of all UK medical schools) to improve undergraduate medical assessment. Again, the GMC referred to the UCL findings in advising that Council to include an assessment of the ethnic equalities implications of these projects [i]. On the basis of his expertise in this area, McManus was invited to contribute to the British Medical Association's 2009 report on Equality and Diversity in UK medical schools [j]. In 2011 he was commissioned by the UKCAT consortium, who run the Clinical Aptitude Test for admission to 26 UK medical schools, to assess the test's validity in 4811 EM and non-EM students, and to explore differences between medical schools in the performance of EM students. The impact of the research on the priorities of professional standards bodies is further evidenced by the GMC's October 2012 statement of its intention to include an exploration of "the complex reasons for the gap in attainment" set out in [9] in a commissioned review of the effects of Tomorrow's Doctors 2009, the document in which it had previously outlined the knowledge, skills and behaviours required of UK medical schools students [k]. On 5 ^th March 2013, the GMC commissioned an independent data review of pass rates of UK and non-UK EM and non-EM doctors taking the Membership of the Royal College of General Practitioners exam, with UCL work cited in the independent report and the GMC's subsequent Terms of Reference review [l].

As well as engaging professionals involved in the provision of medical education and maintenance of professional medical standards with the sometimes uncomfortable evidence pertaining to EM underachievement, UCL research has also increased public awareness of the ethnic attainment gap. This was, in fact, one of the researchers' original aims: despite criticism for making their findings public in the days before Royal Colleges routinely published exam results on their websites, they published their MRCP(UK) data in an open-access journal in 2007 [3] precisely in order to share its findings with a wide public audience. The range of the work's impacts on public awareness of issues relating to EM underperformance — and of the need to redress the problems giving rise to that — has since been significantly expanded by international media coverage. That medical practitioners themselves, both within and well beyond the UK, have also engaged directly with the research findings is evident in the 41 rapid responses on the BMJ website from doctors across the UK, Australia, Taiwan, Nepal and the USA to Woolf's 2011 meta-analysis and 2008 qualitative study; the articles were also discussed on medical websites and blogs around the world. The Maori Health Review in New Zealand, for example, described the 2008 study as "noteworthy", adding: "The fact that teachers had the same negative `stereotypical' ideas about `Asian' students is somewhat unsettling, particularly if it impacts on knowledge transfer. It also confirms the need for clinical teachers to be culturally safe" [m].

Sources to corroborate the impact

[a] The impacts of the research [3] and the consequent decision to publish exam data publically are described in MRCP UK Newsletter: http://bit.ly/1dHPdDj p.1

[b] For citation of our work by the Membership of the Royal Colleges of General Practitioners in relation to its decision to publish examination data by ethnicity: http://bit.ly/195Ezml p. 4

[c] For reference to the influence of [9] in the Federation of Royal Colleges of Physicians' 2011 decision to publish pass rates by gender and ethnicity http://bit.ly/1dHPxSo p. 1

[d] The influence of the research on the GMC emphasis on ethnicity data is evident in the GMC Annual Specialty Reports Exam Summary 2010/11: http://bit.ly/1amkE4F especially para. 9 (p. 5)

[e] For reference to the research by GMC Chief Executive Niall Dickson: http://bit.ly/16IkzlN

[f] The GMC's 2012 Being Fair Conference report includes a summary of our meta-analysis as a key discussion document http://bit.ly/1aE3kXa especially p. 28

[g] For discussion of our work at the Being Fair conference: http://bit.ly/18y0K03

[h] For reference to the research in the 2011 GMC report The state of medical education and practice in the UK: http://bit.ly/1hpI2j1 p. 61

[i] For the GMC Education Board's recommendation that the MSC examine the link between ethnicity and attainment in light of our research findings: http://bit.ly/16iOgwG p. 9 para. 40

[j] McManus' contribution to the 2009 BMA report Equality and Diversity in UK medical schools is acknowledged in that report http://bit.ly/1aQDbks p. iii. References to his research appear on pp. 65 and 108-118 of the same.

[k] For reference to the research in the GMC Undergraduate Board paper (October 2012) outlining the terms for the review of Tomorrow's Doctors: http://bit.ly/17uhb1y p. 16

[l] For discussion of [9] as a key reference in the 2013 GMC Terms of Reference for the Review into the MRCP Examination: http://bit.ly/1dHPdDj pp. 3-4. For its citation in the associated review of ethnic differences in MRCGP examination attainment: http://bit.ly/185qiZ1 p. 37

[m] For an example of international media coverage of the research published in 2008 see article in the Maori Health Review (number 17, 2008) http://bit.ly/1dlhqNA p.2