Summary of the impact

Applied psychology research undertaken at City University London has had a major impact on improving high stakes medical selection. Previously doctors were selected by curriculum vitae and unstructured interviews which resulted in bias and discrimination in selection. The research has provided evidence for using new selection methodologies: machine-marked tests (job knowledge and situational judgment tests) and assessment centres, informed by best practice in Occupational Psychology. These apply to all medical specialties (e.g., surgeons and anaesthetists), several healthcare professions (e.g., doctors, nurses, dentists) and selection points of entry (medical school through to specialty roles). Impact includes:

• UK Medical Royal Colleges and Medical Schools Council adopted the selection methods
• Major UK policy impact and significant cost savings for the NHS, the second largest organisation in the world
• Internationally, new methods were adopted due to the research outputs.

Underpinning research

The underpinning research for this case study was carried out at City University London by Fiona Patterson (Professor of Organisational Psychology 2003 to 2010, Visiting Professor 2010 to date), Dr Lara Zibarras (Research Assistant/PhD 2005 to 2011, Lecturer since 2008) and the Work Psychology Group, a research-led occupational psychology firm based in the UK. The underpinning research builds on earlier work conducted while Patterson was at the University of Nottingham, but focuses on significant new research undertaken at City from 2003. The research was conducted in partnership with senior medical figures such as directors of Postgraduate General Practice (GP) Education (e.g., Professors Simon Plint and Bill Irish, Former Chair and Chair, GP National Recruitment Office) and Professor Paul O'Neill (Medical Schools Council); and key NHS stakeholders (Royal Colleges and Deaneries) to design and evaluate improvements to medical healthcare selection processes. This partnership developed over several years by focusing on the needs of the stakeholders. Previously there had been little validated research investigating selection issues in medical education and training. Most selection entailed a panel interview focusing on technical and clinical competencies, with limited focus on non-clinical attributes such as communication, empathy, problem solving and team-working.

Research has been carried out in three areas: (1) Identifying selection criteria for success in job roles (via job analysis); (2) Designing and validating selection methods to measure these criteria; (3) Evaluation and validation of selection processes (fairness, reliability, validity).

Identifying selection criteria
Job analysis techniques identified theoretical competency-based models of the knowledge, skills, abilities and other attributes required for successful performance (e.g., Patterson, Ferguson & Thomas, 2008). Uniquely, these models included important non-cognitive attributes (empathy, integrity) required by healthcare professionals. The models have been tested through validity studies exploring the short- and long-term performance of doctors (Koczwara et al, 2012) and showed that non-cognitive attributes contribute to successful performance as a doctor. These new competencies were assessed during the selection of all doctors during annual selection processes.

Design and validation of selection methods
Patterson and her team have led the field, with extensive research to design and implement best practice selection methodologies for selecting healthcare professionals. Examples include:

• Design, implementation and validation of selection methodologies, including the first application of Situational Judgment Tests (SJT) and assessment centres in GP medical training (Patterson et al. 2009b)
• Design, implementation and validation of an assessment centre to select GPs (with the GP National Recruitment Office, see Patterson et al 2005, 2010, 2011).
• Developing the Royal College of Physicians' selection test methodology; including machine-marked job knowledge and SJTs for Clinical Medicine(Patterson et al. 2009a)

Research findings provided evidence to introduce innovative methodologies in medical selection such as SJTs and assessment centres. This has changed the way doctors are selected via all medical Royal Colleges in the UK (through the Academy of Medical Royal Colleges) and the Medical Schools Council. For example, the "foundation programme" (the two years of training that doctors are required to undertake following medical school) now uses SJTs as part of its selection process.

Validation and evaluation of selection processes
Validation activities have included (1) predictive validation studies; (2) a focus on candidate reactions and perceptions (i.e., face validity); and (3) the development of the new concept political validity. Due to the high stakes nature of medical selection, selection processes must be seen to be fair by candidates and by other stakeholders who hold potentially diverse views (political validity). Research outputs from this case study have shown the process to be fair from the point of view of applicants and to predict future job performance (Patterson et al. 2009 a, b).

References to the research

Patterson's research group has published more than 20 articles underpinning this impact. The underpinning research is supported by an exceptionally strong evidence base widely published in the medical (e.g., BMJ, Medical Education) and psychology literature (Journal of Applied Psychology, International Journal of Selection & Assessment), all of which are highly regarded peer-reviewed journals. In particular, Medical Education is a leading international journal for health professionals; its readership includes medical educators, teachers and researchers globally so it reaches key stakeholders.

c. Patterson F., Carr V., Zibarras L., Burr B., Berkin L., Plint S., Irish B. & Gregory S. (2009). New machine-marked tests for selection into core medical training: Evidence from two validation studies. Clinical Medicine, 9(5) 417-420 http://www.ncbi.nlm.nih.gov/pubmed/19886098

d. Patterson F., Baron H., Carr V., Lane P. & Plint S. (2009) Evaluation of three short-listing methodologies for selection into postgraduate training: the case of General Practice in the UK. Medical Education, 43 50-57 10.1111/j.1365-2923.2008.03238.x This paper won the "Silver Quill Award" in 2013 for the most downloaded research article in Medical Education in 2012.

f. Patterson F., Zibarras L., Carr V., Irish B. & Gregory S. (2011) Evaluating candidate reactions to selection practices using organisational justice theory. Medical Education, 45 289-297 10.1111/j.1365-2923.2010.03808.x

This research was initially funded by several grants:

Two funded PhD studentships (2003, Ruth Price, ESRC funded; 2005, Lara Zibarras, City University London Psychology Department studentship).

(2008 to 2009) Fiona Patterson: South York and Humberside workforce development, £150,000, New Competency-based Selection Systems for Secondary Care.

(2011 to 12) Lara Zibarras was awarded £4,000 by the Work Psychology Group Ltd for consultancy.

Details of the impact

Prior to this research, an unstandardised and unstructured approach to selection and assessment of healthcare professionals existed and resulted in an unfair and biased way of selecting individuals. The underpinning research has identified clear criteria with which to identify and select healthcare professionals and the evidence to implement better and innovative selection and assessment methodologies (e.g., SJTs, Assessment Centres) that are proven through validation and evaluation techniques to identify the best individuals for the job roles.

The major impact of this research has been:

1. Improved selection criteria: Patterson and her colleagues have conducted several job analyses across a range of specialties, initially with GPs and Surgeons, but extending to Obstetrics & Gynaecology; Psychiatry; Public Health; Paediatrics; and Histopathology.
2. Substantial improvement in the assessment and selection practices for doctors through the design and widespread implementation of new selection methodologies. The SJTs designed and validated by Patterson and her colleagues have been used since 2006 for selection into postgraduate training, including General Practice and Public Health. SJTs have been piloted for other specialties including Surgery, Radiology, Histopathology, Core Medical Training, Anaesthesia, Dentistry and Acute Specialties. In addition, Patterson and her colleagues have designed assessment centres which are used across different specialties: Neurosurgery; Paediatrics; Public Health; and Obstetrics & Gynaecology^g.

Patterson's profile has garnered invitations to advisory roles and appointments. Initially, she was invited to become assessment and psychometric advisor for the Royal College of General Practitioners, UK (2005 to 2010) and for the Royal College of Surgeons, England (2006 to 2011). During this time she was also advisor to the UK Department of Health and the Academy of Medical Royal Colleges on assessment and selection (2006 to date). Patterson and her research team therefore worked in close collaboration with Royal Colleges and the Department of Health to develop and implement the selection methodologies. This meant that they were not only relevant to the target audience but gained significant buy-in from these key stakeholders.

As a result of Patterson et al.'s work in developing selection and assessment methodologies, following publication of the Government White Paper, Liberating the NHS, in 2010, she was invited by the Royal College of GPs to direct research and evaluation to define knowledge, skills, abilities and attributes required for future UK GPs. In May 2012 she was awarded an Honorary Fellowship by the Royal College of GPs in recognition of her contribution to selection for GP specialty training. In 2010, Patterson was invited by Professor Sir John Tooke and the Medical Schools Council to chair an International Expert Panel to review selection for entry into foundation training in the UK (the point after medical school when doctors do two years of generalist training). The work was conducted in collaboration with the UK Medical Schools' Council and resulted in the use of SJTs for selection into foundation training.^h,i,j,k

The extensive work conducted in the UK led to Patterson being an invited expert in the development of an international policy statement on Assessment for Selection in Healthcare Professions; Ottawa Conference on the Assessment of Competence in Medicine and Healthcare Professions, Miami, USA, 2010. In addition, she was invited to be Selection/Assessment Advisor to GP Education Training (GPET), Canberra, Australia (2009 to date) and was invited to advise on selection methodology being adopted internationally (e.g., Royal Colleges of different medical specialties, Australasia; Australian GP Education & Training; Netherlands; Denmark; and Asian-Pacific network including Malaysia and Indonesia).

Methodologies implemented as a direct result of the research include:

• July 2012: Australian General Practice Training document outlining use of SJTs in Australia, referencing Patterson et al's work:
  www.vma.com.au/documents/VMA_AGPT_Program-FAQ_2012.pdf.
• 2011 — Medical Journal of Australia, Roberts and Tongo outline selection for GPs, based on UK national "selection-centre" approach:
  www.mja.com.au/journal/2011/194/2/selection-specialist-training-programs-approach-general-practice.

The beneficiaries of the work include the candidates going through the selection process, the administrators of the process and key stakeholders such as the Medical Royal Colleges and patients.

Candidates: Since better selection methods now exist, tens of thousands of candidates (medical school students, doctors, nurses, dentists etc.) now benefit from improved fairness and evidence- based transparent selection processes which also have a positive influence on the diversity of health-related professionals^l. Improvements to the selection process also mitigate the longer-term costs of appointing doctors who do not complete training or follow a medical career.

Recruiters/professional staff: Currently tens of thousands of applicants go through selection processes each year. Taking General Practitioners alone, 8,000 candidates sit the new assessments annually (initially adopted in 2006). Selection using the new procedures represents a significant annual cost saving for the Department of Health/NHS and time saving for recruiters and professional services staff. Plint and Patterson (2010) estimate the cost of the new machine- marked tests used in the GP selection process as £20 per applicant, where the previous hand- marked application form questions cost £50 per applicant. The changed approach reduces costs by £240,000 annually. This saving is even more significant when extended beyond General Practice to Surgery, Radiology, Histopathology, Core Medical Training, Anaesthesia, Dentistry and Acute Specialties.

Key Stakeholders: The underpinning research has provided evidence to implement various selection criteria and assessment methodologies. This has led to a major change in UK policy where healthcare professionals are increasingly being assessed and selected using SJTs and assessment centres^g,h,i,j,k,l with beneficiaries including the Royal Colleges with whom the methods have been developed and validated, including the Royal College of General Practice, Royal College of Physicians, Deaneries (e.g., Kent, Surrey & Sussex, Yorkshire and the Humber, Oxford) and the National Recruitment Office for General Practice Training.

Patients and the general public: the quality of care for patients has improved through the reform and improvement of medical education and training. Identifying the right person for the job ensures better patient outcomes and experiences.

The impact of this work is therefore far reaching and highly significant in ensuring that appropriate staff are chosen for critical roles.

Sources to corroborate the impact

g. March 2010: Selection in Speciality Training. Report supporting the move towards the use of assessment centres (refers to Patterson and colleagues' work): www.gmc-uk.org/Final_Selection_report_PMETB_Board_March_2010__2_.pdf_34128269.pdf.

h. August 2011: Final Report of Improving Selection to the Foundation Programme Project Group. Outlines the work conducted by Patterson and colleagues with regards to: Job Analysis; and development and piloting of situational judgement tests for foundation doctors:
www.medschools.ac.uk/SiteCollectionDocuments/Final%20Report%20of%20ISFP%20Project%20Group.pdf.

i. May 2012: Final report of Foundation recruitment exercise — Medical Schools' Council report explaining introduction and use of SJTs for foundation doctors, live implementation in 2013. www.medschools.ac.uk/AboutUs/Projects/studentfitnesstopractise/resources/Documents/Final_report_of_PRE_Full_Appendices.pdf and details of the SJT development and validation: www.isfp.org.uk/AboutISFP/Pages/default1.aspx.

j. All applicants to Foundation Programme from August 2013 will complete an SJT, based on Patterson et al's work: www.foundationprogramme.nhs.uk/pages/home/how-to-apply/SJT/EPM.

k. February 2013: First SJT for Foundation Programme applicants a success: hailed an "effective and innovative way of testing applicants' attributes". Professor Derek Gallen says, "I am delighted that the SJT has been such a success in its first year as part of the application..." www.mddus.com/mddus/news-and-media/news/february-2013/first-sjt-test-%E2%80%9Ca-success%E2%80%9D.aspx.

l. February 2013 — General Medical Councils report argues that using best practice selection methodologies will widen access and participation into medical school. www.gmc-uk.org/Identifying_best_practice_in_the_selection_of_medical_students.pdf_51119804.pdf — References Patterson and colleagues' work as best practice selection methods.