Summary of the impact

This case study demonstrates how research conducted at Plymouth University on the recruitment and selection methodologies for postgraduate speciality training in medicine has impacted on the development and implementation of the recruitment process for core training and specialty training posts in medical related fields throughout the UK. The impacts take the form of a new and improved shortlisting methodology and model for selection centre recruitment at a national level. It overcomes the problems revealed with recruitment to medical training during the introduction of the Modernising Medical Careers (MMC) initiative and the Medical Training Application Service (MTAS) debacle in 2007-08.

Underpinning research

The MMC and MTAS inquiries found inadequacies in the national model for selection to medical training posts in the UK, leading the NHS to revert to locally governed recruitment programmes. In 2008 the South West (SW) Peninsula Deanery was funded to run selection pilots to evaluate evidence based approaches to selection and education-skills assessment. The pilot involved research on the educational principles supporting selection methodology to specialty training, focusing on shortlisting and interview methods for applicants to anaesthesia and acute care common stem (ACCS) training posts. The research was delivered by Plymouth University staff, including Dr Tom Gale (Honorary Fellow, 2005-11; Senior Lecturer, 2011-present) as project lead and Martin Roberts as Research Fellow (2008 onward) under the supervision of Gale.

The project first explored shortlisting methodology. Focusing on the educational requirements underpinning selection, a situational judgement test (SJT) was administered by Roberts and Gale alongside a machine marked clinical problem solving test (CPS) prior to interview during the 2008 recruitment round in the SW Peninsula Deanery. Results confirmed that a tailored SJT for anaesthesia / ACCS posts had acceptable reliability and correlated well with subsequent scores at interview compared to other shortlisting methods. A self-score portfolio assessment prior to interview was also piloted in the SW Peninsula Deanery in 2009 as a potential mechanism for shortlisting applicants. Scores were verified by assessors in the portfolio station of the subsequent selection centre (Gale, et. al., 2009).

In 2010 the SJT was administered by Gale and Roberts (in an on-line computerised format) alongside a CPS test as part of the Academy of Medical Royal Colleges (AoMRC) selection pilot for recruitment to core training posts across multiple specialties as part of a national evaluation. Gale and Robert's results confirmed that: the SJT was fair with high reliability (Cronbach's α=0.84) that is desirable for high stakes assessment; the majority of items were suitable for inclusion in an operational test; there was good criterion related validity with strong correlations with live shortlisting; and, that the SJT was a strong predictor of performance at selection centres (Gale, et. al., 2011; 2012; 2013a).

Turning to multi-trait selection within the SW Peninsula Deanery, Gale and Roberts reviewed job-analysis literature on non-technical education and skills required of anaesthetists for assessment in a multi-method selection centre. As a result, a multi-station selection centre model was developed incorporating simulated work-related tasks and assessing candidates' past behaviour, career achievements to date, and current performance under stress. Content validity of the selection centre was ensured by mapping to the Department of Health (DH) person specifications for anaesthesia trainees.

Candidate performance was independently rated by up to twelve assessors. Statistical analysis of the selection centre scores showed good internal consistency and reliability, while consistently favourable feedback from both applicants and assessors supported the face validity of the process. Results for predictive validity were positive and generalisability analysis of selection data established that the reliability of the selection process is increased further by increasing the number of stations rather than the number of raters per station (Gale et. al., 2012; 2013b).

References to the research

Details of the impact

The research described in section two has led to the implementation of an improved recruitment process for core training and specialty training posts in medical related fields in the UK, primarily in shortlisting methodology and selection centre recruitment.

Results from the SJT pilot on shortlisting methodology were presented to the DH Medical Programme Board (MPB) in November 2011. It was highlighted that the SJT developed in the research provides an objective marker of candidate performance that is standardised across multiple units of application, and recommended that further work should progress to explore its development in support of the MPB's aim to improve the validity and reliability of specialty selection. This recommendation has been taken forward by the UK Postgraduate Medical and Dental Selection and Recruitment Project Board with Gale acting as an advisor to its Quality Group.

In 2011 the Royal College of Anaesthetists' national recruitment committee implemented the self-scoring method for shortlisting based on the Portfolio self-score template developed by Gale and Roberts et al. Strong correlations between self-score and verified portfolio assessor scores at interview were found (r=0.92 p<0.001), which has obviated the need for assessors in each Deanery in the UK to take part in scoring applications for the purposes of shortlisting (therefore creating time efficiencies): for example, in the first round of interviews for anaesthesia in 2013, 1012 applicants were interviewed in 18 deaneries; all applicants were ranked for interview based on portfolio self-score and no assessors were needed to shortlist in any Deanery.

In 2006 the British Medical Association stated that there was insufficient evidence to support the recommendation of any one particular interview method for recruitment to specialty medical training in the UK, despite traditional panel interviews being subject to error and biased scoring. Overcoming this lack of guidance, Roberts and Gale's pilot phase research on the selection centre model for recruitment to anaesthesia, drawing on recommendations from the Douglas Review, employed a four station model including: structured interview; presentation of a portfolio; an oral presentation; and a simulation excise. Two further stations were introduced in 2009, including: difficult communication utilising simulated patients, and telephone communication. The concluding report on the success of the pilot approach, presented to the RCoA and DH in 2010, led to a new national model for recruitment to anaesthesia and the UK: the national selection centre model was standardised to include three (of the six pilot) stations with two independent raters in each station: structured interviews, portfolio and oral presentation. Gale and Robert's research was central to this development, as highlighted by the Recruitment Committee Chair of the RCoA... "this work, of national importance has had significant positive impact to anaesthesia recruitment and set the conditions for further improvement towards coordinated national recruitment".

In addition, scoring matrices for non-technical skills developed by Gale and Roberts have been introduced nationally and an on-line assessor training tool for all interviewers involved in national selection for anaesthesia training posts developed. This training provides familiarisation with best practice in interview methods and includes film examples with actors playing the role of the applicant. The training has been accessed by 539 assessors across England, Wales, Scotland and Northern Ireland, and "has been instrumental in ensuring standardised interview techniques across all Units of Application, meeting GMC requirements" (RCoA, President).

The changes to national recruitment established a single application-interview process for anaesthesia and ACCS training posts, which impacted considerably on the efficiency of the recruitment system. As part of the AoMRC selection pilot, data on all applications to acute specialties was obtained from 13 of 14 English deaneries in 2010; the data showed that a total of 1,498 doctors made 5,118 applications for acute specialty training posts. The number of applications per doctor ranged from 1 to 33 (median = 2). After the introduction of the single interview process, RCoA data for 2012 showed that recruitment for core training in anaesthesia and ACCS involved only one application by each candidate, with 972 doctors being interviewed for 538 posts. The efficiency in the recruitment system in turn means that resources (both human and capital) are able to be invested elsewhere in the medical system, ultimately leading to improved services for patients and beneficiaries.

Sources to corroborate the impact

The following sources corroborate the claim that the research led to an improved shortlisting methodology in UK Deaneries:

1. SJTs: Report to Medical Programme Board, Department of Health on future selection methods 23rd November 2011: http://www.mee.nhs.uk/pdf/MPB%20November%202011%20Minutes%20(Approved).pdf
2. SJTs: ST1 Selection Pilot Project Report 2010 - Academy of Medical Royal Colleges
3. Self-scoring method - Recruitment Lead, Anaesthetics National Recruitment Office, NHS Midlands and East, West Midlands Workforce Deanery

The following sources corroborate the claim that the research led to a single interview method in selection centres:

4. National Recruitment - The Silent Revolution. RCoA Bulletin 2011; 65: 40-42 http://www.rcoa.ac.uk/document-store/bulletin-65-january-2011
5. Royal College of Anaesthetists - President (also relevant for the online training tool)

Single application-interview process:

6. Statement from Recruitment Lead, Anaesthetics National Recruitment Office, NHS Midlands and East, West Midlands Workforce Deanery