The Department of Health’s report ‘The Next Stage Review: A High Quality Workforce’ (2008) highlighted concerns about the methods of selection to the Foundation Programme in use since 2005. As part of the Option Appraisal, the ISFP Project Group analysed a number of potential selection tools to the Foundation Programme, including interviews, assessment centres and a national exam.
The ISFP Project Group recommended the use of a measure of aptitude for employment in combination with a measure of performance at medical school for selection to the Foundation Programme. The Option Appraisal, as well as experience gathered during the piloting and development phase, resulted in the recommendation by the ISFP Project Group to the Department of Health the Situational Judgement Test and the Educational Performance Measure as methods of selection to the Foundation Programme.
SJTs are proven to be a valid, reliable and effective measure of aptitude for employment, and the FY1 SJT has been designed to specifically target the attributes of an FY1 doctor. The development of a standardised EPM framework uses existing information held about the applicant by the medical school, and a shared set of principles to calculate the school rank, which enables greater consistency and thus comparability between applicants. The two measures in combination produce a score with greater differentiation between applicants. Alongside applicants’ preferences for specific foundation schools, the score is used to rank and allocate them to the foundation schools.
See below for justification for implementing the SJT and EPM in place of “white space” questions and academic quartiles for selection to the Foundation Programme.
Selection always needs to evolve
There is a constant process of development of new selection methods and publication of research on their effectiveness. The Department of Health was keen to ensure that new doctors starting work in the NHS are selected using the most appropriate methods.
SJTs are an aptitude test proven to predict future job performance, and there is evidence of their validity and reliability for selection into Foundation Programme. SJTs are increasingly employed in large-scale selection processes, for example by the police in the UK and by the FBI in the USA. In medicine, SJTs have been used successfully for many years for selection into GP training in the UK and Australia, and they are being piloted for selection to other postgraduate medical specialties.
SJT scenarios accurately reflect the FY1 role
It is widely acknowledged that non-cognitive or professional attributes (e.g. communication, integrity, empathy, team working) are essential requirements for being a doctor. SJTs are able to target these important professional attributes, which are difficult to assess through other methods. The design of the FY1 SJT was informed by a literature review, review of best practice and piloting. The SJT items themselves target the attributes expected of an FY1 doctor, as specified in the national person specification, and as identified through the multi-source, multi-method job analysis.
SJTs undergo extensive development, review, and piloting process before they are used. SJT items are written by Subject Matter Experts (clinicians working closely as or with foundation doctors) in cooperation with psychologists, foundation doctors and other clinicians involved in the review and refinement of items, as well as agreement on the answer key. This ensures that the scenarios presented are a realistic reflection of what FY1s encounter in their role.
The SJT is less labour intensive than “white space” questions
Whereas the “white space” questions were labour intensive to mark, SJTs are machine-marked according to a pre-determined answer key. Clinicians continue to be involved in the process; however with SJTs, clinicians contribute to the item-writing, review and development, rather than to marking individual answers.
The SJT is invigilated
The SJT is taken on national dates in invigilated conditions in the UK. There is no risk of plagiarism, as there was with the previous system of “white space” questions completed online.
Research and piloting have proven the validity of the SJT
Research evidence suggests that SJTs have good levels of predictive validity (i.e. they are able to predict a person’s performance in the job) and good reliability. An independent validation study published in 2015 found that the SJT and the Educational Performance Measure are complementary in predicting FY1 doctors’ performance.
The SJT has been extensively piloted as part of the ISFP project. Seventeen UK medical schools and two non-UK medical schools piloted items during the 2010/2011 academic year, involving over 1,000 final year medical students. The analysis and evaluation of the Parallel Recruitment Exercise, which involved 7,000+ applicants to the Foundation Programme 2012, again confirmed that the SJT, in conjunction with the EPM, is a reliable, valid and appropriate method for selection. These findings support the extensive research published on SJTs, which have been used successfully over many years for selection to GP training in the UK, as well as for large-scale selection to other professions. A Validation of F1 Selection Tools Report. describes the methodology and findings of a preliminary validation study that explores the potential links between applicants’ performance during application to the UK Foundation Programme and their subsequent performance during Foundation Year One
Academic performance alone does not predict future success
The stakeholders in the ISFP Project Group supported the use of a measure of academic performance as part of the selection method. However, it is widely acknowledged that in addition, non-cognitive attributes constitute essential requirements for being a doctor. SJTs are able to target these important professional qualities, which other methods do not necessarily capture. This complementarity has also been confirmed by the SJT validation study published in 2015.
EPM is a standardised and transparent medical school performance measure
Academic quartile scores used prior to recruitment to FP2013 were a very rough measure which did not allow for much score differentiation between applicants. In addition to this, there was a suggestion that although many did, not all medical schools calculated their academic scores in a way which was transparent to applicants. The timing of the application process means that final exams usually take place too late to be used as an indicator; and the range of assessments taken at different times at different schools in the UK and abroad inhibited creating a single measure. An EPM Task and Finish Group, including medical students, medical schools and employers, was convened in 2010 to review EPM pilot data, and to develop a method for calculating educational performance.
The EPM framework recommended as a result of the Group’s work in 2011 set out the principles agreed by all UK medical schools to inform the calculation of decile points using assessments taken at a particular school. All assessments used in the determination of a student’s performance must:
- Be summative (and therefore subject to more formal controls)
- Cover clinical knowledge, skills and performance
- Cover non-clinical performance
- Cover all aspects of the curriculum assessed up to the end of the penultimate year at medical school
- Represent the average performance of applicants over time
- Include written and practical assessments
As part of the Parallel Recruitment Exercise, each UK medical school consulted with students on the local composition of the EPM decile points. Many schools undertook extensive consultation with students and staff, and the agreed frameworks have been published locally to ensure transparency.
EPM framework is a single measure, which also captures additional academic achievements
Applicants are awarded a decile point score, depending on their performance at medical school in relation to their graduating cohort up to the point of application. In addition, applicants may each be awarded points to recognise additional degrees and additional academic achievements.
EPM enables greater differentiation between applicants
Within the EPM framework, applicants are awarded points for their decile rank in relation to the graduating cohort. Under the application system prior to FP2013, applicants were ranked into quartiles. The EPM Task and Finish Group analysed pilot EPM data for 5000+ applicants and recommended that moving to a system of decile ranks meaningfully increased differentiation between applicants without compromising the underlying resolving power of assessment marks. The points difference between ranks has also been reduced, in such a way that the decile points are awarded over a similar range as quartiles, making the process fairer, particularly for applicants at the margins.