Design of the FY1 SJT

A literature review was carried out to assist the development of the initial SJT design specification and to provide background information about the research evidence available regarding SJT development and evaluation. The literature review, which covered all existing policy documentation and relevant publications, was conducted in two parts. The first part aimed to identify the key skills and professional attributes of a FY1 doctor. Part two of the literature review focused on Situational Judgement Tests (SJTs) and their use in high stakes selection. The results help to put the purpose of the job analysis in context and to understand the methodology, which guides the SJT development. The literature review can be found in the job analysis.

To assist in the development of the initial SJT design specification, a series of ‘micropilots’ (i.e. medical school student reviews) were carried out at four medical schools: Cambridge, Manchester, Sheffield and UEA. The aim of the micropilot reviews was to assist in establishing the appropriateness of the SJT design specification including test length, appropriateness of content, response format and response instructions. Students were presented with a number of sample SJT questions in a range of SJT item formats, and asked to complete them in invigilated conditions. A focus group was then held, where participants shared their views on the item content, format and structure.

A systematic, multi‐method job analysis of the Foundation Year One (FY1) role was prepared. It consisted of a literature review, semi-structured critical incident interviews and observations. This triangulation approach was used to increase the credibility and validity of the results. In total, 51 interviews were conducted with Foundation doctors, clinical/educational supervisors, Directors/Deans/Heads of Medical Education at Medical Schools, Trust Medical Directors, Foundation School and Programme Directors and patient representatives. 29 hours of shadowing were completed at four locations across the UK (Cambridge, Manchester, Aberdeen and Stornoway). In addition, there were 230 responses to a validation survey.

The Job Analysis resulted in a Professional Attributes Framework, which identified the professional attributes that are important for a FY1 doctor, and provide target attribute domains for the SJT.

nine domains

Although the items are mapped in relation to domains for item development and test construction purposes, they do not exclusively measure any single domain. The SJT encompasses complex, realistic work scenarios; therefore it should not be viewed as assessing five individual constructs or as a blueprint. The advantage of this approach lies not only in the increased complexity of the items, but also in avoiding the possibility that a template answer could be provided to applicants. The target domains can be seen more as facets relating to the professional attributes required of a Foundation doctor, while the SJT provides an overall measure of this group of relevant professional attributes.

The design of the FY1 SJT is reviewed on a yearly basis. For related reports, please refer to PRE Report 2012 and annual technical reports.