SJT Development and Piloting

Item development

A job analysis of the FY1 role identified nine professional attributes of a good FY1 doctor. The SJT items draw on these domains identified through shadowing of Foundation doctors as far apart as Cambridge, Manchester and the Scottish Isles. In addition, over fifty interviews with doctors, patients and other professionals working with FY1s took place to define the attributes.

The nine domains identified are:

  • Commitment to professionalism
  • Coping with pressure
  • Effective communication
  • Learning and Professional Development
  • Organisation and planning
  • Patient Focus
  • Problem solving and decision-making
  • Self-awareness and insight
  • Working effectively as part of a team

SJT items are developed at item writing workshops involving clinicians and others who work closely with Foundation doctors. These items are subject to a number of quality assurance stages before being reviewed by FY1s and FY2s to ensure they are a realistic reflection of the FY1 role.

If you are a clinician working with Foundation doctors and you would like to become involved in the development of SJT items, please contact MSC Assessment.

Large Scale Pilot

Between October 2010 and April 2011, SJT pilots took place at 15 UK and two non-UK medical schools involving over 1,100 participants. Final year medical students were asked to take a two hour invigilated SJT, answering 60 questions.

The pilot was released in three paper versions in order to maximise the number of new SJT items piloted. Each paper version consisted of 60 items (40 ranking and 20 multiple choice) to be completed within two hours.

Results confirmed that the SJT is a power test and that two hours is an appropriate amount of time to complete the 60 items. In terms of psychometrics, test statistics showed a relatively high level of internal reliability, which was sufficient for the SJT being used in a high stakes assessment. The range of scores showed that the SJT was able to differentiate between applicants.

The pilots further demonstrated that applicants scoring highly in SJT were also ranked high on extraversion, openness to values and achievement. On this basis, it was concluded that the SJT broadly measures attributes relevant to the work in the Foundation programme.

The full analysis and evaluation of the design, development and piloting work can be found in the Final Report of Pilots.

PRE

On the recommendation of the ISFP Project Group, a full-scale Parallel Recruitment Exercise (PRE) took place alongside the existing selection to FP 2012. The main aim of the PRE was to test logistics ahead of live implementation for FP 2013, but with the added benefit of piloting new SJT content for the item bank, and raising awareness of the forthcoming changes among potential applicants and other stakeholders.

As part of the PRE, a one hour, 30 item SJT was taken by 6,842 participants in 72 venues between November 2011 and January 2012. This was equivalent to an overall participation of 90% of all applicants to the Foundation Programme. All applicants were invited to take part in the pilot, and in March 2012 they were provided with feedback in the form of a decile.

The results were used to help to validate the SJT items, evaluate their effectiveness, and to ensure that structures were in place for delivery of the SJT for FP 2013. Feedback from applicants indicated widespread satisfaction that the content of the assessment was relevant, fair and appropriate for selection to the Foundation Programme.

Detailed information is available in the Final Report of the PRE.