Review Article

The FC Rad Diag(SA): Stretched, but still in step – Report on a Royal College observership

Richard Pitcher, Coert de Vries, Zarina Lockhat

Received: 16 Sept. 2016; Accepted: 09 Oct. 2016; Published: 31 Jan. 2017

Copyright: © 2017. The Author(s). Licensee: AOSIS.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

A delegation of the College of Radiologists of the overarching Colleges of Medicine of South Africa observed the spring sitting of the Part 2B Examination of the Fellowship of the Royal College of Radiologists (FRCR) in London, in April 2016. Although the principal focus of the Observership was the Part 2B examination, the delegates also assessed broader aspects of the FRCR. This report presents an overview of current FRCR practices, including the findings of an independent review of the FRCR, and reflects on the implications for the South African Fellowship examination. The report is based on discussions with key Royal College role players, direct observation of the Part 2B examination and web-based documentation. It allows appreciation of the continued close alignment of the FC Rad Diag(SA) (Fellowship of the College of Radiologists of the Colleges of Medicine of South Africa) with the FRCR and highlights expected trends in the FC Rad Diag(SA). It also documents the increasing human resources required for successful conduct of examinations. It is hoped that the report will be of interest and relevance to the broad South African radiological community and to those contemplating specialist training in the discipline. It is trusted that it will encourage wider involvement of radiological and medical physics colleagues in the various FC Rad Diag(SA) examination processes.

Introduction

Historically the FC Rad Diag(SA), the Fellowship of the College of Radiologists (CR) of the Colleges of Medicine of South Africa (CMSA), has been aligned with the Fellowship of the Royal College of Radiologists (FRCR). CMSA representatives have periodically observed the FRCR examination to ensure CR assessment methods remain abreast of international trends.

The authors observed the spring sitting of the Part 2B Examination of the FRCR in London, from Saturday, 9 April, to Friday, 15 April 2016.

Report structure

Although the principal focus of the Observership was the Part 2B examination, CR delegates assessed broader aspects of the FRCR, including the findings of an independent review of their examination by medical education experts, commissioned in 2014.¹

This report presents a broad overview of current FRCR practices, followed by a detailed analysis of each component of the examination, including review findings, and reflects on the implications for the FC Rad Diag(SA).

The report allows appreciation of the continued close alignment of the FC Rad Diag(SA) with the FRCR and highlights expected trends in the FC Rad Diag(SA). It also documents the increasing human resources required for successful conduct of examinations. It is thought that the report will be of interest and relevance to the broad South African radiological community and to those contemplating specialist training in the discipline. It is hoped that it will encourage wider involvement of radiological and medical physics colleagues in the various CR examination processes.

The Fellowship of the Royal College of Radiologists

Overview

The examination has three components.

Part 1

• Radiation physics
• Radiological anatomy
• Completion required by the end of the first year of training.

Part 2A

• The written examination, comprising six modules
• Completion required by the end of the third year of training.

Part 2B

• The clinical examination, comprising three modules
• Completion required by the end of the fourth year of training.

Progress through the FRCR stages is aligned with progress in UK registrar training. The consequences of failure include extension of training time or even exclusion from training.

Part 2B admission requires completion of Parts 1 and 2A, as well as satisfactory completion of an approved period of speciality training, determined by workplace assessments.

Six attempts at any FRCR examination are permitted.

Examination committee structure

Overall FRCR integrity is governed by the Fellowship Examination Board (FEB), established in 2012.

Each FRCR component has a dedicated examination board that sets examinations, evaluates candidate performance and reports to the FEB.

Part l (Physics)

Fellowship of the Royal College of Radiologists format

Candidates must be in a registrar training post.

The 2-hour examination comprises 40 multiple choice questions (MCQs). There is no negative marking. The pass mark varies at each sitting, based on question difficulty, determined by a criterion-referenced, standard setting technique.

Fellowship of the Royal College of Radiologists review findings

The true or false MCQ format should be abolished, since this only tests factual recall.

Single best answer (SBA) questions should be introduced, as these evaluate knowledge application and allow more meaningful post hoc analysis.

Royal College of Radiologists response

It is difficult to examine the basic sciences in SBA format.²

A forum has been established to review the Physics syllabus and examination format.

Comparison with and implications for the FC Rad Diag(SA)

SA candidates may write the examination prior to embarking on registrar training. No more than three attempts are permitted, including University Master of Medicine (MMed) Part l exams. (There is reciprocity between the CR and University Part l exams.)

The current CR 3-hour Physics examination, combining short written answers and true–false responses, will likely be replaced by the SBA format following appropriate examiner training.

In the interests of continuity, it is proposed that a 3-year, fixed-term FC Rad Diag(SA) Part l Physics examination committee be appointed. This should have representation from all training centres and the private sector.

Part l (Anatomy)

Fellowship of the Royal College of Radiologists format

Candidates must be in a registrar training post.

The 90-minute examination is a 100-image ‘spot test’, using a ‘name the structure’ approach. Candidates work at individual work stations. To accommodate the large number of candidates, four different image sets are used, in four sequential sessions per examination sitting.

A broad, two-dimensional ‘blueprint’ based on modality and body system is used. (The blueprint simply refers to the examination weighting that is given to sections in the curriculum.)

There is comprehensive post hoc statistical analysis.

Fellowship of the Royal College of Radiologists review findings

Difficulty and passing standards should be comparable between sets within a single examination sitting, and across sittings.

The emphasis should be on testing cognitive skills and the application of knowledge, rather than factual recall.

There should be more formal blueprinting aligned with core learning outcomes.

The coverage of each assessment depends on the expertise and knowledge of the examiners setting the paper at the time. There is no historical record to guide comparisons of paper difficulty over time and no use of anchor or reference questions to assess comparability of candidate performance and standards.

Royal College of Radiologists response

An ‘automation project’ has been initiated to manage digital image banks, with a view to an online digital Part l Anatomy examination. This will address many concerns of the review.

Comparison with and implications for the FC Rad Diag(SA)

The current examination format mirrors that of the FRCR but has 60 images, lasts 3 hours and totals 300 marks.

It is suggested that a fixed-term Part l Anatomy examination committee be appointed, as for Physics.

Written component (Part 2A)

Fellowship of the Royal College of Radiologists format

The core-knowledge modular assessment comprises six papers, written in a staged fashion during the second and third years of training. Candidates are examined on clinical radiology and the basic sciences, according to the Specialty Training Curriculum for Clinical Radiology.³ Any number of papers may be written at one sitting.

Each examination lasts 2 hours and comprises 75 SBA items directly linked to one of the six major components of the curriculum:

• Module 1: Cardiothoracic and vascular
• Module 2: Musculoskeletal
• Module 3: Gastro-intestinal
• Module 4: Genitourinary, female imaging
• Module 5: Paediatric
• Module 6: Central nervous system, head and neck.

There is a simple blueprint, which defines the proportion of questions by sub-topic.

Fellowship of the Royal College of Radiologists review findings

A single assessment is recommended.

There should be more formal blueprinting, aligned with core learning outcomes.

The coverage of each assessment is dependent on the knowledge of the examiners setting papers at the time, as for the Part l Anatomy.

Royal College of Radiologists response

From spring 2018, the structure will revert to a single SBA examination covering the core curriculum.

A two-dimensional blueprint is being considered.

Comparison with and implications for the FC Rad Diag(SA)

The FC Rad Diag(SA) Part 2 written component is an exit examination taken in the same semester as the oral examination. It may not be attempted prior to 36 months of registrar training.

Currently, FC Rad Diag(SA) Part 2 examiners conduct two consecutive sittings of the written and oral components and then step down. All training institutions and some private radiologists are represented on the examiner panel. Appointment of a 3-year, dedicated Part 2 written examination committee is recommended.

The current format includes only short written answers, with no SBAs. It is recommended that SBAs be included, following formal examiner training.

Fellowship of the Royal College of Radiologists Part 2B examination

Fellowship of the Royal College of Radiologists format

The examination represents a three-component assessment, evaluating workplace skills. The authenticity of the assessment is a key characteristic.

This is not an exit examination, but a core-knowledge examination and may be attempted from 30 months of training.

The examination is convened in spring and autumn each year and comprises the following:

• rapid reporting (RR)
• long case reporting (LCR)
• oral assessment.

Part 2B Examiner Board

The board comprises 24 examiners, appointed following written application and formal recruitment, and after thorough evaluation of each applicant’s curriculum vitae and teaching experience.

Appointments are for 6 years and governed by a formal job description and code of conduct. Four new examiners are recruited annually, replacing four who have completed their term.

Examiner profile

Examiners are fellows with at least 7 years of consultant experience in a UK training program and significant involvement in registrar teaching and assessment.

Examiner duties

• Prepare images for oral, RR and LCR examinations
• Mark the RR and LCR examinations
• Conduct the oral examination
• Review the performance of individual examinations
• Continually develop the examination, ensuring its ongoing relevance.

For the first year, new examiners observe the spring and autumn sittings and prepare at least 100 examination cases.

Senior examiners select the RR and LCR material. One senior examiner is the lead for each sitting and moderates differences in mark allocation that cannot be resolved by consensus.

There is diversity of examiner age, gender, ethnicity and UK geographical distribution.

Candidates

Candidates are only identified by number: 290 candidates sat the examination; 150 were from the UK.

Examiners do not examine their own candidates.

Digital platform

The examination has a digital platform, utilising Osirix (vendor) software. This software has a steep learning curve and is subject to the usual software gremlins but facilitates complete simulation of the clinical environment.

Rapid reporting

Fellowship of the Royal College of Radiologists format

The exam is convened over the weekend prior to the oral examination. It is also conducted on Osirix software, on Apple hardware, in a computer laboratory that is leased and set up at the Royal College of Radiologists (RCR) for each sitting.

Candidates receive an excellent briefing on Osirix, are assigned to individual workstations and work at their own pace, completing 30 cases in 35 minutes. Only plain radiographs are included. At least 40% – 50% of cases are normal. Abnormal cases have a single abnormality. All abnormalities are strictly emergency- or trauma-related.

For each case, candidates are required to state whether the radiograph is normal or abnormal. If abnormal, the pathology must be specified.

All images are held in the RCR database. Material is subject to post hoc review of suitability.

A RR pass is not required to pass the examination as a whole.

Fellowship of the Royal College of Radiologists review findings

The assessment is authentic, simulating the reporting environment.

Difficulty should be comparable between sets of cases within a single sitting, and across sittings. A greater number of items should be included to improve reliability.

Some form of standard setting is required.

Royal College of Radiologists response

There is no current consensus on the optimum method of standard setting.

Comparison with and implications for the FC Rad Diag(SA)

The FC Rad Diag(SA) RR mirrors that of the FRCR, but 60 cases are included, enhancing reliability. The RR is included in the written component. A pass in the RR is required to progress to the oral examination.

Defined case-selection inclusion criteria must be rigorously defined. Abnormal cases should have a single abnormality and be strictly within the realm of emergency/trauma radiology. More detailed post hoc analysis is required.

Long case reporting

Current Fellowship of the Royal College of Radiologists format

The exam is conducted in the same computer laboratory as the RR and written over the weekend preceding the week of the oral examination.

Cases are selected from the RCR digital database, which is managed by one of the permanent administrative RCR staff.

Candidates are shown an introductory video on the Osirix software immediately prior to the examination and work on two dummy cases before the start.

The examination comprises six clinical cases of varying difficulty, to be completed in 60 minutes. Candidates work at their own pace.

For each case, candidates are provided with the age, gender, clinical presentation and a list of included images.

Marking takes place during the week of the orals but is independent of the orals. It is undertaken by a small, separate group of examiners.

LCR markers see the cases for the first time when marking begins and before viewing marking memoranda. This provides a reality check in terms of expectations and assigning level of difficulty.

Examiners mark the first few scripts, assessing candidate performance, and decide whether the marking memorandum requires modification.

Long cases are double-marked, each marker being blinded to the score of the other marker.

Memoranda provide detailed guidelines on mark allocation. To pass a case, candidates are required to identify the majority of key radiological signs and to reach a reasonable diagnosis. Discrepancies between individual markers are resolved by consensus wherever possible.

Concise answers are expected. The answer sheet has the following subdivisions: Observations, Interpretation, Principal Diagnosis, Differential Diagnosis and Management.

Candidates are not required to pass a minimum number of cases.

The large number of candidates dictates that candidates take one of four possible LCR sets.

Fellowship of the Royal College of Radiologists review findings

The comparability of set difficulty and passing standards between sets within and across sittings is a concern.

Although it is difficult to achieve high reliability in clinical assessments, the reliability of Part 2B and its individual components is lower than acceptable for high stakes assessments.

The marking scheme should make provision for the award of the full range of marks, from very low to very high.

Standard setting is required.

Royal College of Radiologists response

There is no consensus on the optimum method of standard setting.

Comparison with and implications for the FC Rad Diag(SA)

The FC Rad Diag(SA) Part 2 LCR comprises seven cases, with 20 minutes for each case. Marks are allocated according to a rubric that is broadly comparable with that of the FRCR. Candidates must pass at least four out of seven cases and achieve an overall pass mark for the LCR component.

It is proposed that future marking memoranda define criteria for allocation of a wider range of marks.

For the past 5 years the CR has employed a part-time IT consultant to configure the LCR examination and to manage a central repository of LCR examination cases. Since the 2016 RCR Observership, a senior academic radiologist has been appointed as convener of the LCR examination, for a fixed term. This is seen as the first step towards creating an LCR examiner panel.

From the second semester of 2016, LCR scripts will be double-marked. Marking will be conducted by a separate group of examiners, in parallel with the oral examination. There will also be more comprehensive moderation of LCR marking.

Orals

Fellowship of the Royal College of Radiologists format

The examination is conducted in soundproof rooms and on a digital platform, utilising Osirix software. There are separate monitors for examiner and candidate.

Examiners provide their own images. Prior to each session, examiner pairs show each other their films, explaining what is expected, and how marks will be allocated. This is the only moderation. Candidates see two examiner pairs, for 30 minutes apiece. There should be at least 10 scoring opportunities in each 30-minute session. Discussion of a single modality is a scoring opportunity. Examiner pairs are kept constant throughout the week and combine inexperience with experience. For each image set, examiners prepare a memorandum, listing the cases to be shown, the diagnosis and the salient radiological features, with space for comment on each case.

The complete examination blueprint is shown by each examiner pair.

The overriding question is whether the candidate is ‘safe’. The emphasis is on protecting the public against unsafe practice.

Examiners prompt candidates if specific signs have not been identified, but such candidates are marked down.

At the end of each oral, examiners independently assign a mark before conferring and reaching a consensus mark.

There is no requirement to pass a minimum number of cases.

Borderline candidates are reviewed at an examiner meeting at the end of each day. There is no formal moderation of mark allocation.

Fellowship of the Royal College of Radiologists review

There is good training for examiners, briefing of candidates and feedback to failing candidates and their supervisors.

The value of clinical oral examinations in specialised fields such as radiology was acknowledged. However, the current format is not considered good practice. There is examiner-related variance in case-selection and behaviour within stations, posing risks to robustness and defensibility. The system is dependent on the skill and conduct of individual examiners. There is reliance on examiners to police their own material and there is thus no means of defending challenges arising from this. There are no anchor or reference cases across examiner pairs.

A more structured format was recommended to mitigate risk while preserving important positive features. Each case should be started with a clinical request form or scenario, preserving authenticity while ensuring that candidates understand what is being asked. All images should be centrally pooled and held by the College. Global reference cases are required. This would improve defensibility of the examination, reduce unnecessary variance between stations, limit examiner-related variance within stations and enhance standard setting.

Although examiners can distinguish failing, borderline and excellent candidates, significant unintentional and intentional weighting could occur with the current narrow assessment scale, making it challenging to distinguish between candidates.

A test-specific blueprint is required to ensure careful coverage in case selection.

Recognised standard setting should be applied to each component of the Part 2B. In addition, success in the Part 2B should be subject to achieving a defined minimum mark for each component.

Examiners should score independently and only use the examiner meeting to discuss significant differences within examiner pairs.

Royal College of Radiologists response

There is no consensus on the best method for oral examination standard setting. The scoring system will be reviewed.

Comparison with and implications for the FC Rad Diag(SA)

In line with RCR review recommendations, the following changes were implemented in the first semester of 2016:

• A digital examination was delivered on a Microsoft PowerPoint platform.
• All cases were moderated at a full-day preparatory examiner meeting, where expectations and pass or fail criteria were defined. All cases had unequivocal radiological signs.

Material for each session was standardised. A text slide provided the clinical context for each case. Candidates were not prompted to help them identify radiological features. Only once the main radiological features had been identified did examiners engage in a discussion of the differential diagnosis and management. Where candidates failed to identify the main radiological signs, examiners moved to the next case. The full blueprint was covered for each candidate.

Candidates must achieve an overall (combined) oral mark of at least 50% and may not score less than 40% in a single oral.

Since the 2016 RCR Observership, the CR has employed a part-time IT consultant to configure the digital oral examination and to manage a central repository of oral examination cases. A senior academic radiologist has also been appointed as convener of the oral examination, for a fixed term. This is seen as the first step towards creating an oral examiner panel.

Royal College of Radiologists appeals process

This provides a formal means of reviewing a candidate’s examination results. While not a complaints procedure, it enables candidates to challenge a fail result on the basis of procedural irregularity in the conduct or the determination of the result of an examination.

Any appeal that challenges the academic judgement of examiners or that is based on extenuating personal circumstances or lack of awareness of examination regulations is considered invalid.

Comparison with and implications for the FC Rad Diag(SA)

The CMSA website defines the current unifying appeals process for the constituent colleges.⁴

Conclusion

The format of the FC Rad Diag(SA) remains closely aligned with that of the FRCR. The FC Rad Diag(SA) and the FRCR currently face similar challenges. The following are recommended.

Examiner resources

Initiate a system to ensure long-term, continuous, active involvement of all South African academic radiologists, as well as representatives of the special interest groups of the Radiological Society of South Africa, in the CR examination processes.

Constitute formal Examination Boards, with fixed-term appointments, for each component of the FC Rad Diag(SA).

Examination format

• Part I Physics: Convert to a SBA format.
• Part II Written: Convert to a hybrid system of short answers and SBAs.

Examination content

Ensure careful blueprinting of each component of the FC Rad Diag(SA) and close alignment of examination content with the published curriculum. Develop a central repository of examination material for each component of the examination.

Marking procedures

Revise mark allocations in the long case and oral components, in line with international best practice.

Post hoc analysis

Once appointed, examination boards should be responsible for the conduct of formal, post hoc analysis of each component of the examination process, utilising standard software packages.

Candidate feedback

Once appointed, examination boards should introduce a sustainable system for candidate feedback, thereby allowing candidates to address deficiencies. The precise mechanism of candidate feedback will vary across the various components of the examination.

In-course assessment

Strengthen and formalise institution-based workplace assessment, supervisor reports and an annual review of competence progression.

Acknowledgements

The authors declare that they have no financial or personal relationships that may have inappropriately influenced them in writing this article.

R.P., Z.L. and C.d.V. conceived the project and collected and collated all data. R.P. drafted the initial manuscript with editorial contributions by Z.L. and C.d.V.

References