key: cord-0868283-h5z7d8wv
authors: Chaurasia, Avinash R.; Page, Brandi R.; Walker, Amanda J.; Salerno, Kilian; Camphausen, Kevin; Kwok, Young; Bajaj, Gopal K.; Ambrocio, Daisy; Erickson, Delnora
title: Lessons to Learn from a Successful Virtual Mock Oral Examination Pilot Experience
date: 2020-08-08
journal: Adv Radiat Oncol
DOI: 10.1016/j.adro.2020.07.011
sha: 8625cd8f48f31887abf6bd1a1c936def6c43da0f
doc_id: 868283
cord_uid: h5z7d8wv

nan

Novel coronavirus 2019 (COVID- 19) has had a marked impact on clinical care and graduate medical education (GME), including in radiation oncology (RO) . Nearly all RO physicians face challenges balancing personal and professional obligations amidst unprecedented difficulties. Appropriately, the American Board of Radiology (ABR) postponed the oral and written board certification exams for 2020.

One of the most daunting aspects of any radiation oncologist's career is facing the oral certifying examination (OE). The American Board of Medical Specialties (ABMS) has been phasing out OEs from board certification, including the ABR recently terminated the Diagnostic Radiology OE. Despite recent controversy related to administration of the written exams, OE in RO boardcertification has been well-accepted in recent editorials by ASTRO and the ABR addressing RO board-certification reform [1, 2] .

How best to prepare for successful completion of the OE is undefined and non-uniform; there exists no standardized method to prepare for the OE. A recent survey reported mixed results with 73% of residency directors indicating that their faculty conduct MOE ≥ 50% of the time, however only 56% of the corresponding chief residents reported the same finding [3] .

The benefits of and logistics of implementing a rigorous mock oral examination (MOE) curriculum have been documented in other specialties represented in the ABMS [4] [5] [6] . In general surgery, Fingeret et al. reported the correlation between MOE performance and first-time pass rate of OEs, as well as the educational value of sequential MOEs [7] . Given these purported benefits of MOE and the necessity of social distancing, we conducted a virtual MOE in lieu of in-person MOE. We report here our pilot experience with a virtual MOE to highlight its ease of implementation for RO GME and to demonstrate its feasibility as an option for the ABR's OE.

In the National Capital Consortium (NCC) RO Residency, we conduct annual MOEs for all residents each April which mimic the published guidelines for the OE by the ABR [8] . We additionally incorporate a minimum of 3-5 minutes of feedback per site. All residents participate in the MOE, with the particular site and number of sites required dependent on level of training. We also welcome recent graduates from our program and regional board-eligible junior staff to participate in this experience. Due to COVID-19 constraints, we decided to conduct the exam using an interactive web-based software meeting platform (Zoom Communications Inc., San Jose, CA).

Eight examinees (six residents, two junior attendings) and eight examiners representing four academic institutions participated in this MOE. All were offered an optional test run in advance of the MOE to explore features of the virtual platform. "Breakout Rooms" allowed for multiple simultaneous examinations to run within one larger meeting. "Screen Share" and "Annotate" allowed for anatomy review, contouring, and treatment plan evaluation. Patient-identifying information was not permitted in any exam materials.

On the day of the virtual MOE, we asked all participants to join a brief orientation. The residency coordinator proctored the exam and switched examinees between each site ("Breakout Room") every 30 minutes, after a 25-minute exam and 5 minutes of feedback. The exam was conducted successfully with all examinees rotating through their sections in the allocated timeframe. Screenshots with diagrams of the exam administration are shown in Figures 1 and 2 .

A survey was developed using Google Forms (Alphabet Inc, Mountain View, CA) by the chief resident and the residency program director. It aimed to assess the preparation, administration, and overall MOE experience for both examiners and examinees. Survey items to assess each of these areas were developed with most questions either using a 5-point scale (not at all, minimally, somewhat, moderately, and very confident) or a yes/no answer format. Most questions also allowed for a free text input if there was feedback that could not be captured appropriately by the survey questions.

Following the exam, we conducted a post-exam survey for both examiners and examinees. We had an excellent response rate of 100% for examinees and 87.5% for examiners.

The majority of examinees and examiners (87.5% and 85.7% respectively) reported that the virtual platform (Zoom) was "easy to use" or "very easy to use". 87.5% of examinees and 100% of examiners (who participated in test session) reported the test run left them "very well prepared". 100% of examinees and 85.7% of examiners reported that web-based software meeting platform was felt to be "very adequate" or "somewhat adequate" in quality of examination. There was a free-text section in the survey regarding any technical issues encountered during the virtual MOE, however all responses reflected that no technical difficulties were experienced and that it was felt to be a fairly seamless experience.

As compared to their previous in-person MOE experience, 87.5% of examinees and 83.3% of examiners reported that the virtual MOE experience was either easier or the same for ease of understanding the cases. 100% of examinees and 100% of examiners reported the preparation for virtual MOE to be equivalent or less time-consuming than preparing for in-person MOE.

75% of examinees and 71.4% of examiners reported that they would "definitely use" or "consider using" a video recording of each encounter to enhance learning/teaching. In regard to use of unauthorized resources (i.e. any reference material not be available during in-person OE) during the exam, examiners reported that 57.1% were "not concerned at all", while 28.6% reported they were "somewhat concerned." When asked if examinees should have access to certain resources ("open book") during the MOE and OE, examinees reported that 25% and 37.5% should "definitely" and "maybe" have access to such resources, respectively. 37.5% of examinees reported being "unsure". When the same question was asked of examiners, 42.9% and 14.3% reported that examinees should "definitely not" and "likely not" have access to such resources, respectively. This pilot experience and survey were conducted prior to the ABR's official announcement of a move to a virtual platform for the 2020 OEs. When asked how strongly they felt the ABR should J o u r n a l P r e -p r o o f move to a virtual OE, 50% of examinees and 42.9% of examiners preferred virtual OE, 25% of examinees and 42.9% of examiners had no preference, and only 25% of examinees and 14.3% of examiners preferred in-person OEs. Nevertheless, 87.5% of examinees and 85.7% of examiners reported a virtual MOE was a feasible alternative to an in-person MOE.

In our small pilot virtual MOE, we found the virtual platform to allow for high reported rates of quality and adaptability of virtual MOE. There appeared to be little to no learning curve to using the web-based software meeting platform and all participants found this format be no more timeconsuming than an in-person MOE. These findings are expected given the numerous applications of virtual platforms routinely in the clinic (i.e. virtual grand rounds, virtual chart rounds, virtual treatment planning).

Additional benefits to virtual MOE and OE can include a video recording for both an appeal process of pass/fail decisions and for QA of examination administration. In our survey, approximately 70-75% of respondents reported they would consider using this as a tool for their learning and/or teaching.

One common concern with virtual MOE is the possibility of the use of unauthorized resources. Our examiners reported a small majority reporting no concern of the use of such resources. In an era when some have questioned the modernity of RO board certification [1, 9] , we also surveyed the use of open resources on the test. Examinees reported a slight majority favoring use of open resources, while examiners reported a slight majority against the use of open resources. This is subject to bias in both cohorts with respect to experiences preparing for and/or having taken the OE.

While this is the first report of virtual MOE in RO, it is not unique in its effort to successfully implement the latest technology to bridge gaps in RO education, with multiple successful international RO education initiatives dating back to 2007 [10] [11] [12] . Additionally, a small randomized trial of in-person versus virtual MOEs for emergency medicine residents found no differences between the groups with respect to competency evaluation and quality of assessment [13] . This study also found that the examinees preferred the virtual format and found it to be less intimidating [13] .

There are numerous potential advantages to virtual OEs: increased accessibility, decreased costs, decreased time away from family, increased QA and feedback using video recordings, and decreased psychological intimidation. Another significant concern that can be mitigated is the potential discomfort of an examinee in the closely cramped quarters of an unmonitored exam room. These come at the cost of minimal disadvantages of virtual MOEs, nearly all of which can be easily accounted for: increased potential for use of unauthorized resources, potentially altered exam quality due to virtual environment. These can be mitigated with the use of proctored, remote testing centers near each examinee, allowing appropriate monitoring while maintaining the advantages of virtual OE. Of note, the case for virtual OEs is clearly demonstrated by the shift of two members of the ABMS from in-person OEs to a virtual OEs for 2020: the American Board of Surgery [14] and the American Board of Ophthalmology [15] .

There was an overwhelming response to the ABR's initial decision to maintain an in-person certification OE in December 2020 by the Association of Residents in Radiation Oncology (ARRO) [16] , American Society for Radiation Oncology (ASTRO) [17] , and Society of Chairs of Academic Radiation Oncology Programs (SCAROP) [18] . In light of consulting with these stakeholders and various organizations, the ABR has appropriately decided to create a virtual OE that will occur in 2021. Our small virtual MOE pilot experience adds evidence that virtual MOE are an important consideration for RO education in the technological age and that a virtual OE is a feasible option.

Examinations in Radiation Oncology: Listening, Learning, and Looking Forward Together

The American Board of Radiology Initial Certification in Radiation Oncology: Moving Forward Through Collaboration

Program director and chief resident perspectives on the educational environment of US radiation oncology programs

National Landscape of General Surgery Mock Oral Examination Practices: Survey of Residency Program Directors

Evaluating the effectiveness of a mock oral educational program

Cardiothoracic surgery mock oral examinations: A single institution's 5-year experience

Sequential Participation in a Multi-Institutional Mock Oral Examination Is Associated With Improved American Board of Surgery Certifying Examination First-Time Pass Rate

Certifying Exam for Initial Certification, Radiation Oncology

A Call for Change in the ABR Initial Certification Examination in Radiation Oncology

Virtual teaching offers practitioners new style of radiotherapy training

A hybrid integrated services digital network-internet protocol solution for resident education

Virtual study groups: Internet-based collaborative revision for the FRANZCR examinations

Virtual alternative to the oral examination for emergency medicine residents

Marcrom S on behalf of American Society of Residents in Radiation Oncology. Letter to: American Board of Radiology Initial Certification Advisory Committee

Letter to: Brent Wagner, American Board of Radiology

on behalf of Society of Chairs of Academic Radiation Oncology Programs. Letter to: Brent Wagner, American Board of Radiology

Written permission was obtained