key: cord-0789241-o09szlki
authors: Ren, Ronnie; Parekh, Kendra; Franzen, Doug; Estes, Molly; Camejo, Melanie; Olaf, Mark; Zhang, Xiao Chi
title: Emergency Medicine Clerkship Director Experience Adapting Emergency Remote Learning During the Onset of COVID‐19 Pandemic
date: 2021-03-04
journal: AEM Educ Train
DOI: 10.1002/aet2.10594
sha: 110f5c74e35179cff67af8852d8e4878571f9219
doc_id: 789241
cord_uid: o09szlki

OBJECTIVES: The recent outbreak of the COVID‐19 altered the traditional paradigm of clinical medical education. While individual clerkships have shared their curricular adaptations via social and academic networking media, there is currently no organizational standard in establishing a non‐clinical, Emergency Medicine (EM) virtual rotation (VR). The primary objective of this study was to describe EM clerkship directors’ (CDs) perspectives on their experience adapting an EM VR curriculum during the onset of the COVID‐19 pandemic. METHODS: A 21‐item survey with quantitative and qualitative questions was disseminated between June and August 2020 to EM CDs via the Clerkship Director of Emergency Medicine (CDEM) Listserv to describe their experience and perspectives in adapting a VR during the spring of 2020. RESULTS: We analyzed 59 out of 77 EM clerkship survey responses. Among respondents, 52% adapted a VR while 47.5% did not. Of those who adapted a VR, 71% of CDs had 2 weeks or less to develop the new curriculum, with 84% reporting usual or increased clinical load during that time. Clerkships significantly diversified their asynchronous educational content and utilized several instructional models to substitute the loss of clinical experience. Reflecting on the experience, 71% of CDs did not feel comfortable writing a standardized letter of evaluation for students based on the VR, with the majority citing inability to evaluate students’ competencies in a clinical context. CONCLUSION: A crisis, such as COVID‐19 necessitates change in all facets of medical education. While EM educators demonstrated the ability to create emergency remote learning with limited time, this was not equivalent to the formal development of pre‐planned VR experiences. Future faculty development and curriculum innovation are required to fully transition an in‐person immersive experience to a non‐inferior virtual experience.

based on the VR, with the majority citing inability to evaluate students' competencies in a clinical context.

A crisis, such as COVID-19 necessitates change in all facets of medical education. While EM educators demonstrated the ability to create emergency remote learning with limited time, this was not equivalent to the formal development of pre-planned VR experiences. Future faculty development and curriculum innovation are required to fully transition an in-person immersive experience to a non-inferior virtual experience.

Emergency Medicine (EM) is a unique, exciting medical specialty in which emergency physicians are given a brief period of time to interact with, examine, diagnose, and treat patients who present with a wide array of undifferentiated pathologies. The skills of the emergency physician and an understanding of the capabilities and limitations of the emergency department (ED) have been shown to be valuable to medical student education regardless of the student's chosen specialty. 1 The argument has been made for many years that EM should be a required portion of every medical school curriculum. [2] [3] [4] Moreover, the EM clerkship experience is integral for students being able to pursue EM as their chosen specialty, because it can generate a Standardized Letter of Evaluation (SLOE) required by EM residency programs to consider students for an interview. 5 While each EM rotation experience may vary, the goals of the clerkship have always been consistent: 1) to expose medical students to a wide array of acute pathologies within the ED clinical environment, 2) to teach them to work as part of a complex, interprofessional team that provides the highest quality unscheduled health care, and 3) to evaluate their clinical skills as they extend from basic history gathering and physical examination to diagnosis and applied management. 6, 7 In response to the unprecedented outbreak of COVID-19, the Association of American Medical Colleges released an announcement on March 17, 2020 strongly encouraging all medical schools to pause student clinical rotations. 8 Institutions across the United States immediately altered the traditional paradigm of clinical medical education, employing educational platforms such as video conferencing and virtual simulation to allow for distance learning in order to preserve resources and reduce COVID-19 transmission. 9,10 Schools and educators were forced to focus on how to minimize the loss of student learning due to reduced clinical exposure. Faculty from all specialties struggled to find ways to convert the traditional bedside learning obtained by students naturally over the course of a clinical rotation to specific virtual learning exercises and experiences.

In the aftermath of the emergent switch to remote learning, EM faculty are still struggling to discover the impacts this might have had on students. While individual clerkships have

This article is protected by copyright. All rights reserved independently shared their curricular adaptations via social and academic networking media, there is currently no aggregate collection or review on how EM clerkship directors as a whole have adapted their individual curriculum with the new pandemic restrictions. This knowledge can provide the impetus to create a lasting shift in the traditional clinical bedside learning for our students, thus ultimately affecting their knowledge acquisition and essential clinical skills for future global pandemics.

The primary objective of this study was to describe EM clerkship directors' (CDs) perspectives on their experience adapting an EM virtual rotation (VR) curriculum during the onset of the COVID-19 pandemic. Understanding insights from CDs and impacts these curricular changes had on students will help stakeholders better prepare future virtual learning experiences in the face of similar disasters. Inclusion criteria were: 1) age greater than 18 years old; 2) active CD at an accredited US EM residency program. Programs that did not answer whether they offered a VR or not were excluded from the final analysis.

This article is protected by copyright. All rights reserved

The survey was administered from late June to August 2020, and distributed via email to all members of Clerkship Directors in the Emergency Medicine (CDEM) listservs for dissemination to a convenience sample of all potential current EM CDs. Specific consideration was made to exclude any mention of "Virtual Rotation" in the recruitment email to promote greater survey participation amongst non-VR-adaptees. Given the known redundancies in membership within the CDEM listserv (N=383), the total eligible subject pool was extrapolated from the number of LCME accredited medical schools (N=155). 11

Eligible participants received a brief study summary that included details on enrollment, inclusion/exclusion criteria, data collection, result dissemination, as well as an anonymous link to the Qualtrics (https://qualtrics.com) survey for interested participants. Personal identifiers were not directly collected for data analysis; however, surveyed participants could indicate willingness to participate in a phone interview separate from the survey study for details regarding their curriculum. Emails were sent to the CDEM listserv every 2 weeks to encourage survey responses. The survey study was exempted by the Thomas Jefferson University Institutional Review Board (IRB study #20E.550).

Quantitative data were analyzed with descriptive statistics using Excel 2016. Thematic analysis using a constructivist research paradigm was performed on the qualitative comments. Authors performing the analysis were all EM faculty involved in undergraduate medical education at their respective medical schools. Their experience as faculty ranged from less than one to 17 years and roles varied, including clinical faculty, clerkship director, and assistant dean for undergraduate medical education. This diversity of experiences helped guard against potential gaps in analysis. The authors had no contact with the respondents and were not affiliated with their respective institutions. This helped prevent undesired subjectivity during analysis such as interpreting certain results with reference to the local institutional context. Two authors (XCZ, RR) independently generated common themes from open coding of free-response survey narratives. All discrepancies were arbitrated by third and fourth authors (KP, ME). Final results were reviewed and approved by the entire research team.

The survey received a total of 77 responses, 15 were excluded (4 did not meet the inclusion criteria, 11 were excluded for not answering whether they adapted a VR or not). Of the remaining 62 responses, 3 were duplicates which were merged after checking for contradictions.

This resulted in 59 analyzed EM clerkships (37 required rotations, 22 elective rotations).

Among these, 52.5% (n=31/59) clerkships adapted a VR while 47.5% (n=28/59) did not. Of those conducting VRs, 22.6% (n=7/31) were for MS3 only, 61.3% (n=10/31) were for MS4 only, and 16.1% (n=5/31) were offered to both. Statistical calculation for the remainder of the data was based on completed optional survey questions. Respondents' experiences adapting and running a virtual rotation are described in Table 1 .

Regarding the VR curricular content, respondents estimated that an average of 28.89% was adapted from the existing in-person clerkship, 35.37% was newly created by the program, and 35.74% was adapted from outside sources such as ALiEM Bridge to EM. Respondents were asked about changes in their utilization of several instructional methods while adapting from an in-person to a virtual rotation. Respondents were also asked to reflect upon their spring VR experience and how it might impact their future EM rotations through a series of free-response questions. Major themes, their frequency, and their example quotes are described in Table 2 . Finally, respondents were asked to provide some resources that they found useful while conducting the VR. They are described in Table 3 , arranged by instructional category and cost.

This article is protected by copyright. All rights reserved A crisis, such as the current COVID-19 pandemic, necessitates change and challenges all facets of medical education, including the EM clerkship experience. Emergency medicine is a unique clinical experience and a profession in which many aspects of society and medicine merge together in often unparalleled acuity, making it particularly difficult to replicate the experience in any other venue. Despite these challenges, just over half of our survey respondents utilized a virtual curriculum, the majority of which were oriented towards fourth-year medical students.

The ability to create and deliver such curricula in a time-compressed fashion is an example of adaptive expertise in medical education. Experienced educators are able to apply their knowledge and skills to adapt to a novel situation.

The timeline for VR curriculum development was particularly short when compared to the usual timeline of months to years for individual courses or larger institutional curriculum reform. 12, 13 Nearly all virtual curricula were generated in less than four weeks with most in less than two weeks (Table 1) . Online course development typically requires more time investment than inperson course development and CDs who were developing VR curricula overwhelmingly had the same or increased clinical responsibilities. 14 This emphasizes the need to distinguish between emergency remote learning (e.g., a sudden need to transition to online only or virtual learning) versus the development of planned online educational experiences. While EM educators demonstrated that they can accomplish emergency remote learning, this is not equivalent to formal development of pre-planned VR experiences.

A strong majority of faculty felt that the virtual format limited the ability to interact and connect with students, and also felt that it was more difficult to evaluate students ( Table 1) . We suspect a direct link between these two elements. Though a causal relationship cannot be formally established here, an education trend in EM prior to COVID-19 pandemic stressed the importance of direct observation in an authentic clinical environment. 15 We suspect that the limitations on both formal and informal student-faculty interaction in the work place challenged faculty's ability to assess these learners, leading to their reluctance to formulate a SLOE.

The narrative data also identified areas that might be amenable to assessment during VRs, namely teamwork, professionalism, and problem solving. This is consistent with reports in

This article is protected by copyright. All rights reserved literature of these qualities being assessed in online health professions course work. 16 Furthermore, for the areas of teamwork, professionalism, and problem solving, respondents suggested that the assessment methods, rather than the assessed content, were the real limitations in their ability to evaluate students. As assessments typically used in the clinical setting, such as direct observation, may not translate easily to a VR, it may be prudent for medical educators to consider this when designing future programmatic, learning, and clerkship objectives.

Prospective attention to this matter may help with the integration of virtual learning within the medical school.

Further, a lack of confidence in an online assessment tool is endorsed by the trend of shifting grading schemes to pass/fail (74%), a significant increase from 12.7% reported in literature. 17 A lack of confidence in assessment tools likely yielded the use of binary grading in order to not penalize students for limited assessment methods and to protect faculty from discerning too much detail from limited interactions.

These results are not unexpected, as EM clerkship faculty and directors would not be expected to routinely assess students in a virtual platform and therefore not have robust or well-developed virtual assessment tools. It stands to reason that faculty development targeted at the creating and utilizing virtual assessment tools would be a worthwhile investment should this type of education continue or be forced upon institutions during a critical sub internship season (the summer months for EM).

Survey respondents overwhelmingly felt that a virtual clerkship held in the future should solely be utilized as a "backup" resource, with a small minority suggesting that some combination of live and virtual experiences might be beneficial in the future. Given the limitations acknowledged by our respondents, one could speculate that the EM virtual curriculum, without the accompanying clinical experience, is not ready for "prime time." However, as the country continues to experience increased COVID-19 cases and face potential for other pandemics or natural disasters that may impact in-person learning in the future, one might posit that the development of a virtual curriculum, even as a backup, requires further investment.

This article is protected by copyright. All rights reserved Our study group doubts that a virtual clerkship would replace a fully immersive in-person EM experience. However, this study reflected the underlying flexible nature of EM, as educators sought innovative ways to teach, assess, and evaluate learners through a critical time period.

As is common with survey data, the primary limitation of this study is in the response rate (37%) from available EM clerkships nationwide despite the email reminders. There may be selection bias toward those adapting VR though we made specific efforts to avoid leading words, such as "Virtual Rotation," in the recruitment emails. Our survey also did not distinguish between elective and required rotation experiences which may have had differing curricular design due to their respective learning priorities and expectations. While the qualitative data was insightful, the responses were often concise and limited by our inability to ask follow up questions within the survey.

The survey evaluated the spring EM clerkship experience during the height of restricted learner presence within the clinical environment, a time in which most EM-bound students had already completed their sub-internship or away rotations. While our survey questions about SLOEs do not fully reflect true perspectives from sub-internship experiences later in the summer, we intentionally chose to sample the earlier experience as all EM clerkships across the country were forced to ban in-person learning during this time. We hypothesize that by summer, many hospitals may have instituted rigorous protocols to re-introduce medical learners into the ED, and this availability of choice would impact perspectives on SLOE writing for a virtual-only rotation.

More data is needed to assess the sub-internship experience of students during the summer and fall of 2020 with consideration to differences between virtual and in-person rotation experiences and changes in available in-person clinical experiences for students given ongoing COVID-19 restrictions from medical schools. It is also concerning that feedback suggests that education directors may not accept a VR SLOE as a viable assessment of a student's clinical skills.

Specific requirements for acceptable evaluation need to be defined within our community to allow for beneficial rotation experiences for students.

This article is protected by copyright. All rights reserved Many of our survey respondents had limited or no experience in designing a virtual curriculum prior to being required to institute one. Although we sincerely hope we may never encounter a situation necessitating student withdrawal from EDs again, it is not unimaginable to picture a future where this may happen. Our education faculty should have access to appropriate training to allow for them to feel more qualified and comfortable instituting a VR experience. 

The Macy Foundation Report

Emergency Medicine: a Unique Opportunity for Medical Students

The Emergency Medicine Rotation: a Unique Experience for Medical Students

American College of Emergency Physicians. Guidelines for Undergraduate Education in Emergency Medicine

Choosing Emergency Medicine: Influences on Medical Students' Choice of Emergency Medicine

The State of the Clerkship: a Survey of Emergency Medicine Clerkship Directors

Association of American Medical Colleges. Important Guidance for Medical Students on Clinical Rotations During the Coronavirus (COVID-19) Outbreak. Press Releases Medical Education

A Year Without Conferences? How the Coronavirus Pandemic Could Change Research

Leveraging Existing Quality Improvement Communication Strategies during the COVID-19 Crisis: Creation of an Emergency Department COVID-19 Case Conference

Association of American Medical Colleges. 2020 FACTS: Applicants and Matriculants Data

Leadership Lessons from Curricular Change at the University of California

Internet Resources for Curriculum Development in Medical Education

Instructor time requirements to develop and teach online courses

Direct Observation of Clinical Practice in Emergency Medicine Education

Principled Assessment Strategy Design for Online Courses and Programs

Distribution of Honors Grades Across Fourth-Year Emergency Medicine Clerkships

This article is protected by copyright. All rights reserved