key: cord-0004339-2yqkoslh authors: Yiu, Stella H. M.; Spacek, Alena M.; Pageau, Paul G.; Woo, Michael Y. C.; Curtis Lee, A.; Frank, Jason R. title: Dissecting the Contemporary Clerkship: Theory‐based Educational Trial of Videos Versus Lectures in Medical Student Education date: 2019-07-16 journal: AEM Educ Train DOI: 10.1002/aet2.10370 sha: 7a1bfd0bc9417d7b7a80c4e7edd817d8b279a15b doc_id: 4339 cord_uid: 2yqkoslh BACKGROUND: Despite increasing use of the flipped classroom (FC) technique in undergraduate medical education, the benefit in learning outcomes over lectures is inconsistent. Best practices in preclass video design principles are rarely used, and it is unclear if videos can replace lectures in contemporary medical education. METHODS: We conducted a prospective quasi‐experimental controlled educational study comparing theory‐based videos to traditional lectures in a medical student curriculum. Medical students enrolled in an emergency medicine clerkship were randomly assigned to either a lecture group (LG) or a video group (VG). The slide content was identical, and the videos aligned with cognitive load theory‐based multimedia design principles. Students underwent baseline (pretest), week 1 (posttest), and end‐of‐rotation (retention) written knowledge tests and an observed structured clinical examination (OSCE) assessment. We compared scores between both groups and surveyed student attitudes and satisfaction with respect to the two learning methods. RESULTS: There were 104 students who participated in OSCE assessments (49 LG, 55 VG) and 101 students who participated in knowledge tests (48 LG, 53 VG). The difference in OSCE scores was statistically significant 1.29 (95% confidence interval = 0.23 to 2.35, t(102) = 2.43, p = 0.017), but the actual score difference was small from an educational standpoint (12.61 for LG, 11.32 for VG). All three knowledge test scores for both groups were not significantly different. CONCLUSIONS: Videos based on cognitive load theory produced similar results and could replace traditional lectures for medical students. Educators contemplating a FC approach should devote their valuable classroom time to active learning methods. perhaps we can design a better medical student curriculum. The FC classroom incorporates active learning, with teachers explaining in-depth knowledge instead of merely providing factual information, and students spend time discussing with peers. 9 Apart from being interactive, 8, 10 there is much heterogeneity in how inclass activities are conducted, and little is known about how specific in-class activities contribute to learning. 7 When we focus on preclass videos, literature comparing videos to lectures is unclear, with some studies favoring videos, 1, 2, [11] [12] [13] [14] and some finding no difference in learning outcomes. 12, [14] [15] [16] [17] In other studies, video groups showed worse learning outcomes, 3, 12, [18] [19] [20] [21] with authors citing reasons including no peer interaction, low compliance with learning plans, and lack of accountability. 1, [3] [4] [5] Studies reported suboptimal use of videos, 7 with view rate from around 60% [22] [23] [24] to 83%. 25 With no clear evidence to support videos, some authors have urged educators to stop replacing lectures with them. [26] [27] [28] If we dissect these videos further, the variation in learning outcomes might be due to differences in design. 29 Few curriculum developers subscribed to guidelines guided by pedagogical principles. 29, 30 As there is diverse configuration, methods, and presentation, 31 , 32 Mayer's multimedia design principles based on cognitive load theory have been espoused to guide future designs. [33] [34] [35] In Mayer's approach, multimedia instruction has the potential for cognitive overload, as the learner's cognitive processing exceeds the available capacity. 34, 36 His principles reduce cognitive load to maximize learning. 22, [36] [37] [38] However, with few exceptions, most studies did not explicitly reference any design. 36, 39, 40 Scant description of video production was typically limited to the software programs and hardware equipment used. Format, if mentioned, varied. 27 Examples included interactive games, 31 animations, 33 or videotaped lectures. 22, 37 Given the lack of detail in these preclass materials, study replication is difficult. Therefore, while the in-class components are important, it is possible that video design partly accounts for the variability in published FC outcomes. To compare and replicate studies, it would be useful for preclass videos to adhere to good practice for format and design. More specifically, we need to establish whether videos adhering to Mayer's principles could replace lectures in the FC model. We set out to determine if educational videos adhering to Mayer's multimedia design principles resulted in any difference in learning outcomes for medical students when compared to traditional lectures in the setting of an emergency medicine undergraduate clerkship. We conducted a quasi-experimental controlled study to evaluate the effectiveness of videos compared with lecture for teaching medical students trauma assessment. All 113 third-year English stream medical students entering their mandatory emergency medicine rotation from the university were invited to participate and enrolled in a continuous fashion from March 2013 to March 2014. We received approval from the research ethics board from the Ottawa Research Health Institute and the University of Ottawa. As per institutional review board-approved protocol, students were informed that their participation and results would not affect their rotation assessment. One to two keywords only: "airway assessment," "pneumothorax." Multimedia principle: present words and pictures rather than words alone On "mechanism" of trauma, a single photo of a wrecked car is shown. Signaling principle: highlight essential material Included intro and summary. Slide heading emphasized key point. Contiguity principle: place printed words near corresponding graphics On slide with tension pneumothorax keywords "tension pneumothorax." Pretraining principle: provide pretraining in names and characteristics of key concepts On primary survey slide included the list of "airway, breathing, circulation, disability and exposure." An emergency medicine clerkship director created the content for an "approach to trauma" presentation with university curriculum objectives. The PowerPoint (Microsoft Corp.) slide design adhered closely to Mayer's principles 38 (see Table 1 ). There was minimal text (coherence principle). Instead, there was relevant nonanimated graphics with a few words next to them (multimedia principle). Each section started with a highlighted introduction and summary (signaling principle). These slides formed the basis of the lecture and the videos. The planned content was written as a script. The same instructor used the slides and script to record five voice-over videos (spanning 6-12 min). The videos further engaged Mayer's personalization and voice principles. Specifically, content was segmented into five videos (segmenting principle). Video contained narrated script (modality principle) in a conversational style (personalization principle) by a person (voice principle). The videos also use contiguity principle by showing words at the exact time of narration placed near corresponding graphics (see Figure 1 ). The slides and videos were pilot-tested by another clerkship director and students. After suggested modifications, the videos were uploaded to a password-protected Web page. An embedded Google Analytics tracker logged visits anonymously. In the lecture, the instructor presented the slides over a 2-hour session with the planned script. The first lecture was recorded and audited by another instructor to ensure consistency with the planned script. Once consent was obtained, participants were enrolled in the study. The university had already randomly assigned students into groups (of either 15 or 16) prior to start of clerkship. These groups were created to rotate student through the various disciplines during clerkship training. These groups were then assigned to either lecture group (LG) or video group (VG) in an alternating fashion. On day 3, LG students attended the lecture and VG students were given access to the videos. On day 8, each participant performed alone in an observed structured clinical examination (OSCE) in a high-fidelity simulation scenario in trauma assessment and management. Each performance was recorded. Participants also completed three written knowledge tests on day 1 (pretest), day 8 (posttest), and week 5/6 (retention test). After data collection, LG students were also given access to the videos. Measurements. Our primary outcome measure was application of trauma management skills. Secondary measures included acquired knowledge and attitudes to the learning methods used. OSCE. The OSCE assessment consisted of a checklist (total 20) and an anchored global rating scale. 41 We delineated three domains of trauma management (airway, breathing, and circulation) and identified specific objective elements for each domain. Each item on the checklist requires an application of the cognitive domain of the objective. Each performance was recorded. Three raters (staff emergency physicians) independently reviewed two pilot videos and were trained on the checklist. After that, they discussed and revised unclear items on the checklist through consensus. These raters all had 5 or more years of experience rating OSCE performances. Two independent raters scored each performance on a later date. These raters were blinded to the group allocation and each other's score. There were three raters-rater 1 scored all recordings, with rater 2 and 3 scoring 50% of recordings, respectively. Knowledge Assessments. We created 60 multiple-choice questions based on representative sample of items in varying difficulty linked to the content domain. We followed previous guidelines for question construction. 42, 43 The questions matched the learning objectives of the videos and the lecture. They were reviewed by three expert content reviewers (staff emergency physicians) and piloted on a separate group of students before the study. We refined ambiguous wordings. The questions were randomly divided based on the main content domains tested into three 20-item tests. No interitem correlation for the content domains was analyzed. The participants answered these three knowledge tests on day 0, day 8, and week 5 or 6 depending on the rotation schedule. Survey. The survey assessed the students' experience and attitudes toward their assigned modality and learning style prior to the simulation. The question "Would you have preferred what you were assigned to?" was repeated after the OSCE simulation assessment. A research assistant extracted and collected the data. A random 10% of sample data was reviewed for accuracy. Study data were analyzed using SAS software. The experts agreed by consensus that a 10% difference in the OSCE score (an absolute value of 2) was educationally significant, in keeping with prior studies. 44 Assuming a normal distribution, 100 participants (50 in each group) would provide a power of 0.80 to detect a 10% difference between the groups (two-sided test at a = 0.050). Differences in examination scores between groups and their two-sided 95% confidence intervals (CIs) were calculated using Student's t-tests. Cohen's d was calculated for practical significance. Inter-rater reliability was calculated between the independent raters. Categorical qualitative data were analyzed using Fisher's exact test. For ordinal data in the surveys (Likert scales), nonparametric statistics were used. Participants were analyzed in the groups they were assigned to. All 113 students entering the emergency medicine rotation provided consent. Half of these groups (two VG, two LG) have had prior surgery rotations while the rest (two VG, two LG) have not. They all had previous OSCE experience. There were 104 recordings (49 LG, 55 VG) available for analysis (see Figure 2 ). The means of the OSCE scores were 12.61 and 11.32 for the LG and VG, respectively. The mean difference Figure 3 ). Inter-rater reliability was good between rater 1 and 2 (intraclass correlation [ICC] = 0.834, 95% CI = 0.685 to 0.909) and excellent between rater 1 and 3 (ICC = 0.953, 95% CI = 0.920-0.973). Secondary Outcomes Knowledge Tests. Test scores from 101 participants were available for analysis (48 LG, 53 VG). There were no statistically significant differences between the pretest, posttest, and retention scores for the two groups (see Figure 4 ). Cohen's d was 0.105 and 0.034, respectively, indicating no practical difference. Video Use. The highest numbers of visits occurred on the day prior to simulation (mean = 11.75, [10] [11] [12] [13] [14] [15] [16] . Of the VG group, 90% (45 of 50) self-reported that they had viewed all the videos. There were 25% (13 of 52) students who reported repeated viewing. The number of page visits over the rotation totaled more than the number of students, also suggesting repeated visits. These visits also captured potential LG students who were given access to the videos after data collection. Survey. There were 103 students who participated in the survey (51 LG, 52 VG). Most students agreed (somewhat or strongly) that there were clear learning goals (88% LG vs. 94% VG, p = 0.1562) and the correct level of complexity (90% LG vs. 94% VG, p = 0.4878) and organization (85% LG vs. 96% VG, p = 0.0923). The majority of students (73% LG vs. 65% VG) agreed (somewhat or strongly) that it was better than the average teaching session (p = 0.5754). Significantly more LG (80%) than VG (33%) agreed (somewhat or strongly) that the session encouraged collaboration (p < 0.0001) and was interactive (88% LG vs. 34% VG, p < 0.0001) but was more demanding (28% LG vs. 13% VG, p = 0.0109). Most students agreed (somewhat or strongly) that the session was flexible to meet their needs (82% LG vs. 85% VG, p = 0.5195) and answered their own questions (85% LG vs. 73% VG, p = 0.3348). Learning Preferences. Significantly more LG (71%) than VG (37%) preferred learning with a lecturer (p = 0.002), while more VG (73% VG vs. 55% LG, p = 0.1216) stated that they preferred learning on their own time. A third of both groups (33% LG vs. 34% VG, p = 0.9659) stated that they were prompted to look up their own resources. Application. Most students (75% LG vs. 81% VG, p = 0.4248) felt that the session would help them apply their knowledge in a case scenario. Prior to simulation, 81% of the LG and 73% of the VG preferred what they were assigned to (p = 0.3319). After simulation, 81% LG versus 67% VG preferred their assigned group (p = 0.0994). In our study, we set out to determine if educational videos resulted in any difference in knowledge application for medical students compared to traditional lecture in an emergency medicine undergraduate clerkship. We found no educationally significant differences in knowledge application, acquisition, and retention. Students were satisfied with the group they were assigned to, with no significant difference before or after the application test. Therefore, videos adhering to multimedia design principles could replace traditional lectures in producing similar knowledge acquisition and application in an OSCE setting. Inconsistency in learning outcomes in multimedia learning literature could be partly due to inconsistent adherence to design guidelines, 30 such as Mayer's multimedia design principles based on cognitive load theory. [33] [34] [35] Multimedia learning requires a substantial amount of cognitive processing in the verbal and visual channels. 34 This cognitive processing is limited. 45 Mayer's guidelines decrease channel overload and increase cognitive processing capacity by removing extraneous and confusing material. 34 In Mayer's controlled experiments, students did better in tests when these individual principles were used, 46 leading him to conclude that "cognitive load is a central consideration in the design of multimedia instruction." This echoes previous study that suggests that sound pedagogy is more important than technology. 47 The strength of our study is that we adhered strictly to Mayer's multimedia principles and attempted to keep curriculum confounders to a minimum. While a similar recent study with multimedia principle-based videos focused on satisfaction and attitudes, 48 we used skill application in an OSCE setting as our primary outcome. Our study contributed to the voices of previous work that videos adherent to multimedia design principles could replace traditional lectures. These videos could become reusable learning objects, freeing up valuable expert faculty time to afford interactive in-class learning time. Rather than comparing videos to lectures, future research should focus on what specific interactive elements would be suitable for in-class learning. There are a few limitations in our study. We did not randomize students individually, but used an alternating block sequence for the intervention (VG) and control group (LG) to allow for a contemporaneous control group with respect to clinical maturity and exposure to trauma cases. As the university assigned student groups before the study, it is possible that students with preferences to clinical topics might have requested to be in specific groups. This preference was not captured. While half of VG and LG had exposure to trauma patients before our study through previous mandatory surgery rotations, we did not collect data on previous experience in assessing trauma patients. We also did not collect data on previous exposure to multimedia learning and simulation training. These confounders could potentially affect our outcome, as those with previous exposure to trauma patients, simulation, or multimedia learning might have done better in the OSCE. As students were not blinded to the group they were assigned to, there is potentially bias in their performances. The lecture (1.75 hours) was longer than the combined length of the videos (0.76 hours). While we attempted to keep the material identical (same slides, annotations, and script and by auditing the first lecture), students asked questions during lectures. Those questions and subsequent answers therefore represented minor deviations from the planned script. Even though a tracker was used, we could not identify separate page visits to measure individual views. The LG had a one-time exposure to the lecture and the VG could access the videos at any time. It is possible VG had accessed the videos multiple times, as some VG self-reported watching them repeatedly prior to the OSCE and posttest. This could have skewed the results toward VG via spaced repetition. Cook 49 has suggested that media-comparative studies such as this should not be done as there are confounders even if the content is identical. Instead, he suggested research is best done within rather than between levels of hierarchy (configuration, instructional method, and presentation) to answer the question of "how" and "when" to use e-learning effectively. Other authors lamented that the scope of evaluation was often limited to satisfaction or knowledge acquisition. 50, 51 Addressing the concerns of Cook 49 about comparing different media, we attempted to keep potential confounders between the two methods (narrator, material, script, slides, vignettes, etc.) to a minimum. Videos strictly adhering to Mayer's multimedia design principles could replace traditional lectures in medical student skill acquisition and application. Medical educators should use well-designed videos to transmit content and devote classroom time to active learning activities. We thank the Clerkship Program and the Dean for Undergraduate Medical Program at the University of Ottawa and the University of Ottawa Simulation and Skills Centre for participation in this study. Lecture halls without lectures-a proposal for medical education Medical education reimagined Remixing chemistry class: two Colorado teachers make vodcasts of their lectures to free up class time for hands-on activities The flipped classroom: an opportunity to engage millennial students through active learning strategies The flipped classroom for professional development: part I. Benefits and strategies The use of flipped classrooms in higher education: a scoping review A systematic review of the effectiveness of flipped classrooms in medical education Flipped classroom improves student learning in health professions education: a meta-analysis Flipping the medical classroom: Effect on workload, interactivity, motivation and retention of knowledge Improvements from a flipped classroom may simply be the fruits of active learning Center for Technology in Learning. Evaluation of Evidencebased Practices in Online Learning: A Meta-analysis and Review of Online Learning Studies Otolaryngology -head and neck surgery in undergraduate medical education: advances and innovations Using comprehensive video-module instruction as an alternative approach for teaching IUD insertion Video-based instructions for surgical hand disinfection as a replacement for conventional tuition? A randomised, blind comparative study The Flipped Classroom: A Survey of the Research. 120th ASEE National Conference & Exposition A peer-reviewed instructional video is as effective as a standard recorded didactic lecture in medical trainees performing chest tube insertion: a randomized control trial Emergency skills learning on video (ESLOV): a single-blinded randomized control trial of teaching common emergency skills using self-instruction video (SIV) versus traditional face-to-face (FTF) methods Flip Your Classroom: Reach Every Student in Every Class Every Day Effect of a web-based curriculum on primary care practice: basic skin cancer triage trial Are live instructors replaceable? Computer vs. classroom lectures for EFAST training Hybrid E-Learning Acceptance Model: Learner Perceptions The flipped classroom in emergency medicine using online videos with interpolated questions Effectiveness of the surgery core clerkship flipped classroom: a prospective cohort trial The flipped classroom: a modality for mixed asynchronous and synchronous learning in a residency program Evaluation of an interactive webbased nursing course with streaming videos for medication administration skills What is the role of e-learning? Looking past the hype The Flipped College Classroom: Conceptualized and Re-Conceptualized Preparing for the changing role of instructional technologies in medical education Online lectures in undergraduate medical education: how can we do better? Quality specifications in postgraduate medical e-learning: an integrative literature review leading to a postgraduate medical e-learning model Flipping" the introductory clerkship in radiology What do we mean by web-based learning? A systematic review of the variability of interventions Web-based versus traditional lecture: are they equally effective as a flexible bronchoscopy teaching method? Nine ways to reduce cognitive load in multimedia learning Instructional design variations in internetbased learning for health professions education: a systematic review and meta-analysis Cognitive load theory in health professional education: design principles and strategies Pharmacy student engagement, performance, and perception in a flipped satellite classroom Applying the science of learning to medical education A novel collaborative e-learning platform for medical students -ALERT STUDENT Teaching medical procedures at your workplace Value of the OSCE as an assessment tool Set a multiple choice question (MCQ) examination Multiple-choice questions revisited Procedures can be learned on the web: a randomized study of ultrasoundguided vascular access training Cognitive load theory and the format of instruction A split-attention effect in multimedia learning: evidence for dual processing systems in working memory Technology Is the Tool, Teaching Is the Task: Student satisfaction in Distance Learning. San Antonio: ERIC Institute of Education Sciences Creating a contemporary clerkship curriculum: the flipped classroom model in emergency medicine The research we still are not doing: an agenda for the study of computer-based learning A global model for effective use and evaluation of e-learning in health Internet-based learning in the health professions: a meta-analysis