key: cord-0944477-hipdbikl authors: Na, Seung-Joo; Ji, Young Geon; Lee, Dong Hyeon title: Application of Bloom’s taxonomy to formative assessment in real-time online classes in Korea date: 2021-08-27 journal: Korean J Med Educ DOI: 10.3946/kjme.2021.199 sha: 8153f46a71a6b84f8f2ef29baea9b41c1da8a3b9 doc_id: 944477 cord_uid: hipdbikl PURPOSE: This study aims to design that using formative assessment as an instructional strategy in real-time online classes, and to explore the application of Bloom’s taxonomy in the development of formative assessment items. METHODS: We designed the instruction using formative assessment in real-time online classes, developed the items of formative assessment, analyzed the items statistically, and investigated students' perceptions of formative assessment through a survey. RESULTS: It is designed to consist of 2–3 learning outcomes per hour of class and to conduct the formative assessment with 1–2 items after the lecture for each learning outcome. Formative assessment was 31 times in the physiology classes (total 48 hours) of three basic medicine integrated. There were nine “knowledge” items, 40 “comprehension” items, and 55 “application” items. There were 33 items (31.7%) with a correct rate of 80% or higher, which the instructor thought was appropriate. As a result of the survey on students’ perceptions of formative assessment, they answered that it was able to concentrate on the class and that it was helpful in achieving learning outcomes. CONCLUSION: The students focused during class because they had to take formative assessment immediately after the learning outcome lecture. “Integration of lesson and assessments” was maximized by solving the assessment items as well as through the instructor’s immediate explanation of answers. Through formative assessment, the students were able to utilize metacognition by learning what content they understood or did not understand. Items that consider Bloom’s taxonomy allow students to remember, understand, and apply to clinical contexts. In the spring of 2020, most medical school classes switched from face-to-face classes to online classes in Korea due to the coronavirus disease 2019 (COVID- 19) pandemic. Within a short period of time, it has become an era of full-scale online classes and professors either have real-time online classes through online-conference platforms such as Zoom or Google Meet, or make a video for the class and distribute it to the students. Real-time online classes have the advantage of being of enabling interaction between professor and students; however, some teachers complained that it was difficult to check if students were participating in the class or understanding the content as students would either turn off their cameras or the small frames made it hard for them to see the students' facial expressions or gestures clearly. Serhan [1] in 2020 found that 48.4% of the students displayed a negative attitude toward Zoom classes compared to face-to-face lectures, while 61.3% of the students answered Zoom classes did not help improve their learning. It is especially necessary to focus on the negative answers regarding Zoom classes, such as "difficulty focusing on the lecture", "dissatisfaction over the quality of interaction and feedback", and "poor class quality" [1] . Against this background, Bao [2] in 2020 proposed the following five high-impact teaching practice principles to increase the quality of online learning: (1) maintain appropriate relevance between the amount, level, and length of class content and students' level; (2) control the pace of classes while effectively delivering content; (3) provide sufficient support through timely feedback, including e-mail guidance, from professors and teaching assistants to students after class; (4) heighten the level and depth of student participation during class; and (5) prepare a contingency plan to deal with unexpected problems that may occur on the online education platform. Thus, it has been confirmed through studies of Serhan [1] and Bao [2] that the sudden online classes due to the COVID-19 pandemic require a strengthening of design, operation, and students' feedback of the classes, in addition to the various efforts to improve the quality of learning. Recent assessment trends have moved past the summative assessment where the students' achievements were checked to emphasize the importance of formative assessments, where the assessment itself is used for learning and applied in learning. Formative assessment refers to an assessment to give feedback to students while teaching and learning is in progress and to improve the curriculum and teaching methods. Black and Wiliam [3] in 1998 regarded all feedback activities partaken by the teacher or students during class for the improvement of teaching and learning as formative assessment, and claimed that "assessment for learning" should be emphasized because formative assessments have a significant impact on student achievement. Formative assessment is emphasized and is being implemented and conducted in many classes and subjects. Many studies are also being conducted on the theories, effects, and method of formative assessment [3] [4] [5] [6] [7] [8] [9] . However, not many studies have been conducted on the "items" used in the formative assessment, and there exist studies that used the cognitive domain of Bloom's taxonomy (BT) [9] and applied cognitive diagnostic models [10, 11] . Kim et al. [12] in 2017 argued that when developing formative assessment items, all important elements of the learning unit and each stage of taxonomy of educational objectives should be included. In addition, in formative assessment, items of difficulty equivalent to the minimum criterion should be presented to ascertain whether a student has achieved or not [12] . However, there are insufficient studies on this matter. Based on literature review of Bao [2] and Kim et al. [12] , this study aims to exploring that (1) using formative assessment as instruction strategy to increase student's participation and to enhance understanding of contents in real-time online classes, (2) to achieve the goal of improving student learning and improving classes, which are the main functions of formative assessment, BT is applied to item development and its implications are considered. This is the study that developed the items of formative In order to increase the quality of online learning, Bao [2] in 2020 proposed to adjust the level and length of lessons appropriately, and to adjust the speed of lessons in consideration of the level of students and the contents of the lessons. Accordingly, the contents of the existing learning outcomes were adjusted to 2-3 per hour, and this allowed time to participate in online formative assessment and to enhance student participation and the interaction between the professors and students, the class was designed by including a formative assessment on related content, which was presented as the learning outcomes, as illustrated in Fig. 1 . Kim et al. [12] in 2017 said that taxonomy of educational objectives is important when developing formative assessment items. In Korea, primary and secondary school teachers use taxonomy of educational objectives when developing test or exam items, and taxonomy of educational objectives is composed of a two-dimensional matrix of assessment contents and behavioral elements [13] [14] [15] . The sub-elements of the behavioral elements may be different for each teacher, but the most used is the cognitive sub-behavioral elements of Bloom's cognitive domain [12] . BT (1956) 1) consisted of six major categories [16] : knowledge, comprehension, application, analysis, synthesis, and evaluation, and each category has a hierarchical structure in cognitive domain. The lowest level of 1) Bloom's taxonomy was revised in 2001, divided the existing cognitive domains into cognitive domains and knowledge domains. In addition, the cognitive domain was modified into "remember", "understand", "apply", "analyze", "evaluate", and "create" [16] . "knowledge" and the highest level of "evaluation" are required for cognitive abilities. In this study, only "knowledge, comprehension, and application" among BT were applied in the behavioral elements. Because there was a study that pointed out that teachers had difficulty distinguishing between knowledge-comprehension, application-analysis, and analysis-synthesis, behavioral element beyond "application" were excluded [13] . And we also thought that the higher-level "analysis, synthesis, and evaluation" require student's time to internalize the content of the lesson into their own learning. When developing the items, each item was developed with the content element as a learning outcome and the behavior element as "knowledge, comprehension, and application" were matched, and all items were developed as multiple-choice items. Application items that used physiology knowledge in clinical context were presented as formative assessment items; it was designed so that the students learned how basic medical knowledge could be applied in clinical context by solving these items. The number of formative assessments carried out in each physiological class during the three subjects, the frequency of items and analysis of the item difficulty by the categories of BT. For student responses to the questions related to formative assessment in the survey, descriptive statistics and frequency analysis were conducted using JAMOVI ver. 1.6.15 (JAMOVI, Sydney, Australia; https://www.jamovi.org), and the answers to the open-ended questions were summarized accordingly. While this study collected formative assessment data from the results of the previous year's class operation and survey data for the improvement of classes, it did not collect personal identification data of the study subjects. Thus, the institutional review board approved this study to be exempt from deliberation (1044308-202105-HR-032-01). Table 1 presents the number of classes, number of formative assessments, and number of formative assessment items per categories of BT for the FHB, BSCRS, and BSURS subjects taken by first-year students in 2020. Each subject gave students 2-4 formative assessment items after each class, which were distinguished into "knowledge, comprehension, and application" items. The instructor maintained 45%-55% inclusion rate for "application" items, so that the basic medical knowledge learned in class have been used in clinical context through formative assessment items. As mentioned earlier, in formative assessment, items of difficulty equivalent to the minimum criterion should be presented to ascertain whether a student has achieved or not. The instructor expected 80% of the correct answer rate to take each item. However, some Application items were thought to be difficult for students, but they were included in the formative Table 3 . Most students answered that they learned what content was most important through formative assessment. It was indicated that "comprehension" items helped the most in helping them understand the class content and the "application" items helped the most in achieving learning outcomes. As for the adequate timing of formative assessments, most students answered that it would be best to conduct formative assessments once at the end of each class. This The number of questions should be increased as it greatly helps students understand the content and apply the content in clinical situations. By solving the questions, I can check if I understood the content correctly. Some clinical situation application questions were too difficult to solve with just the content learned during class. The professor's explanation of the correct answers greatly helped learning, and solving one problem had the same effect as solving multiple problems. seems to reflect their burden toward the fact that they experience an assessment once or twice every class. Other opinions included the increase in the number of items and a modification of the difficulty. Online classes have become common in educational institutions around the world due to the COVID-19 pandemic. Agarwal and Kaushik [17] in 2020 proposed in their research that most of the learners of online classes using Zoom will be a part of medical education and believed online classes will be a part of the post-graduation curriculum even after the end of the COVID-19 pandemic. Due to such changes in teaching methods and the learning environment, many schools and professors are seeking ways to design effective online classes and increase student participation. Against this background, we would like to propose the utilization of formative assessments to increase the students' concentration during online classes, to allow instructors to immediately check how much the students understood the learning content, and to allow interaction between instructors and students. This study presented 1-2 formative assessment items after one learning outcome lecture to utilize formative assessments as an online instruction strategy. And when developing the items, "knowledge, comprehension, and application"-the categories of BT were matched to each item. The results are as follows: First, the students focused during class because they had to take formative assessment immediately after the learning outcome lecture, thus being able to utilize the knowledge acquired during the class. Second, instructor was able to immediately check the students' answers in Google Classroom, thus being able to provide instant feedback. In addition, instructor was able to immediately improve his classes, because he could assess the students' This study is about the timing of formative assessment during class, and the content of item development. Through this study, we would like to consider the method of formative assessment, the cognitive level to be measured during item development and appropriate difficulty, and the effect of formative assessment perceived by students. First, although formative assessment was not included in grades, it was found that students were stressed just by being frequently exposed to assessment situations. Studies on formative assessment through online platforms or mobile apps like Kahoot! existed before the COVID-19 outbreak [4] [5] [6] . In these pre-studies, several methods were suggested so that students could have fun and be interested in participating in assessment. Using a mobile app, students can be entertained like a game, or they can reward a student who answers more or faster than other students. It is beyond the assessment of the student's understanding of the lesson, and it is seen that evaluation itself becomes a part of learning, that is, lecture and evaluation are integrated. Second, the feedback provided after formative evaluation enables students' self-reflection and selfassessment, and through this, "assessment as learning" Earl [18] in 2013 has presented "assessment of learning", "assessment for learning", and "assessment as learning" as the paradigms of student assessment. According to Earl [18] , "assessment as learning" is a Earl [18] stated that "assessment as learning" must be most widely used, where students become the subjects of assessment and they check and adjust their own knowledge for further learning. Then, "assessment for learning" must be used, to enable instructor obtain information for their instructional judgement in teaching situations and provide effective feedback to students. What these two assessment paradigms have in common is that assessment is done to support learning rather than to provide information about the results, and that formative assessment as an assessment method is preferred [19] . Third, items that can maximize the function of formative assessment should be developed. According to Seong [10] in 2018, the general characteristics of test tools for formative assessment are as follows: (1) Transitioning from face-to-face to remote learning: students' attitudes and perceptions of using Zoom during COVID-19 pandemic COVID-19 and online teaching in higher education: a case study of Peking University Assessment and classroom learning Integrated online formative assessments in the biomedical sciences for medical students: benefits for learning Using Kahoot! as a formative assessment tool in medical education: a phenomenological study Correlation of online assessment parameters with summative exam performance in undergraduate medical education of pharmacology: a prospective cohort study Studies of the effect of formative assessment on student achievement: so much more is needed Students' perception of formative assessment as an instructional tool in medical education Medical teachers' action research: the application of Bloom's taxonomy in formative assessments based on rain classroom Development and analysis of formative assessment for mathematics 'Number and Operation' in the first grade of middle school by the Cognitive Diagnosis Model Effects of computerized formative assessment with the Cognitive Diagnosis Model on achievement and affective attitude for math Formative assessment for classroom practice A study on the improvement for building the taxonomy table of two dimensional objectives based on it's actual condition survey in the standards based assessment of achievement An analysis of content validity of behavioral domain of descriptive tests and factors that affect content validity: focus on the fifth and sixth grade science An analysis of secondary school teachers' evaluation professionality on Bloom's taxonomy and achievement standards A taxonomy for learning, teaching, and assessing: a revision of Bloom's taxonomy of educational objectives Student's perception of online learning during COVID pandemic Assessment as Learning: Using Classroom Assessment to Maximize Student Learning A study on the future direction of the online formative assessment system for learning Korean Language Arts Acknowledgements: None.