key: cord-0059998-9iyuzic8
authors: El-Din, Doaa Mohey; Hassanein, Aboul Ella; Hassanien, Ehab E.
title: The Effect CoronaVirus Pendamic on Education into Electronic Multi-modal Smart Education
date: 2020-07-29
journal: Big Data Analytics and Artificial Intelligence Against COVID-19: Innovation Vision and Approach
DOI: 10.1007/978-3-030-55258-9_15
sha: 87c1b2bdd54a4011beb426ae17f59adfe4c5a179
doc_id: 59998
cord_uid: 9iyuzic8

This paper presents how coronavirus drives education to smart education in interpreting multi-modals. It uses to improve the electronic learning in multiple data types. This paper is a survey paper about the importance of smart education and the effect of coronavirus on drives education into smart online education. It also presents many changes in the education vision around the world to utilize multi-modal for enhancing E-learning. The combination of artificial intelligence and data fusion plays a vital role in improving decision making and monitoring students remotely. It also presents benefits and open research challenges of a multi-modal smart education. This main objective of this paper is to highlight the deepening digital inequality in smart education in emergencies due to Coronavirus, the concept of digital equality has been defined as equal opportunities in accessing technology as hardware and software as well as equal opportunities in obtaining equal digital education through Ease of access to high-quality and interactive digital content based on the interaction

The Coronavirus pandemic forced the governments of the world to close educational institutions that caused 89% (more than 1.5 billion learners) from 188 countries to be forbidden access to educational institutions to receive face-to-face education as the UNESCO report in 2020 [1] . Many of these institutions are undertaking a large unplanned experience which is remote education in emergency situations, distance education in emergency situations [2] in order to limit the spread of the virus. The sudden transformation of smart education in emergency situations, to shock and tension among students and faculty members, whether on the personal or professional level, because of the process's need for redoubled efforts, or psychological instability due to the outbreak of the epidemic, as well as several unusual obstacles for school students And universities: such as lack of appropriate time, poor infrastructure, inadequate digital content, etc.

The disparity between countries in terms of technological development of technology operations in developing various fields in those areas to digital inequality and social justice, whether between countries with each other or within the countries themselves. The hypothesis that digital technology helps to achieve justice and social equality has been brought down, as in reality, digital inequality has deepened in distance teaching in emergency situations. Digital equality and the right to education is one of the United Nations' sustainable development goals. Previous researches reach the digital equality in education, especially higher education, which is a big obstacle that hinders the adoption of digital tools in higher education all over the world.

To achieve the goal of the paper, which is to highlight the deepening digital inequality in E-learning in emergencies due to Coronavirus, the concept of digital equality has been defined as equal opportunities in accessing technology as hardware and software as well as equal opportunities in obtaining equal digital education through Ease of access to high-quality and interactive digital content. This paper presents a review paper of smart education in multi-modals of multiple data types. It shows the importance of smart education. It also presents many changes in the education vision around the world to utilize multi-modal for enhancing Elearning. The combination of artificial intelligence and data fusion plays a vital role in improving decision making and monitoring students remotely. It also presents benefits and open research challenges of a multi-modal smart education. It presents the advantages and challenges of implementation the smart education system in various countries. It also shows open research challenges of smart education.

The rest of the paper is organized as the following: Sect. 2, the effect Coronavirus on Education and discusses the importance of smart education, Sect. 3, Presents a discussion of Smart Education and previous works in this area, Sect. 4, benefits and challenges of smart education, Sect. 5, open research challenges and research directions in smart education, section Finally, Sect. 6, targets the conclusion outlines.

Coronavirus becomes a global pandemic around the world. The World Health Organization (WHO) reports that the number of confirmed cases by a coronavirus (COVID-19) reaches 4,993,470. And the number of deaths reaches 327,738 deaths that achieves more than 15.3% [3] . It has a big spreading rapidly in all Continents. The United Nations Educational, Scientific and Cultural Organization estimates that 192 countries have closed schools across the country [4] and that many other countries have implemented regional or local closings. It causes of closed schools, educational institutions, faculties, Scientific organizations, and universities. Millions are forbidden from their education suddenly due to protect their lives and their families.

The educational institutions are under the closure's role of the United Nations Educational and World Health Organization for 80% of students worldwide. That causes several challenges for applying smart education applications due to the poverty and lack of automated applications of smart education and the available internet for all students concurrently. It also requires the effectiveness application based on various data types such as video, audio, and text. The development of applications not only for education but also used for training in many domains such as medical schools about how to deal with Coronavirus disease.

UNESCO recommends for all countries to turn to infer the technology in the automated education remotely. It depends on the reliability of Internet-of-things for smart education [5] . So, smart education becomes a new trend to benefit from the combination of internet-of-things and artificial intelligence in smart education. Millions of petabytes of data are used for smart applications based on the combination of artificial intelligence and internet-of-things. The artificial intelligence is utilized for interpreting big volume of extracted data, fusing heterogeneous data with multiple data formats, and understanding the velocity of the interpreting data. The fusion process between huge sensed data allows the capture of complementary information or trends [6] . The simulation of multimodal application is described by high exciting, fast, and high quality.

The properties of smart education, self-directed, motivated, adaptive, resourceenriched, and technology-embedded [6] . A smart education environment presents access to ubiquitous resources and interacts with learning systems anytime and anywhere. The vital learning orientation, suggestions, or helpful tools to them in the right form, at the right time and in the right place.

Smart education environments can provide accurate and rich learning services by using learning analytics. It depends on many important features for any education application, location, context-aware, social relationship, Interoperability, Adaptability, flexibility, Flexibility, Adaptability, Ubiquitous, Human computer interaction multimodal, and High Engagement. Sensed Locations, it can determine the location of each learner remotely from the location sensor in real-time. These sensors often are based on mobile sensors such as GPS in the mobile that use for detecting and tracking locations. Context-Aware, it examines various scenarios in various contexts and their information about each activity. There are several targets and properties for each educational scenario such as lectures, exams, student affairs, Graduate Studies, Make certificates, etc. Social Relationship Awareness, it discovers the social relationship from used sensors. Flexibility, it is high flexible to fuse multiple information from multiple sensors. Interoperability, it puts many standard roles that depends on various resources, activities, or services. It is considered an important step and obstacle till now to fuse multiple data source with one or variant data formats. Adaptability, it infers the education access, preference, and demand from education resources. Ubiquitous, it is based on the predication of education demand to construct system clear and visualized to access education resource to the learner. Human Computer Interaction (HCI) multimodal, it is based on the fusion between sensed multimodal interactions that consists of the position and facial expressions. High Engagement, it enters a multidirectional interaction for education experience with a technologyriches environment. The interconnected and interoperable education experience are considered high significant issues in a smart education system in the future.

Smart education or E-learning is the process of exploiting technology in communicating with the elements of the educational process among them as shown in the main architecture in Fig. 1 . Electronic learning is a complex process that starts with planning to design decisions based on the needs of the target group. The evaluation process was designed and then the course was published. Although there are many benefits to simulating smart education as shown in Fig. 4 [7] [8] [9] , for example, facing increasing student numbers, reducing the financial burden on educational institutions resulting from this by reducing the numbers of students in its facilities, as well as the number of employees in them, and providing an opportunity for students who live Far areas of enrollment without having to change the place of residence, or refrain from sending children to school, and finally the inevitable solution to dealing with the problems that resulted from the closure of schools and universities in light of the spread and spread of the Corona virus. The smart education is designed based on the combination of artificial intelligence and internet-of-things. The powerful role is shown in interpreting big data and data fusion as shown in Fig. 2 [9] . Smart education includes multi-data fusion in various data types from various sources that may be sensors or smart devices on education applications. It provides educational services such as digitalized context awareness, adaptive content, collaborative and interactive tool, rapid evaluation, and real-time feedback. Online smart education applications are effective and efficient in meaningful learning. It depends on the combination of increasing the effective of many interfaces, using smart IoT sensors or devices, and different education scenarios with various features and data [7, 8] .

Previous motivations in the implementation of smart education, Researchers in [9] , presents a framework of a Smart Education Environment System (SEES), that support the integration database based on the incorporating three core sub-systems. These sub-systems include 'Electronic Bookshelves', use for the automation access and remotely for each book shelve that is entitled 'Virtual White Space'. The examination of library data includes Database for social network. It is considered integrated system that improves adaptive system.

Researchers in [10] , present a stem education system based on multimodal fusion for smart classes to improve management and analytics. Researchers in [11] , shows a smart system for generative learning based on cloud computing, fog computing, and swarm computing. Researchers in [12] , present an application for smart learning for multiple contexts that can fuse several virtual education applications from a physical classroom. A recent idea of Education is considered it is as a Service, so it includes many challenges for the markets. Researchers in [13] , create a new Application Framework for Smart Education System as entitled SES Framework. It depends on a Model-View-Controller (MVC) with multi-aspect model.

Researchers in [14] , present the system is designed for the open-source technologies and services in order to make it capable of supporting the open IT-infrastructure and providing from various commercial hardware/sensor vendors furthermore opensource solutions. The application is developed for offer new app-that relies on educational solutions for various educational objectives or for controlling the efficiency remotely. The application is described by replicable and adaptable to settings that may be various than the scenarios envisioned here.

Researchers in [15] , present a university model that reduces complexity and high adaption, which requires improving performance. Multimodal data contains the expressions of facial and review when the students monitor online videos and label them with two dimensions (interestingness, difficulty) in the subjective learning status. Then after pre-processing such as face recognition from video screenshots and normalization of review, they constructed a new model for fusion based on artificial neural network methods to compute the real-time learning status in the two dimensions. Smart education includes multi-data fusion in various data types from various sources that may be sensors or smart devices on education applications. It provides educational services such as digitalized context awareness, adaptive content, collaborative and interactive tool, rapid evaluation and real-time feedback. Online smart education applications are effective and efficient in meaningful learning. It depends on the integration of increasing interfaces, smart devices, and different learning data [7] (Table 1) .

Using Multimodal for smart education applications has a great effect on predication and making decisions [16] . The construction of multimodal is based on fusing data from multiple sensors. It improves predictions and detecting changes from various sources. the multimodal for Audiovisual properties that are fused for detecting the depression symptoms in [17] based on using a constructed dataset from dyadic interactions between an interviewer and paid participants.

Several fusion techniques are constructed based on the properties of video, audio, and transcripts [18] . The dataset is made through interviews conducted by an animated virtual interviewer managed by a human in another room. Researchers in [19] , present a survey paper about a comprehensive study of recent motivations on multimodal deep learning from three dimensions: learning multimodal representations, fusing multimodal signals at various levels, and multimodal applications. The construction of multimodal is based on determining types of data formats and determining the fusion technique, signals, text, video, and image. Researchers in [20] , multimodal learning analytics is presented by recent insights into student learning trajectories in more complex and open teaching applications. Many models based on deep learning are used for extracting powerful data from multiple modalities. The applications rely on Convolutional Neural Networks (CNNs) use for extracting the visual and audio properties and a word embedding model for textual analysis [21] . Researchers in [21] , present a novel fusion technique for integrating different data representations in two levels, namely frame-level and video level. The results reach enhancing accuracy by more than 16% and 7% compared to the best results from single modality and fusion models.

The fusion process of modalities from various data formats is a very significant process in order to reach the highest performance and harvest relevant data. Classical data fusion techniques often contain the level of fusion, early, late, or middle fusion [22] . The existing studies of multimodal fusion studies contain the fusion level, but are not limited to, video (audio-visual) analysis [23] , social networks [24] , or human-computer interaction [25] . Researchers in [24] show a modal for analyzing sentiments that is based on tri-modalities textual, video, and audio. It uses a tri-modal Hidden Markov Model (HMM) for classifying models and determining the hidden interaction among them.

Deep learning techniques have been recently proposed and applied in many research, competitions, and real-world applications [26, 27] . The researchers in [23] presents a deep learning model as entitled Microsoft Residual Networks (ResNet) that solves the overfitting and vanishing gradients challenges. Researcher [28] presented a simple CNN based on word vectors for sentence-level classification. The proposed sentiment application targets analyzing sentiments and questions. Recurrent Neural Network (RNN) and its extended version Long Short-Term Memory (LSTM) has also been leveraged to classify speech [29] and text.

Although the fusion topic is an old topic, it has recent motivations especially in smart environments. Multimodal is the essential topic of research in fusion in Internet-of-things due to the massive extracted of sensory data. The used techniques of multi-modal fusion target fusing multiple data types from multiple resources.

The implementation of smart education applications is very powerful and useful for electronic learning. This section discusses the benefits and challenges of smart education applications.

The smart education has several advantages as shown in Fig. 3 .

(1) Saving time Electronic learning is described by quick communication, save time in asking lecturer, save time in mobility or transportation. The E-learning enables students to make online presentations and get the review about their work concurrently.

Internet-of-Things (IoT) big data fusion (DBF) Ar fical Intellgnace (AI) Fig. 3 The smart education application is constructed based on three levels (AI, IoT, and BDF)

(2) Increased productivity Productivity is raised by sharing information based on multiple data formats from multiple data sources. The good smart education system aims to increase the productivity of lecturers and students [30] .

(3) High flexibility

The interaction between student and lecturer becomes high flexible in requirements achievement [31] .

(4) Improving Learning performance Using smart education online technology is higher performance of understanding photos, maps, graphs, flowcharts and animated videos. This makes learning more attractive, interesting, and easy to understand. It encourages the ability of students to learn and memorize the topic for a prolonged period.

It is a universal truth, when we learn through visuals, we grab the subject easily rather than just looking into the blackboard & listening.

The interactive process between lecturers and students become reliable for smart technology by 55% [32]. The smart education system becomes easy and simple to construct a fast FAQ session between lecturers and students that makes a wonderful learning environment in the electronic classrooms.

(6) High interactive Access system A smart interactive system is constructed based on some practical solutions. This system becomes adoptive to reach 70% of smartboards. This is an obvious indication that educational institutes are embracing this advanced technology.

The world countries are forced to construct smart education systems although the variant of availability of capabilities, training, and fund in each country. These smart educations become very important due to protect students' lives. So, the implementation of smart education in various countries face many obstacles and challenges are classified intro three types as Fig. 4 as the following:

(1) Technical challenges in Hardware, software and the availability of the internet The internet services are not enough in speed and quality in all students and states whether that serves universities' students or schools' students. That requires to find suitable hardware as computer or laptop or mobile to be available online, present work and share assignments online. variety of importance of specific branches in explanations in high school or universities, and hardness of implementation the unification multi-modal of education system. The lack of the digital content in high-quality for various student's levels, that challenge shows the problem in construction digital quality digital educational tutorials, assignments, and projects. It also the processes of the exchanging file between lecturer or teacher and students or other lecturers. That requires a huge database and complementary tutorials to cover changes in lecturers' tutorials and exploitations. A large percentage of teachers are unable to use digital tools in teaching, which reflects negatively on controlling the digital class through remote teaching in emergency situations.

Suitable, centralized digital content with many local languages as found in many countries that have multiple local languages, so there is a need to constructing digital content with many local languages to be suitable with people's expectations due to the lack of appropriate time for the lecturer/teacher to teach.

The variety of importance of specific branches in explanations in high school or universities. Each education level has one subject, or more is very important. So, most of teachers or lecturer's emphasis on these scientific explanations and ignoring emergency situations. Finally, the challenge of construction interpreting multi-modal for interpreting or summarize contents and evaluating students in multiple data types.

That is shown in two problems, the hardness or follow students in schools from parents and the difficulty of monitoring student progress in cumulative exams, assignments, and summaries.

The Hardness of follows school's students from Parent. The technical inability of parents follows their school student's children in their homework at home. The hardness of learning parents' new smart educational models in the digital equality process. The second problem is shown in the difficulty of monitoring student progress in cumulative exams, assignments, and summaries.

This paper presents an open research direction to implement and improve the construction of smart education systems. these systems can benefit from the artificial intelligence and internet-of-things in high-quality digital content for electronic education and monitoring the students' progress based on fusion multi-modals of their data. E-learning has become a refuge for any educational system in the world, and its officials must quickly make a decision to impose e-learning within our teaching methods. If the society was ready for e-learning, moving to it in the event of suspending the study for any reason would be easy for everyone, and when e-learning took all this debate from educators.

A Flexibility of constructing a multi-modal [33] for digital content. The process of preparing curriculum components fuses multiple data sources from multiple teachers in various tools or from various sensors such as mobile sensors. The data fusion is a vital opportunity to build multi-modal smart education applications. The proposed multi-modal systems are based on flexibility to fusing and interpreting data from various sources in multiple data sources as shown in Fig. 5 .

The implementation of multi-modal smart education applications for monitoring the students' level and their progress in multiple exams, assignments, or delivery reports in the two dimensions parents/them or lecturers. These applications can reduce the effect of digital inequality in remote teaching in emergency situations, it is preferable to use an open-source that allows the creation of responsive interactive digital content. Another target of these applications achieve justice in evaluating the students from multiple deliveries in multiple data types such as video format, online broadcast, or text reports as shown in Fig. 6 . 

CoronaVirus makes smart education becomes a hot trend in the implementation these applications whether for faculties or schools. This paper presents the effect of coronavirus on education and the importance of using Internet-of-things in automated remote education systems. It presents the discussion of smart education n and the importance of uses it. It also shows the obstacles and open research challenges in the multimodal fusion of smart education. The objective of smart education is to improve learner's quality of lifelong learning. It focuses on contextual, personalized and seamless learning to promote learners' intelligence emerging and facilitate their problem-solving ability in smart environments.

Smart education: a review and future research directions

The research trends and the effectiveness of smart learning

A survey on role of ınternet of things in education

Affective computing in education: a systematic review and future research

Standards for smart education-towards a development framework

The Ontology of Next Generation Smart Classrooms

Smart education: ıntroducing active learning engineering classrooms in the Middle East

A research framework of smart education

An application framework for smart education system based on mobile and cloud systems

An IoT-based solution for monitoring a fleet of educational buildings focusing on energy efficiency

Conceptual framework for smart university

Detecting Depression Symptoms in Higher Education Students Using Multimodal Social Media Data

Multimodal learning analytics and education data mining: using computational technologies to measure complex learning tasks

A linguistically-informed fusion approach for multimodal depression detection

Multimodal Intelligence: Representation Learning, Information Fusion, and Applications. Arxiv

Multimodal learning analytics and education data mining: using computational technologies to measure complex learning tasks

Multimodal deep representation learning for video classification

Multimodal fusion for multimedia analysis: a survey

Deep residual learning for image recognition

Towards multimodal sentiment analysis: harvesting opinions from the Web. Int

Affective multimodal human-computer interaction

Automatic video event detection for imbalance data using enhanced ensemble deep learning

Training oscillatory neural networks using natural gradient particle swarm optimization

Convolutional neural networks for sentence classification

Artificial ıntelligence, smart classrooms and online education in the 21st century: ımplications for human development

Design of an online education evaluation system based on multimodal data of learners

CentralNet: a multilayer approach for multimodal fusion