key: cord-0697915-nyx6tsay
authors: Roy, Pradeep Kumar; Kumar, Abhinav
title: Internet of Medical Things for early prediction of COVID-19 using ensemble transfer learning()
date: 2022-04-28
journal: Comput Electr Eng
DOI: 10.1016/j.compeleceng.2022.108018
sha: f0115d2042e3022895e166333077963922b542b5
doc_id: 697915
cord_uid: nyx6tsay

In the wake of the COVID-19 outbreak, automated disease detection has become a crucial part of medical science given the infectious nature of the coronavirus. This research aims to introduce a deep ensemble framework of transfer learning models for early prediction of COVID-19 from the respective chest X-ray images of the patients. The dataset used in this research was taken from the Kaggle repository having two classes- COVID-19 Positive and COVID-19 Negative. The proposed model achieved high accuracy on the test sample with minimum false positive prediction. It can assist doctors and technicians with early detection of COVID-19 infection. The patient’s health can further be monitored remotely with the help of connected devices with the Internet, which may be termed as the Internet of Medical Things (IoMT). The proposed IoMT-based solution for the automatic detection of COVID-19 can be a significant step toward fighting the pandemic.

The Internet of Medical Things (IoMT) is the collection of medical devices and applications that connect to healthcare IT systems through online computer networks [1] . An IoMT-based solution for the automatic detection of COVID-19 is urgently needed given the infectious nature of the disease. If a system is developed that can detects the infected person remotely, then it will be helpful to minimize the spread rate of the coronavirus. The COVID-19 pandemic continues to have a devastating effect on the health and well-being of the global population and has put the world under lockdown in 2020 [2] . The population of China was The Reverse Transcription Polymerase Chain Reaction (RT-PCR) test is a typical real-time COVID-19 test used to assess the existence of antibodies against the virus. Furthermore, molecular testing of respiratory samples is recommended for diagnosing and laboratory confirmation of COVID-19 infection. However, it takes a long time and is prone to producing false-negative results. In the meantime, several developing countries cannot perform large-scale COVID-19 tests due to the high expense where the immediate diagnosis is based on the appearance of symptoms.

Early detection is critical for controlling and preventing COVID-19. If COVID-19 disease is diagnosed early, the disease's prevalence and the number of people who get infected further will be reduced. However, at the same time, disease detection has been slowed by a shortage of detective resources, and shortcomings in medical equipment and experts [2] . In consequence, the number of patients and casualties rises. One of the most common symptoms of COVID-19 is difficulty in breathing, which can be diagnosed using chest X-Ray (CXR) imaging. Opacity related findings in the CXRs have been associated with COVID-19 [3] . This important feature may be useful in developing a Deep Learning model for screening large amounts of radiographic images for COVID-19 suspect cases. The IoMT plays an important role, where the patient health information will be monitored remotely and reduce the physical load of the medical hospitals.

Deep Neural Networks, especially the Convolutional Neural Network (CNN), have achieved state-of-theart accuracy in a variety of domains over the last decade, ranging from computer vision to text classification [4] . Deep Learning (DL) have been used for the diagnosis of various diseases ranging from Tumor analysis [5] , Cancer classification, Diabetic Retinopathy etc. This study looks into the problem of implementing a Transfer Learning models (a DL based approach) for COVID-19 predictions. Transfer learning architecture has been shown to not only produce state-of-the-art results but also to outperform in many computer vision tasks [6] . The objective of this study is to assess the efficacy of state-of-the-art pre-trained CNN Many frameworks are reported recently with the individual transfer learning models. However, they suffer from false-positive prediction issues. The false-positive prediction means even though a person is not infected from the COVID-19, the model predicts COVID Positive, which might not be acceptable as it causes frustration. To minimize the false positive predictions and fill the research gap, this research aims to utilize a large number of COVID-19 Positive and COVID-19 Negative samples to build the model. This research used an ensemble learning framework that combined the outcomes of seven transfer learning models. The ensemble model leads to high prediction accuracy, and the false-positive prediction was zero. The proposed ensemble system combines a diverse group of learners to improve the model's consistency and predictive capacity. The ensemble approach resolves the bias-variance trade-off and makes the model more robust to a new dataset. The model can be used at the backend of IoT and smart devices to improve existing testing methodologies. If the proposed ensemble model is used in association with smart devices, it will function as follows: The desktop manager can display the digital X-ray image captured by the machine.

The proposed model will be loaded and executed on the desktop manager's onboard processing chip. In the IoMT environment, the model should predict the image as either COVID-19 Positive or COVID-19 Negative and notify that as its final diagnosis. The major contributions are as follows:

• An ensemble of transfer learning algorithms is suggested for reliable COVID-19 patient prediction with a low false-positive rate.

• The performance of the proposed ensemble model is compared with existing state-of-the-art transfer learning models. • The proposed ensemble framework achieved high prediction accuracy; hence it may be implemented in smart devices for early COVID-19 patient prediction.

The rest of the article is organized as follows: Section 2 discusses the existing research. Section 3 discusses the working of the proposed ensemble framework and highlights the transfer learning models. The experimental outcomes of the proposed methodology are explained in Section 4. Finally, Section 5 concludes the work.

This section discusses an extensive literature survey about the different types of deep learning models that have been proposed to detect COVID-19. Rahimzadeh and Attar [7] used a concatenated framework of transfer learning models to detect COVID-19 from CXR images. They used a dataset that contained images belonging to three classes: (i) COVID-19 -180 images, (ii) Pneumonia -6054 images and (iii) Normal -8851 images which were collected from GitHub 4 and Kaggle 5 respectively in March-April 2020.

Apostolopoulos and Mpesiana [8] used transfer learning models on CXR scans to detect COVID-19.

The data set was divided into two sections to evaluate the model: (i) bacterial pneumonia and (ii) viral 4 https://github.com/ieee8023/covid-chestxray-dataset 5 (https://www.kaggle.com/c/rsna-pneumonia-detection-challenge J o u r n a l P r e -p r o o f Journal Pre-proof pneumonia. The images were rescaled for optimal feature extraction. A total of 1428 X-ray images were examined, with 224 images from COVID19 disease, 700 images of common bacterial pneumonia, and 504 images of normal conditions. Models are analyzed using two types of classification accuracies: (i) threeclass (covid, pneumonia, and normal) and (ii) two-class (covid and normal). VGG-19 and MobileNetV2 are found to outperform other models in classification, with an accuracy of 98.75% for the two-class classifier, and 97.40% for the three-class classifier.

Loey et al. [9] introduced a novel system that uses deep transfer learning models combined with data augmentation and Conditional Generative Adversarial Network (CGAN) on 742 CT images (divided into COVID-19, Non-COVID-19). They gathered pre-prints from bioRxiv1 6 and medRxiv2 7 that recorded COVID-19 CT patient cases from January 19 to March 25, 2020. The dataset was divided into three categories (i) train, (ii) validation, and (iii) test. ResNet50 was found to be the best classifier for detecting COVID-19 in the CT dataset by achieving the accuracy, sensitivity and specificity values as 81.38%, 88.85% and 81.90%, respectively. Taresh et al. [10] did a comparative analysis of transfer learning models on CXR images to determine the efficacy of pre-trained CNN in the automated diagnosis of COVID-19. VGG-16 was superior to other models, with the highest accuracy and F1-score values of 98.72%, and 97.59%, respectively.

Fan et al. [11] also did an analysis of transfer learning models on CXR images for binary classification of COVID-19. Two datasets were used to validate the transfer learning models' performance on X-ray images.

The first dataset 8 contained 74 "normal" and 74 "pneumonia" images for training taken from GitHub. 20 "normal" and 20 "pneumonia" images were used to test the integrity of models. For the second dataset, the same number of images were used, with "normal" scans 9 and infected "pneumonia" scans 10 . The models were built on MATLAB using a 10-fold procedure and trained on two separate datasets for 10 epochs with 140 iterations. According to the performance assessment metrics, the MobileNetv2 and Xception models could accurately diagnose COVID-19 from CXR images with 95% and 96% accuracy, respectively.

Horry et al. [12] used various models for COVID-19 classification on three imaging modes (i) X-Ray, (ii) Ultrasound and (iii) CT Scan. The dataset was collected from open source repositories and analysed on 11

May 2020. It was found that VGG-19 architecture outperformed other models on ultrasound images and achieved an F1-score value of 0.99. Civit-Masot et al. [13] used deep learning to classify the radiographic images into COVID-19, Healthy, and Pneumonia. Their model achieved an AUC value greater than 90%. Azemin et al. [15] used a transfer learning model for COVID-19 prediction on the data collected from source 11 . Their binary classification model achieved an accuracy of 71.90%. Narayan et al. [16] also used the transfer learning approach for COVID-19 prediction. The dataset was collected from 6 different publicly available sources with 2504 COVID-19 images and 6807 Non-COVID-19 images downloaded on 16 May 2020.

ResNet50 outperforms other Transfer Learning models such as Inception-v3, DenseNet201, and Xception.

Using 10-fold cross-validation, it was observed that ResNet50 had an accuracy (%) of 99.34 ± 0.35.

Wang et al. [17] introduced COVID-Net, which is a deep CNN that detects COVID-19 from CXR images. The following hyperparameters were used for training: learning rate=2e-4, number of epochs=22, batch size=64, factor=0.7, patience=5. COVID-Net achieves good accuracy by achieving 93.3% test accuracy with a sensitivity of 91% to detect COVID-19.

Jain et al. [18] used pre-trained transfer learning models to predict the class probabilities and classify the radiographic images as belonging to COVID-19, Normal or Pneumonia. There were 6432 total CXR images in the database 12 , which were divided into a training set (5467) and a validation set (965). XceptionNet had the best performance among all the discussed models with an overall accuracy of 97%. Pham [19] did a study to look into the potential of parameter adjustments in the transfer learning of three popular pre-trained CNNs: AlexNet, GoogLeNet, and SqueezeNet, which are known to have the shortest prediction and training iteration times among pre-trained CNNs.

Islam et al. [20] proposed a combined CNN-LSTM model that detects COVID-19 from CXR images.

CNN acted as a feature extractor in the model while LSTM for classification. The COVID-19 images were classified with good sensitivity, specificity, and F1-score of 99.3%, 99.2%, and 98.9%, respectively. Many research works have been proposed since the COVID-19 began. However, due to less number of training samples or high false positive prediction, the research continues to boost it. Some limitations of the existing 11 https://github.com/ieee8023/covid-chestxray-dataset and shorturl.at/uxBPR 12 https://www.kaggle.com/prashant268/chest-xray-covid19-pneumonia J o u r n a l P r e -p r o o f Journal Pre-proof 

Limitations Rahimzadeh and Attar [7] The high performance may be a result of over-fitting because of the less dataset available for training and testing. Apostolopoulos and Mpesiana [8] They used a small dataset for model, the medical community may determine the likelihood of integrating X-rays into disease diagnosis after analysing and checking the findings with experts. Taresh et al. [10] Due to the class imbalance of data, the problem remained to identify which of the classifiers would perform better in confirming COVID-19 cases. Hence an optimal set of hyper parameters along with an increased dataset is required for better generalizability of the network. Fan et al. [11] Due to low availability of the dataset, the model should not be used for diagnosis and generalizability without consulting with the concerned medical authorities. Ko et al. [14] The research dataset was derived from the same sources as the training dataset, potentially raising generalizability and over fitting concerns. Azemin et al. [15] They built the model with a small number of publicly accessible COVID-19 CXR images. Narayan et al. [16] Their work was undervalues the significance of data augmentation techniques, such as Generative Adversarial Networks (GANs), which generate more training images synthetically even when the COVID-19 dataset is insufficient to train a Deep Learning model from scratch. Wang et al. [17] One of the most significant bottlenecks is the need for expert radiologists to interpret radiography images. As a result, radiologists are in desperate need of computer-assisted diagnostic systems. Jain et al. [18] The high accuracy obtained could be cause for concern because it could be due to over fitting. Thus, the model needs to be validated on a large scale public dataset and consulted with the medical fraternity. Pham [19] The study did not recognize COVID-19 sub-classification into mild, moderate, or extreme disease due to restricted data labeling. Another problem was that each patient only received a single CXR sequence. Because of this data constraint, it's impossible to tell whether patients developed radiographic findings as their illness progressed. Islam et al. [20] Due to the limited size of the dataset, the network's generalizability must be enhanced. Only the posterior-anterior X-Ray view were functional; lateral views and anterior-posterior views were not. The model does not classify COVID-19 disease types (mild, severe), which could be improved with a larger dataset.

research are highlighted in Table 1 . This research aims to address the existing issues and fill the research gap.

This research aims to build a system that predicts COVID-19 infected people in the early stage with The images corresponding to COVID-19 Positive class and COVID-19 Negative class had been converted to .npy files along with their appropriate labels. The .npy file format is NumPy's basic binary file format for storing a single NumPy array on a disk. This way, the shape and the data type information necessary to create the array on a system with different architecture remains intact. This process leads to faster processing of data. All of the images from the source were 299 × 299 pixels in nature, which had been converted to appropriate sizes to suit the respective transfer learning architecture. The code structure appends the data, the images according to their respective classes, the labels and stores it in a standard binary format, .npy file. Threading was used for the parallel execution and utilized the multiprocessing capacity to its optimal usage. A total of 2000 chest X-Ray images were used for the experiment. The training, validation, and testing sets have been split into 60%, 20%, 20% ratios, respectively, for optimal model performance. The detailed break-down of the number of images in the data set has been shown in Table 2 . ResNet50 is a model having 48 Convolution layers along with 1 MaxPool and 1 Average Pool layer [21] .

The general intuition is that with an increasing number of convolutional layers, the performance of the model will improve. However, it has been observed that the model's actual performance degrades in doing so because of over-fitting and vanishing gradient problems. ResNet address this problem by introducing skip connections before the ReLu activation function is applied. Skip Connections address the problem of vanishing gradient by allowing the alternate shortcut path for the gradient to flow through. They also allow the model to learn an identity function that guarantees that the higher layer performs at least and the lower layer, if not better. The input size of the images accepted by the ResNet50 model is 224 × 224.

The ResNet50 model is divided into five stages, each with its own convolution and identity block. Each convolution block has three layers, and each identity block also has three layers. Finally, the network has an Average Pooling layer followed by a fully connected (FC) layer having 1000 neurons. ResNet50V2 applies Batch Normalization and ReLU activation to the input before convolution operation. The second nonlinearity in ResNet50V2 is primarily focused on making it an identity mapping. The contribution of the addition operation between the identity mapping and the residual mapping should be transferred directly to the next block for processing.

To solve the vanishing gradient problem that occurs with the increase in depth of the CNN, DenseNet (Dense Convolutional Network) was introduced [22] . The network is connected in such a way that each layer gets input from the preceding layers and passes the feature maps of its own to the subsequent layers. For with the feature maps, information that needs to be preserved is also sent. The DenseNet architecture distinguishes between information that is introduced to the network and data that is preserved. They add only a small set of feature maps to the network and keep the remaining unaltered. The classifier makes the decision depending upon the feature maps in the network.

InceptionNet is important because, before it, the models would stack layers in the network to get better performance [23] . 

The purpose of the loss function in deep learning models is to measure the overall performance. The Negative only. Hence, a binary crossentropy function is used. However, for the multiclass classification problem, Keras supported another loss function called Categorical crossentropy. The crossentropy-based loss function is suitable for the inputs with larger boundaries than other loss functions. In parallel, it helps the model to find the saturation point quickly.

Mathematically, it can be defined as follows: Suppose for the inputs (x 1 , x 2 , x 3 , . . . , x n ), the output is 13 https://keras.io/api/losses/

Journal Pre-proof (y 1 , y 2 , y 1 , . . . , y 1 ) respectively. Means, x 1 having the actual output as y 1 and their output predicted by the model isŷ, whereŷ for i th input is calculated using the equation 1:

Here,ŷ i is the predicted output for i th input, w T i is the corresponding weights and b i is the bias values, f is a non-linear activation function called Rectified Linear Unit(ReLU) which helps to convert the negative value to zero and positive values remain unchanged. The ReLU activation mathematically defined in Eq. 2.

Finally, the loss function ϕ is for the binary class (i.e., C=2) is defined in Eq. 3.

The cost function for k training samples is calculated using the Eq. 4.

The cost function helps to updated the associated weights and biases of the network.

The ensemble approach involves combining the predictive power of various learners to improve the overall performance and robustness of the model. 

The probability value of each classes for the same i th input (x i ) with model M 2 can be expressed as: 

The final prediction of the input will be decided by finding the maximum values from the predicted values, i.e., max(o 1 , o 2 , o 3 , . . . , o m ). The predicted class will be compared with the actual label of the input to find the error and accuracy value.

As the performance of deep learning-based models is very sensitive to the chosen hyper-parameters, extensive experiments were performed to choose the best-suited hyper-parameters. We varied the learning rate, batch size, epochs to find the best-suited hyper-parameters. The best-suited hyper-parameters of the proposed models are listed in Table 3 . To prevent RAM overhead and for optimal utilization of resources, a callback function was explicitly defined with a patience value of 10 to keep a check on the validation loss and cut off the training when the validation loss did not improve over 10 epochs.

Journal Pre-proof 

This section describes the experimental simulations and results. It consists of a description and testing results of the classification model used to classify chest X-Ray images. It also discusses the performance of the model on unforeseen data, that is, on validation dataset. The Precision, recall, F1-score, accuracy, and AUC-ROC are the main metrics used to evaluate the performance. These are described as follows (Eq. 8 to 11):

Precision = T P T P + FP (10) Negative). The AUC-ROC metric is also used to evaluate the performance of classifiers at various threshold settings. The ROC represents the probability curve, and the AUC represents the degree of separability measure. The higher the AUC, the more accurate the model is in forecasting [26] .

The experiments were started with the selected transfer learning models on the preprocessed dataset. Table 8 . The proposed ensemble learning model achieved the weighted precision, recall and F1-score of 1.00, 1.00, 1.00, respectively, which indicates that the misclassification rate is almost zero. The confusion matrix and ROC curve for the proposed ensemble model can be seen in Figure 7 and Figure 8 . The Accuracy vs Epochs and Loss vs Epochs plots can be seen in Figure 9 (a) and Figure 9 (b), respectively. were performed using chest X-Ray (CXR) data [8, 10, 11, 12, 13, 15, 16] , whereas few researchers used CT scan of the infected people to build the predictive model [9, 14] . Among the listed research of execute a testing sample on all the implemented transfer learning models can be seen in Table 9 . We found that the MobileNetV2 took lesser time to execute on an average among all implemented models, whereas the proposed ensemble model took more execution time slightly but performed significantly well.

This study suggested a framework that detected the COVID-19 infected people with high accuracy. The same can be implemented into any Internet-enabled device with a monitoring facility to connect with medical experts. The COVID-19 disease is infectious. Hence remote monitoring with IoMT based technology is the best solution to control it. As the cases have been increasing at an exponential rate, there is a need to 

The authors do not have any conflict of interest

A novel cryptosystem using dna sequencing and contextual array splicing system for medical internet of things

Covid-19: A comprehensive review of learning models

Classification of covid-19 in x-ray images with genetic fine-tuning

Multilayer convolutional neural network to filter low quality content from quora

Early tumor diagnosis in brain mr images via deep convolutional neural network model

Transfer learning in computer vision tasks: Remember where you come from

A new modified deep convolutional neural network for detecting covid-19 from x-ray images

Covid-19: automatic detection from x-ray images utilizing transfer learning with convolutional neural networks

A deep transfer learning model with classical data augmentation and cgan to detect covid-19 from chest ct radiography digital images

Transfer learning to detect covid-19 automatically from x-ray images

Exploiting multiple optimizers with transfer learning techniques for the identification of covid-19 patients

Covid-19 detection through transfer learning using multimodal imaging data

Deep learning system for covid-19 diagnosis aid using x-ray pulmonary images

Covid-19 pneumonia diagnosis using a simple 2d deep learning framework with a single chest ct image: model development and validation

Covid-19 deep learning prediction model using publicly available radiologist-adjudicated chest x-ray images as training data: preliminary findings

Transfer-to-transfer learning approach for computer aided detection of covid-19 in chest radiographs

Covid-net: A tailored deep convolutional neural network design for detection of covid-19 cases from chest x-ray images

Deep learning based detection and analysis of covid-19 on chest x-ray images

Classification of covid-19 chest x-rays with deep learning: new models or fine tuning

A combined deep cnn-lstm network for the detection of novel coronavirus (covid-19) using x-ray images

Deep residual learning for image recognition

Densely connected convolutional networks

Rethinking the inception architecture for computer vision

Xception: Deep learning with depthwise separable convolutions

Proceedings of the IEEE conference on computer vision and pattern recognition