key: cord-0801479-0qhotfi3
authors: Gour, Mahesh; Jain, Sweta
title: Automated COVID-19 Detection from X-ray and CT Images with Stacked Ensemble Convolutional Neural Network
date: 2021-12-09
journal: Biocybern Biomed Eng
DOI: 10.1016/j.bbe.2021.12.001
sha: 6256b3871bc64f034b358f460c6a6ec3a8a59560
doc_id: 801479
cord_uid: 0qhotfi3

Automatic and rapid screening of COVID-19 from the radiological (X-ray or CT scan) images has become an urgent ion of SARS-CoV-2 worldwide. However, accurate and reliable screening of patients is challenging due to the discrepancy between the radiological images of COVID-19 and other viral pneumonia. So, in this paper, we design a new stacked convolutional neural network model for the automatic diagnosis of COVID-19 disease from the chest X-ray and CT images. In the proposed approach, different sub-models have been obtained from the VGG19 and the Xception models during the training. Thereafter, obtained sub-models are stacked together using softmax classifier. The proposed stacked CNN model combines the discriminating power of the different CNN’s sub-models and detects COVID-19 from the radiological images. In addition, we collect CT images to build a CT image dataset and also generate an X-ray images dataset by combining X-ray images from the three publicly available data repositories. The proposed stacked CNN model achieves a sensitivity of 97.62% for the multi-class classification of X-ray images into COVID-19, Normal and Pneumonia Classes and 98.31% sensitivity for binary classification of CT images into COVID-19 and no-Finding classes. Our proposed approach shows superiority over the existing methods for the detection of the COVID-19 cases from the X-ray radiological images.

The novel coronavirus disease 2019 (COVID-19) pandemic has put the livelihoods and health of the massive population in a critical position. It has led to a disturbance in the public life of the world population. The severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) belongs to the family of coron- 5 avirus, which gets transmitted to the people based on the infection in the form of direct contact or fomites. The primary symptoms of coronavirus infection are fever, cough, and fatigue. In several cases, coronavirus causes severe respiratory problems like Pneumonia, lung disorders, and kidney malfunction. The virus has serious consequences as its serial interval is 5 to 7.5 days, and the reproduc-10 tion rate is 2 to 3 [1] people. The coronavirus infection can incite SARS (Severe Acute Respiratory Syndrome), which might unfold serious health impacts. This pandemic has brought new challenges to the medical world. People are not getting wards, ventilators, and even there is a shortage of doctors and nurses in hospitals. It also affected the diagnosis and treatment of noncommunicable 15 diseases [2] . A critical step to fight against the COVID-19 is to identify the infected people so that they get immediate treatment and isolate them to control the further spread of the infection.

The COVID-19 panic has increased due to the unavailability of fast and accurate diagnosis systems to test the infected people. According to the World 20 Health Organization, the diagnosis of COVID-19 cases must be confirmed by the reverse transcription-polymerase chain reaction (RT-PCR) [3] . While RT-PCR has become a standard tool for confirmation of COVID-19, but it is a very time-consuming, laborious, and manual process, and there is a limitation of availability of diagnostic kits and sample collection. The availability of COVID-25 19 testing kits is limited as compared to the increasing amount of infected people; hence there is a need to rely on different diagnosis methodologies.

The coronavirus targets the epithelial cells that affect patient's respiratory tract, which can be analyzed by the radiological images of a patient's lungs.

Some early studies also show that patients present anomalies in chest X-ray and 30 CT scan images, which are the typical characteristics of COVID-19 infected patients [4, 5, 6] . Hence, the development of the computer-aided diagnosis system for the automatic analysis of radiological images (CT scan or X-ray) can be very helpful in identifying infected patients at a faster rate [7] .

Recently, deep learning-based computer aided diagnosis (CAD) systems have 35 shown great success in the automated detection of COVID-19 disease using chest X-ray images. Wang et al. [8] have proposed a COVID-Net model based on the projection-expansion-projection design pattern for COVID-19 cases detection from the X-ray images. Ucar et al. [9] proposed a SqueezeNet CNN model with Bayesian optimization for the classification of chest X-ray images into normal, 40 pneumonia and COVID-19 classes. Jain et al. [10] have applied pre-trained deep networks in two stages. In the first stage, the ResNet50 model is used to classify the X-ray images into viral-induced pneumonia, bacterial-induced pneumonia, and normal cases. Further, in the second stage, they have detected COVID-19 cases from positive viral-induced pneumonia cases. Similarly, Apostolopoulos et 45 al. [11] have evaluated the performances of state-of-the-art pre-trained CNNs on the chest X-ray images for COVID-19 detection, and they have achieved the best performance with VGG19 and MobileNet-v2 models. Joshi et al. [12] have used YOLO-v3 based-architecture to detect the COVID-19 from the X-ray images, in which they have used DarkNet-53 as a backbone network. They have 50 developed their method for X-ray image classification in binary class as well as multi-classes.

In this study, a new stacked convolutional neural network has been designed for the automatic diagnosis of COVID-19 disease from the chest X-ray and CT scan images. This work is The contribution of this work are as following:

• The proposed stacked generalization approach hypothesises that different CNN sub-models learn different non-linear discriminative features and different levels of semantic image representation. Thus, a new powerful model can be developed by incorporating the best prediction from the different CNN sub-models. Therefore, in the proposed method, different 60 sub-models are ensemble together by using Softmax classifier to build a reliable and accurate model for COVID-19 detection.

• We fine-tune VGG19 and Xception models on the X-ray and CT images.

Thereafter, different sub-models have been obtained from the Xception and VGG19 models during the training to develop a stacked ensemble 65 model.

• We investigate the performance of six pre-trained CNN models for the detection of COVID-19 from the chest X-ray images.

• We also investigate and compare the performances of various classifiers to build a stacked ensemble model with different classifiers.

• We collect CT images of COVID-19 patients to build a CT images dataset and also generate a dataset of chest X-ray images with the combination and modification of three publicly available datasets [13, 14, 15] .

The organization of this paper as follows: Section 2 presents the related work. Section 3 describes the proposed stacked CNN model. Section 4 describes the 75 COVID19CXr and COVID19CTs datasets, details the experimental results and performance comparison. Finally, the conclusion is drawn in Section 5.

Over the past 40 years, many computer-aided systems have been developed for the diagnosis of lung diseases [16] , and these systems have shown promising 80 results for automatic detecting lung abnormality from the radiological images [17, 18] Similarly, Narin et al. [28] have applied ResNet50, Inception-v3 and Inception-ResNetV2 using transfer learning for classification of the X-ray images into Nor-120 mal and COVID-19 class. This method has achieved good performance with an accuracy of 98% for ResNet50. However, the number of X-ray images is only 100, which is very less for developing deep learning models. Oh et al. [29] have proposed a patch-based approach to train the ResNet18 model using image patches that have been extracted from the chest X-ray images. For decision 125 making, they used the majority voting strategy, which resulted in an accuracy of 88.9%. An objected detection based DarkCovidNet model has been proposed by Ozturk et al. [30] for automatic detection of COVID-19 cases from the X-ray images. They have reported an accuracy of 98.08% for binary classification of X-ray images into COVID-19 and no-findings. For the multi-class classification identification in the X-ray images. Sethy et al. [32] extracted deep features of X-ray images from the pre-trained CNN, and support vector machine (SVM) has been applied to the extracted feature to classify X-ray images. The authors achieved an accuracy of 95.38% using ResNet50 with the SVM classifier. 140 Similarly, Minaee et al. [33] also proposed a method based on transfer learning, in which they have fine-tuned four pre-trained networks on the COVID-19 chest X-ray images. They reported a sensitivity rate of 98%. Castiglioni et al. [34] have proposed ensemble-based model in which they have ensemble ten pretrained CNNs. Their proposed approach achieved a sensitivity of 80%. Abra- 

The Convolution Neural Network is the driving concept of deep learning algorithms in computer vision, which led to outstanding performance in most of the pattern recognition tasks such as image classification [39, 40, 41, 42] , object 165 localization, segmentation, and detection [43, 44, 45] . It has also shown its superiority in the medical image analysis for image classification and segmentation problems [46, 47, 48, 49] , especially in lung-related diseases such as lung nodule detection [50] , pneumonia detection [51] , and pulmonary tuberculosis [52] . CNN automatically learns a low to the high level of useful feature representations and 170 integrates feature extraction and classification stages in a single pipeline, which is trainable in an end-to-end manner without requiring any manual design and expert human intervention.

In this work, we have developed a deep learning-based stacked convolutional neural network for the rapid screening of COVID-19 patients using X-ray images.

This study is a continuation and extension of the considerations presented in the preprint [53] publication from the Internet. The proposed COVID-19 detection method includes three modules, as shown in Figure 1 . In the first module, a pre- (1).

Where t i and p i are the target value and predicted probability respectively, for 200 each class i in C.

In the experiment, the hyper-parameter values are set as follows: learning rate to 0.0001, the batch size to 16, and dropout probability to 0.15. We experimentally find that these are the best suitable values of hyper-parameters for network training. 

The VGG19 is a pre-trained network that is trained on the ImageNet dataset, in the training of Xception model.

Stacked generalization [56] is an ensemble approach in which a new model learns how to incorporate the best predictions of multiple existing models. The proposed approach hypothesized that different CNN's sub-models learn non- Algorithm 1 Sub-model Generation process

Input: X-ray images of the chest Output: sub-models 1: Divide the dataset into a training set, validation set, and test set.

2: Apply data augmentation on the training set. Train(Xception, train img, img label, class weight) 13: if (i == l2) then 14: sub-model#3 = save(Xception) for j = 1 to 5 do The mathematical definition of the softmax classifier is as shown following: Next, to train the softmax classifier H(y i |θ j ) a dataset has been paprared.

We have prepared dataset by providing X-ray images from the validation set to the each of the sub-models, and collected output class scores ( predictions).

In this case, each sub-model j is output three class scores S0 ji , S1 ji , and 

This section presents the details of the dataset, evaluation metrics, experiment results, and performance comparison. 

In order to evaluate the performance of the proposed Stacked CNN model, we have build two datasets. The detailed description of the datasets are discussed in the following section:

We have generated first dataset of X-ray images, with the combination and modification of three publicly available datasets [13, 14, 15] , which is referred to as COVID19CXr. The COVID19CXr dataset includes 3040 chest X-ray images Chest X-ray Dataset Initiative" [13] and 2) "COVID-19 Image Data Collection" [14] . Pneumonia and Normal cases chest X-ray images are included from the "Mendeley data" [15] . Figure 2 shows the sample chest X-ray images of COVID- Table 2 . 

To assess the performance of the proposed method, we have used sensitivity, specificity, accuracy, positive prediction value (PPV), F1-score, G-mean [58] and area under the ROC curve (AUC) as evaluation metrics. The mathematical definition for the evaluation metrics is given below (in Eqn. (4),Eqn. (5), Eqn. Total 65 4645

(6), Eqn. (7), Eqn. (8) , and Eqn. (9) respectively): deals with a multi-class problem; therefore, to get the overall metric score of the method, the mean of each metric is calculated.

In order to evaluate the performance of the proposed stacked CNN, a set of experiments have been conducted. In the first experiment, data augmentation Figure 4 and Figure 5 , respectively. It can be observed from the confusion matrix that the proposed Table 4 . It can be observed from Table 4 that the proposed model achieved a sensitivity of 98.31% and classification accuracy of 98.30% for the CT images classification into COVID-19 and no-Findings classes. Figure 6 shows performance of proposed method in-terms of the confusion matrices and Table 5 shows the performance comparison of the proposed model, and pretrained CNN models, namely ResNet50 [40] , Inception-v3 [59] , Xception [54] , 365 DenseNet-121 [60] , MobileNet [61] , and VGG19 on the COVID19CXr dataset.

It is observed from (SVM) [62] , decision tree (DT) [63] , neural network (NN), and K-Nearest Neighbor (KNN) [64] , have been applied to the output of sub-models and obtained different stacked model. Their performances are represented in Table 7 . As Ucar et al. [9] SqueezeNet CNN Studies in [32] and [28] have just developed their deep learning models on the dataset of a very small size, which consists of 50 and 100 images, respectively.

Other studies in Table 8 have used less than 250 COVID-19 images for develop-395 ing their methods, except the studies in [7, 35, 37] . In this study, a total of 3040 X-ray images have been used to develop the stacked CNN model, including 546 COVID-19 images, which is the relatively larger number of COVID-19 images among most of the studies presented in Table 8 , except the studies in [7, 37] .

We can see in Table 8 that the studies in [10, 23, 28, 32, 33, 34, 35, 36, 38] 400 have evaluated for binary classification task, studies in [7, 8, 9, 29, 31, 37] have evaluated for multi-class classification task and studies in [11, 12, 30] have evaluated for binary as well as multi-class classification tasks. For the binary classification of X-ray images, the method proposed by Joshi et al. [12] has outperformed the existing methods. On the other hand, for the multi-class 405 classification task, the proposed stacked CNN model shows superiority over the existing methods.

Some of the salient features of stacked CNN can be summarized as:

• The proposed method is based on the stacked generalization of CNN's submodels, which minimizes the variance of predictions and reduces general-410 ization error. As a result, stacked CNN yields higher diagnosis accuracy in the both CT and X-ray images.

• The proposed stacked CNN model produces very little false positives (type 1) and false negatives (type 2) error, which confirms that the stacked CNN is reliable for clinical use.

• The proposed model is developed based on less complex networks, which is computationally efficient, and shows its stability on a small dataset.

• The stacked CNN model requires, on average, 0.029 seconds of computation time to detect the disease from an image. Therefore, this model could be utilized for rapid screening of the COVID-19 disease.

In this paper, we introduced a new stacked convolutional neural network for the automatic diagnosis of the COVID19 from the chest X-ray and CT images. In the proposed method, CNN's sub-models have been obtained from the pre-trained Xception and the VGG19 models. The proposed stacked CNN 

Serial interval of novel coronavirus (covid-19) infections

International impact of covid-19 on the diagnosis of heart disease

World Health Organization, Laboratory testing for 2019 novel coronavirus (2019-nCoV) in suspected human cases

Imaging profile of the covid-19 infection: Radiologic findings and literature review

Clinical features of patients infected with 2019 novel coronavirus in wuhan, china

Severity assessment of COVID-19 using CT image features and laboratory indices

Comparing ct scan and chest x-ray 470 imaging for covid-19 diagnosis

Covid-net: A tailored deep convolutional neural network design for detection of covid-19 cases from chest radiography images

Covidiagnosis-net: Deep bayes-squeezenet based diagnosis of the coronavirus disease 2019 (covid-19) from x-ray images

A deep learning approach to detect covid-19 coronavirus with x-ray images

Covid-19: automatic detection from 485 x-ray images utilizing transfer learning with convolutional neural networks

A 490 deep learning-based covid-19 automatic diagnostic framework using chest x-ray images

Figure 1 covid-19 chest x-ray data initiative

Covid-19 image data collection

Labeled Optical Coherence Tomog-500 raphy (OCT) and chest X-Ray images for classification

Computer-aided diagnosis in medical imaging: historical review, current status and future potential

Automatic detection of abnormalities in chest radiographs using local texture analysis

Automatic tuberculosis screening using chest radiographs

Review of artificial intelligence techniques in imaging data acquisition, segmentation, and diagnosis for covid-19

The role of imaging in the detection and management of covid-19: A review

A deep learning system to screen novel coronavirus disease 2019 525 pneumonia, Engineering

Diagnosis of covid-19 using ct scan images and deep learning techniques

Automatic detection of coronavirus disease (covid-19) in x-ray and ct images: A machine learning based approach

Prior-attention residual learning for more discriminative covid-19 screening in ct images

Covid-caps: A capsule network-based framework for identification of covid-19 cases from x-ray images

Mohammadi, Application of deep learning technique to manage covid-19 in routine clinical practice using ct images: Results of 10 convolutional neural networks

Automated detection of covid-19 from ct scan using convolutional neural network

Automatic detection of coronavirus disease 555 (covid-19) using x-ray images and deep convolutional neural networks

Deep learning covid-19 features on cxr using limited training data sets

Automated detection of covid-19 cases using deep neural networks with x-ray images

Covid-19 identification in chest x-ray images on flat and hierarchical classification scenarios

Detection of coron-570 avirus disease (covid-19) based on deep features and support vector machine

Deepcovid: Predicting covid-19 from chest x-ray images using deep transfer learning

Artificial intelligence applied on chest x-ray can aid in the diagnosis of covid-19 infection: a first experience from lombardy, italy, medRxiv (2020)

Computer-aided detection of covid-19 from xray images using multi-cnn and bayesnet classifier

Application of deep learning for fast detection of covid-19 in x-rays using ncovnet

Covid-19: 590 Automatic detection from x-ray images by utilizing deep learning methods

Ensemble-based bag of features for auto-595 mated classification of normal and covid-19 cxr images

Imagenet classification with deep convolutional neural networks

Deep residual learning for image recognition

Very deep convolutional networks for large-605 scale image recognition

Tomato plant disease detection using transfer learning with c-gan synthetic images

Rich feature hierarchies for accurate object detection and semantic segmentation

2015 IEEE International Conference on Computer Vision (ICCV)

Deeprnnetseg: Deep residual neural network for nuclei segmentation on breast cancer histopathological images

U-net: Convolutional networks for biomedical image segmentation

Residual learning based cnn for breast cancer histopathological image classification

Medical image analysis using convolutional neural networks: a review

Deep learning in medical image analysis

Lung nodule detection in ct using 3d convolutional neural networks

Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning

Tx-cnn: Detecting tuberculosis in chest x-ray images using convolutional neural network

Stacked convolutional neural network for diagnosis 650 of covid-19 disease from x-ray images

Xception: Deep learning with depthwise separable convolutions

Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude, COURSERA: Neural networks for machine learning

Stacked generalization

Stacked regressions

Addressing the curse of imbalanced training sets: one-sided selection

Rethinking the 665 inception architecture for computer vision

Densely connected convolutional networks

Mobilenets: Efficient convolutional neural networks for mobile vision applications

Support-vector networks

A survey of decision tree classifier methodology