key: cord-1034808-mmpdkc0v authors: Sharma, Anubhav; Singh, Karamjeet; Koundal, Deepika title: A Novel Fusion based Convolutional Neural Network Approach for Classification of COVID-19 from Chest X-ray Images date: 2022-05-02 journal: Biomed Signal Process Control DOI: 10.1016/j.bspc.2022.103778 sha: 2510bc5c2b506dda7f91dd6a87efb994ee6f5e63 doc_id: 1034808 cord_uid: mmpdkc0v Coronavirus disease is a viral infection caused by a novel coronavirus (CoV) which was first identified in the city of Wuhan, China somewhere in the early December 2019. It affects the human respiratory system by causing respiratory infections with symptoms (mild to severe) like fever, cough, and weakness but can further lead to other serious diseases and has resulted in millions of deaths until now. Therefore, an accurate diagnosis for such types of diseases is highly needful for the current healthcare system. In this paper, a state of the art deep learning method is described. We propose COVDC-Net, a Deep Convolutional Network-based classification method which is capable of identifying SARS-CoV-2 infected amongst healthy and/or pneumonia patients from their chest X-ray images. The proposed method uses two modified pre-trained models (on ImageNet) namely MobileNetV2 and VGG16 without their classifier layers and fuses the two models using the Confidence fusion method to achieve better classification accuracy on the two currently publicly available datasets. It is observed through exhaustive experiments that the proposed method achieved an overall classification accuracy of 96.48% for 3-class (COVID-19, Normal and Pneumonia) classification tasks. For 4-class classification (COVID-19, Normal, Pneumonia Viral, and Pneumonia Bacterial) COVDC-Net method delivered 90.22% accuracy. The experimental results demonstrate that the proposed COVDC-Net method has shown better overall classification accuracy as compared to the existing deep learning methods proposed for the same task in the current COVID-19 pandemic. In December 2019, a novel type of coronavirus (CoV) namely SARS-CoV-2 or what is commonly referred to as COVID-19, was identified amidst a widespread pneumonia outbreak in local hospitals of Wuhan, Hubei province in China. COVID-19 is considered to be of natural animal origins that started in the bat population and later broke the species barrier to infect humans, who became the carriers for the virus. The virus is responsible for respiratory infections leading to symptoms like fever (mild/severe), cough, dyspnea, bilateral infiltrates in chest imaging and other acute respiratory problems [1] . This virus has spread throughout the globe and has caused around 6 million deaths as of now since the start of the COVID-19 pandemic [2] in December, 2019. This calls for new collaborations to accelerate the research and development of coronavirus vaccine and reliable, quick testing methods all across the world [3] . The World Health Organization (WHO) plays a critical role [4] in the on-going pandemic as many countries face shortages of testing kits, trained healthcare professionals, contact tracing protocols among other essentials and medical equipment. The new medical policies are designed for COVID-19 pandemic [5] which explore the medical facilities provided to CoV patients. Many researchers have contributed to reduce the inefficiency in the current diagnosis of coronavirus. For further improving the diagnosis and better understanding of severe pediatric patients of COVID-19, authors [6] [7] provided the epidemiological, laboratory findings, imaging data and clinical treatments to all healthcare professionals of Wuhan city, China. Jiang et al. [8] provided the necessary clinical study about the symptoms, complications and treatments required by the COVID infected patients. Later on, Tolksdorf et al. [9] introduced the concept of using syndromic surveillance data to assess COVID-19 patients analogous to influenza pneumonia patients for determining the condition of CoV affected patients. In [10] , authors throw light towards the classification of pneumonia on the basis of chest radiology, rapid analysis of biological samples, severity scores and collection of other relevant clinical data. Wang et al. [11] investigated the spread of CoV through the respiratory tract and later on, Bernheim et al. [12] concluded that the frequency of CT findings was related to the infection time course. From all these studies, it can be observed that the CT and X-ray imaging system are two effective tools for preliminary, quick identification and quantification of positive CoV cases. Therefore, a large number of researchers are working towards developing a quick, reliable and effective detection system for CoV disease from chest X-ray images by using modern computer science techniques like deep learning. This also has other advantages over traditional testing like easy scalability, safer sample collection, faster diagnosis. Before COVID-19, in the recent times we have seen many applications of technology and computers in the medical image and signal processing [52] [53] [54] [55] [56] [57] [58] [59] which have aided the professionals in better diagnosis certain ailments. Jaiswal et al. [13] proposed a deep learning-based approach to get a better clarity about the presence of pneumonia in chest X-ray images. Later, Chouhan et al. [14] suggested a transfer learning-based approach for the same. The recent progress of deep learning applications in medical image analysis has proven to be an effective tool for diagnosis of various diseases like breast cancer and many other pulmonary disorders [60] [61] [62] . Y-D. Zhang et al. [63] describe MIDCAN, A multiple input deep convolutional attention network that can handle CCT and CXR images simultaneously. They fused CCT and CXR images in order to improve their individual performance. S-H. Wang at el. [64] proposed a rank based average pooling module (NRAPM), which showed remarkable accuracy and precision values for COVID-19 diagnosis. COVID-Net [15] is a convolutional neural network (CNN)-based approach for detection of COVID-19 patients. The main goal is to provide an effective and reliable treatment to CoV patients which is achieved by patch-based convolutional neural network approach [16] . A similar approach [17] explores the different types of features and automatic detection of SARS-CoV-2 from chest X-ray images. Also, Ozturk et al. [18] have applied the existing DarkNet architecture for developing the DarkCovidNet model which achieves an accuracy of 98.08%, 87.02% for binary and multiclass classification tasks, respectively. Similarly, Ucar et al. [19] and Marques et al. [20] fine-tuned the Bayes SqueezeNet and EfficientNet models respectively, for providing automated diagnosis for SARS-CoV-2 patients. Rahimzadeh CoV detection by using laboratory findings rather than X-ray or CT-scan images of CoV patients. Later, Swapnarekha et al. [41] provided a comparative performance analysis of various machine and deep learning-based models for predicting COVID-19 along with their applications. In addition to aforesaid approaches, Karthik et al. [42] developed a channel-shuffled dual-branched convolutional neural network that uniquely learns X-ray patterns from the chest X-ray images. Authors in [50] [51] employed U-net architecture and a meta-heuristic-based feature selection for the diagnosis of COVID-19 using X-ray images. Our proposed methodology is a confidence fusion of VGG16 and MobileNetV2 which are pretrained on ImageNet dataset in order to classify a sample chest X-ray image for 3-class and 4-class classification tasks. Various Models were tested, and the best performing models were selected for the process of fusion, hence creating a model that has the advantages from the unique feature selection process of both the architectures. This makes better predictions and achieves better performance than any single contributing model. Hybrid models also provide robustness and reduce the spread or dispersion of the predictions which might be an issue when using a single model. The main contributions of the proposed method are three-fold and described briefly as follows. • The proposed method tests various proven models like VGG16, VGG19, MobileNetV2, ResNet50, DenseNet to find the best performing fusion and provides a detailed analysis on the prepared dataset of X-ray images for a better classification of COVID-19 samples. • The proposed novel hybrid CNN architecture integrates the two different types of features obtained from separate CNN architectures with help of confidence fusion method with the aim of providing better assistance and diagnosis to COVID-19 patients. As seen in Table 3 , The proposed model can improve upon the performance of the parent models significantly for the evaluated performance metrics. • The method was tested on multiple models and the best fusion model was selected for this study as described in Table 2 . • The proposed work would help in building the foundation for inventing the hybrid CNN architectures which helps the radiologists to correctly identify and distinguish between the different types of pneumonia and COVID infections from chest X-ray images. • The proposed framework can be deployed as one of the useful techniques in the medical field for classification purposes. It also helps radiologists to diagnose and treat diseases at early stages. The remainder of the paper is organized as follows: Section 2 describes concepts of Convolutional Neural Network, confidence fusion method, dataset description, details of building and training of the proposed methodology with parameters and necessary information about the developed deep learning application. Section 3 presents the experimental results and evaluation criteria including accuracy, precision, recall and F-score. Finally, the conclusion is presented in Section 4. A Convolutional Neural Network is a deep learning algorithm that applies convolution, a mathematical operation between input data i.e., the image array and a set of learnable filters (kernels) to create feature maps that are able to extract local features like edges, sharp corners, shapes, gradients, etc. to solve various classification problems. As we move deeper into the network layers more complex feature maps are created. Let the input image image matrix be of dimensions and filters of size. Then the ( ℎ × ) convolution operation on image and filter is given by Equation (1). (1) The dimensions of filters are generally taken as and filter (F) moves towards 3 × 3, 5 × 5, 7 × 7 right with a certain stride (stride value 1) till it reaches the width of the image. For each stride the values of the kernel are multiplied by the original pixel values and summed to obtain a single value. These values are mapped onto an array thus obtaining a feature/activation map. The result is fed to an activation function which produces the output for a given node. respectively, which can be inspected to better understand how and what type of features are detected for a given input image. The architecture of VGG [48] was proposed by Simonyan and Zisserman and achieved 92.7% accuracy which was among the top-5 in ImageNet. The two variants namely VGG16 and VGG19 have been widely used since. The '16' and '19' refer to the count of weighted layers used in network configuration. In specific VGG16 consists of multiple kernels of size and 3 × 3 convolutional layers which enable the network to recognize and understand various complex features and patterns. The network also consists of average-pooling layers and 3 fully (2 × 2) connected layers. All the hidden layers use ReLU as the activation function and Softmax activation is used to achieve the normalized classification vector. In recent times, the VGG architecture is being employed in various transfer learning approaches. MobileNet is a mobile architecture [43] which has considerable low complexity and size owing to the use of depth wise Separable Convolution, which makes it suitable to run on devices with low computational power. MobileNetV2 expanded the feature extraction and introduced an inverted residual structure. The model architecture consists of a convolutional layer followed by a series of residual bottleneck layers. Kernel size for all spatial convolution operations is taken as ReLU6 is used as the non-linearity along with batch normalization and dropout during the training phase. Each bottleneck block consists of 3 layers, starting with a convolutional layer followed by (1 × 1) the aforementioned depthwise convolution layer and finally another (3 × 3) (1 × 1) convolutional layer without ReLU6 activation. MobileNetV2 has wide applications in the current time due to its excellent feature extraction capabilities and small size. Different layers of a convolutional neural network give different output. Outputs from multiple models at different stages can be used to produce a combined classification results which can often provide better results than the parent models. One such way is to fuse the output vectors [49] (categorical probability vectors) from the last Softmax classifier layer. Let be the output for a given CNN model and is represented as where i represents the CNN under consideration and represents the number of classification classes. The term represents the , probability that the given sample is classified as class. Average value for each class is ℎ calculated as in Equation (2). where is the total number of models being fused. Out of classes the final predicted class is given by Equation (3). where To test our proposed method, we collected a total of 4883 images for the 3 classes and is referred to as Dataset-1 (D1) [65] from Kaggle COVID-19 radiology database [66] . Table 1 . The visualization of X-ray images for each class as shown in Fig. 3 . To evaluate the predictive performance, Precision, Recall and F-score of the proposed method, we have used Stratified K-Folds cross-validation technique. It also helps to avoid bias in the ( = 4) In recent time's deep learning algorithms, CNN models in particular have seen numerous breakthroughs. It is extremely expensive to train due to complex data models. Moreover, deep learning requires expensive GPUs. Since computational constraints have relaxed, and better hardware is more readily available these limitations are no longer a barrier which is a big step forward. High performance models which are publicly available can be effectively tweaked and used as a base for classification of COVID-19 cases. We expanded this approach by modifying the architecture of two models namely, MobileNetV2 and VGG16 which are then used for classification based on confidence fusion. These. Exhaustive testing using various combination of models showed that the selected models had the best evaluation metrics. This shown in Table 2 . The VGG16 and MobileNetV2 provided 2 very different architectures and unique feature extraction stages and show extremely promising results when compared with other models like VGG-19, ResNet. Upon fusion these shows quite significant improvement over the results of shown by either of its parent models for each of the evaluators. The VGG architecture is a simple one with convolutional layers stacked on top of each other 3 × 3 with occasional pooling layers for reducing data dimensionality. The MobileNetV2 has an elaborate architecture with a fully convolution layer with 32 filters followed by 19 bottleneck layers [43] . The framework and architecture of our proposed work is presented in Fig. 4 Equation (6) and finally Equation (7) are calculated for each given sample and represents the predicted class for the given sample. This is shown in Fig 5 . where refers to true covid classifications and similarly for other classes. -19 The evaluation was performed using Stratified K-Folds cross-validation technique where 75% of the data was used for training purpose and the remaining 25% was used for testing purpose. Plots of accuracy and loss on the training and validation sets (10%) over training phase for Fold 2 are shown in Fig. 6 Table 3 . Table 5 . This in turn helps validate the robustness of the proposed method. In this study, a deep convolutional network-based method is proposed for the classification of COVID-19 cases using chest X-ray images. Prior to this, various authors [15, 17, 18, 23] put forward their methods for binary and multiclass classification tasks. The proposed COVDC-Net method was tested on two different datasets and achieved an accuracy of 96.48% on 3-classes and 90.22% on 4-classes. This performance is superior in terms of precision, recall, F-score and accuracy as compared to existing deep learning-based approaches represented in Table 6 . The ROC Curves for the used base models and the proposed fusion are shown in Fig. 8 .  The dataset imbalance as observed in many of the works render their findings inaccurate and not suitable for reference for practical implementations. This is observed in Shibly et al. [44] , Wang et al. [15] . Our method uses a relatively balanced dataset which is randomly selected so as to ensure random nature of the sample and is evaluated using Stratified 4fold Cross Validation method, hence providing consistent results.  Considering various previous works, it is observed that many leave out certain metrics or present them vaguely. Hence these should not be used as a benchmark for practical purposes as it might have unanticipated repercussions. This can be seen in Apostolopoulos et al. [17] where performance metrics are not properly presented or missing for all the classes. We have presented the results for each class and for all the folds with meaningful results which are more suited to give a better understanding of how and where the improvements that are being observed. All the performance metrics can be seen in the Table 3 and show class wise results and improvement after the fusion of the parent models clearly.  Most of the existing works focus on 3-class and binary classification [46, 45, 15, 17, 18] tasks. Our study, extends the 3-class classification task to 4-class classification and continues to show significant improvements. It also performs better on comparison with, from where the dataset (D2) is borrowed which is used in the 4-class classification tasks.  The model showed an improvement of 0.69% for 4-class classification task and 2.13% in overall accuracy for 3 class classification task on the dataset D2 [61] over the method proposed by Khan et al. [24] .  It was observed that the proposed method showed the quite significant improvements in accuracy, precision, recall for COVID-19 class (as seen in Table 3 ), which is paramount for this study as miss-classification of a COVID-19 case can have serious repercussions than other classes. Considering the current scenario there is a serious increase in the work density of radiologists. Manual diagnosis doesn't take into consideration the expert's tiredness may affect the error rate. Hence support systems can be a solution which can eliminate this problem and provide a more effective diagnosis. In this study, a deep convolutional network-based method was proposed for efficient classification of the novel coronavirus from healthy and pneumonia infected patients. For which, we used pretrained MobileNetV2 and VGG16 which have a very different architecture and feature selection process. On fusion of the models, after addition of some extra layers and considering several other crucial factors, the proposed method is able to benefit from various differentiating patterns and visual features for each pneumonia class. Therefore, the fusion performance observed was better than the individual performance of either of the models. These encouraging results show that such models can be used to diagnose other chest related diseases like tuberculosis and pneumonia. In real life scenarios, CNN is an effective method for detecting CoV-affected patients considering the sample collection is safer than a nasal swab and is much faster in terms of diagnosis. The proposed model provides a multiclass diagnosis with other pulmonary diseases which is a key factor as many of these might share the surprisingly similar symptoms and effects on the lungs. Other tests may get rendered useless in such cases and hence showing the supremacy of CNN models like COVDC-Net in practical applications in the coming time. Scaling these to existing hospitals and clinics with X-ray machines doesn't require special infrastructure and testing equipment and is quick, which is another benefit of AI-Aided methods for diagnosis. In future, we plan to evaluate the proposed model on other wider sets of pulmonary diseases and make the model more robust. With an even more in depth understanding of the SARS-CoV-2 virus and better data, the models can be improved further. ➢ Evaluation of proposed COVDC-Net method for both 3-class and 4-class classification. ➢ A novel hybrid fusion-based Deep Convolutional Neural Network model is presented. ➢ The proposed model helps radiologists to distinguish between types of pneumonia and COVID infections. ➢ The performance of COVDC-Net method is better than several other methods proposed for detection of COVID-19. COVID-19), Epidemiology, virology, clinical features, diagnosis, and prevention Naming the coronavirus disease (covid-19) and the virus that causes it, WHO, 2020 Coronavirus: Covid-19 Has Killed More People Than SARS and MERS Combined, Despite Lower Case Fatality Rate Statement on the fifth meeting of the International Health Regulations (2005) Emergency Committee regarding the coronavirus disease (COVID-19) pandemic, WHO Clinical features of severe pediatric patients with coronavirus disease 2019 in Wuhan: a single center's observational study Clinical features of patients infected with 2019 novel coronavirus in Wuhan, china Review of the clinical characteristics of coronavirus disease 2019 (covid-19) Influenza associated pneumonia as reference to assess seriousness of coronavirus disease (COVID-19) The definition and classification of pneumonia Detection of SARS-CoV-2 in different types of clinical specimens Chest CT findings in coronavirus disease-19 (COVID-19): relationship to duration of infection Identifying pneumonia in chest X-rays: a deep learning approach A novel transfer learning based approach for pneumonia detection in chest X-ray images COVID-Net: A tailored deep convolutional neural network design for detection of covid-19 cases from chest radiography images Deep learning COVID-19 features on CXR using limited training data sets Covid-19: automatic detection from X-ray images utilizing transfer learning with convolutional neural networks Automated detection of COVID-19 cases using deep neural networks with X-ray images COVIDiagnosis-Net: Deep Bayes-SqueezeNet based diagnosis of the coronavirus disease 2019 (COVID-19) from X-ray images Automated medical diagnosis of COVID-19 through EfficientNet convolutional neural network A modified deep convolutional neural network for detecting COVID-19 and pneumonia from chest X-ray images based on the concatenation of Xception and ResNet50V2 COVIDX-net: a framework of deep learning classifiers to diagnose COVID-19 in X-ray images CVDNet: A Novel Deep Learning Architecture for Detection of Coronavirus (Covid-19) from Chest X-Ray Images CoroNet: A deep neural network for detection and diagnosis of COVID-19 from chest x-ray images Diagnosis and detection of infected tissue of COVID-19 patients based on lung x-ray image using convolutional neural network approaches A Deep Learning System to Screen Novel Coronavirus Disease Deep learning approaches for COVID-19 detection based on chest X-ray images A new approach for classifying coronavirus COVID-19 based on its manifestation on chest X-rays using texture features and neural networks A combined deep CNN-LSTM network for the detection of novel coronavirus (COVID-19) using X-ray images Computer-aided detection of COVID-19 from X-ray images using multi-CNN and Bayesnet classifier A novel medical diagnosis model for COVID-19 infection detection based on deep features and Bayesian optimization A new COVID-19 Patients Detection Strategy (CPDS) based on hybrid feature selection and enhanced KNN classifier, 2020, Knowl Based Syst COVID-19 identification in chest X-ray images on flat and hierarchical classification scenarios Covid-19 classification by FGCNet with deep feature fusion from graph convolutional network and convolutional neural network Deep-COVID: Predicting COVID-19 from chest X-ray images using deep transfer learning CovXNet: A multi-dilation convolutional neural network for automatic COVID-19 and other pneumonia detection from chest X-ray images with transferable multi-receptive feature optimization CNN-based transfer learning-BiLSTM network: A novel approach for COVID-19 infection detection The ensemble deep learning model for novel COVID-19 on CT images Coronavirus disease (COVID-19) detection in Chest X-Ray images using majority voting based classifier ensemble Comparison of deep learning approaches to predict COVID-19 infection Role of intelligent computing in COVID-19 prognosis: a state-of-the-art review Learning distinctive filters for COVID-19 detection from chest X-ray using shuffled residual CNN MobileNetV2: Inverted residuals and linear bottlenecks, IEEE/CVF Conference on Computer Vision and Pattern Recognition COVID faster R-CNN: A novel framework to diagnose novel coronavirus disease (COVID-19) in X-Ray images Development Of A Deep Learning Model To Classify X-Ray Of Covid-19, Normal And Pneumonia-Affected Patients The effect of deep feature concatenation in the classification problem: An approach on COVID-19 disease detection A weakly-supervised framework for covid-19 classification and lesion localization from chest CT Very deep convolutional networks for large-scale image recognition Fusion methods for CNN-Based automatic modulation classification Automatic detection of COVID-19 disease using U-Net architecture based fully convolutional network Diagnosis of COVID-19 using deep neural networks and meta-heuristic-based feature selection on X-ray images Automated diagnosis of muscle diseases from EMG signals using empirical mode decomposition based method FBDM based time-frequency representation for sleep stages classification using EEG signals A non-parametric approach for multicomponent AM-FM signal analysis Instantaneous voiced/non-voiced detection in speech signals based on variational mode decomposition Decision support system for focal EEG signals using tunable-Q wavelet transform Determination of instantaneous fundamental frequency of speech signals using variational mode decomposition Automatic sleep stages classification based on iterative filtering of electroencephalogram signals Joint time-frequency domain based CAD disease sensing system using ECG signals Deep Learning with ConvNet Predicts Imagery Tasks Through EEG. Neural Process Lett Deep Learning-based Mammogram Classification for Breast Cancer Chronic obstructive pulmonary disease severity analysis using deep learning on multi-channel lung sounds MIDCAN: A multiple input deep convolutional attention network for Covid-19 diagnosis based on chest CT and chest X-ray Deep Rank-Based Average Pooling Network for Covid-19 Recognition COVID-19 Radiography Database Italian Society of Medical and Interventional Radiology (SIRM) COVID-19 Chest X-Ray dataset Dataset: COVID-19 Image Repository COVID-19 image data collection Labeled Optical Coherence Tomography (OCT) and chest X-Ray images for classification Dataset for Covid-19 Classification 3, 4 class Conceptualization, Methodology, Software, Investigation Karamjeet Singh: Data curation, Resources, Writing-Original draft preparation, Editing, Visualization. Deepika Koundal: Visualization, Investigation, Resources, Writing-Reviewing and Editing