key: cord-0721371-pbjlp7af
authors: Hosseinzadeh, Hamidreza
title: Deep Multi-view Feature Learning for Detecting COVID-19 based on Chest X-ray Images
date: 2022-02-23
journal: Biomed Signal Process Control
DOI: 10.1016/j.bspc.2022.103595
sha: c562621927d37bff7a22e81a0de114ff5f4869ac
doc_id: 721371
cord_uid: pbjlp7af

AIM: COVID-19 is a pandemic infectious disease which has influenced the life and health of many communities since December 2019. Due to the rapid worldwide spread of this highly contagious disease, making its early detection with high accuracy important for breaking the chain of transition. X-ray images of COVID-19 patients, reveal specific abnormalities associated with this disease. METHODS: In this study, a multi-view feature learning method for detecting COVID-19 based on chest X-ray images is presented. This method provides a framework for exploiting the multiple types of deep features, which is able to preserve both the correlative and the complementary information, and achieve accurate detection at the classification phase. Deep features are extracted using pre-trained deep CNN models of AlexNet, GoogleNet, ResNet50, SqueezeNet, and VGG19. The learned feature representation of X-ray images are then classified using ELM. RESULTS: The experiments show that our method achieves accuracy scores of 100%, 99.82%, and 99.82% in detecting three classes of COVID-19, normal, and pneumonia, respectively. The sensitivities of three classes are 100%, 100%, and 99.45%, respectively. The specificities of three classes are 100%, 99.73%, and 100%, respectively. The precision values of three classes are 100%, 99.45%, and 100%, respectively. The F-scores of three classes are 100%, 99.73%, and 99.72%, respectively. The overall accuracy score of our method is 99.82%. CONCLUSIONS: The results demonstrate the effectiveness of our method in detecting COVID-19 cases and can therefore assist experts in early diagnosis based on X-ray images.

COVID-19 is an infectious respiratory disease, which has rapidly spread all across the world and caused the death of hundreds of thousands and infected millions of people [1] [2] [3] [4] . COVID-19 has been announced as a pandemic by the World Health Organization (WHO) [5] [6] . One of the important steps to prevent the transmission of COVID-19 infection to healthy population is to effectively screen infected patients to isolate and treat them.

Currently, reverse transcription-polymerase chain reaction (RT-PCR) taken from the respiratory tract is the main screening method for COVID-19 [7] . However, it is a costly and time consuming detection method. Those detection tests that give results within minutes are insensitive and occasionally yield false negatives. On the other hand, RT-PCR test kits are limited in number. Therefore, the development of a fast, low cost, and reliable method for automatic detection of COVID-19 is essential. As an alternative to the RT-PCR testing method, artificial intelligence-based automated detection of COVID-19 from radiological imaging of patients can be used [8] [9] [10] . Radiographic imaging of the chest, such as computed tomography (CT) and X-ray, is helpful for early COVID-19 detection. In this study, X-ray is preferred to the CT due to cost-effective, low ionizing radiation exposure to patients, and widespread availability of X-ray machines in almost every hospital.

Ismael and Sengür [11] presented three deep convolutional neural networks (CNN) approaches including extraction of deep features, fine-tuned pre-trained CNN models, and end-to-end trained CNN model to detect COVID-19 cases. In the first approach, SVM classifier with various kernel functions was used to classify the deep features. Uçar et al. [12] presented a deep learning method in which extracted deep features from X-ray images on different color spaces were applied to a bidirectional LSTM network for classification the data into COVID-19 or pneumonia. In [13] , features of the convolution and fully connected layers of the CNNbased AlexNet model were extracted and combined. The important features were selected by the Relief algorithm and then fed into the SVM classifier for detecting COVID-19 cases. In [14] , a CNN model based on class decomposition and transfer learning was developed to detect COVID-19 cases. Khan et al. [15] presented an Xception deep learning architecture named CoroNet for the detection of COVID-19 cases. In [16] , deep features were extracted from MobileNetV2 and SqueezeNet models, and then the feature sets obtained were processed with social mimic optimization (SMO) for feature selection and combination. Finally, the combined feature set was fed into a SVM classifier to detect COVID-19 cases. Elkorany and Elsharkawy [17] presented a COVID-19 detection method in which the ShuffleNet and SqueezeNet models were employed as a feature extractor and the SVM used as classifier. Ashour et al. [18] presented an ensemble-based bag-of-features (BoF) model to detect COVID-19 and normal cases. They utilized the grid method, and SURF descriptor for the keypoints determination and their feature vectors extraction, respectively. In [19] , CNN based transfer learning approaches were used to detect COVID-19 cases. The overall accuracy score of 94.72% was achieved using MobileNetV2. Canayaz [20] used two meta-heuristic algorithms containing binary particle swam optimization (BPSO) and binary gray wolf optimization (BGWO) to select the efficient features among the features extracted from deep CNN models. The selected features were then applied to the SVM to classify the data as COVID-19, normal, and pneumonia cases. The image contrast enhancement algorithm was also used to preprocess the image set. Wang et al. [21] presented an FGCNet model to fuse the extracted features from graph convolutional network (GCN) and CNN for detecting COVID-19 based on CT images. In [22] , deep features were extracted from CT images with different pre-trained models, and then the best two features were fused using discriminant correlation analysis. Hasoon et al. [23] presented a method in which the LBP, HOG, and Haralick features were used as features and fed into the KNN and SVM classifiers to detect COVID-19 cases. Al-Waisy et al. [24] presented a COVID-DeepNet model for detecting COVID-19 cases, which fuses the predictions of two deep learning models to make the final decision. Another fusion of deep learning models, COVID-CheXNet, was presented in [25] . In [26] , a comprehensive investigation was presented to detect COVID-19 cases using machine learning and convolutional deep learning models. In [27] , a feature-fusion based approach was presented to sort COVID-19 related medical waste.

Most previous studies discussed use a single pre-trained CNN model. The extracted features from different pre-trained CNN models often describe information of the same image from different views. Therefore, we propose using a multi-view feature learning framework to exploit useful information from different views so that more comprehensive feature representation may be learned for the diagnosis of COVID-19. The main contributions of this study are as follows:

 A deep multi-view feature learning method for detecting COVID-19 based on X-ray images is presented.

 Deep features are extracted using pre-trained deep CNN models of AlexNet, GoogleNet, ResNet50, SqueezeNet, and VGG19.  The method transforms the multi-view feature space into a feature space where both the complementary and the correlative information of different views are preserved.  Accurate classification of the feature representation of images into COVID-19, normal, and pneumonia cases is performed using ELM classifier.

The organization of the paper is as follows. Section 2 demonstrates the dataset and introduces the proposed method used for COVID-19 detection. Section 3 describes the experimental analysis of the study and discussions. Finally, Section 4 provides the conclusion and comments on further extension of this method.

The dataset used here consists of COVID-19, normal, and pneumonia chest X-ray images [20] . This dataset was a collection of 1092 images with three classes created from publicly available chest X-ray datasets [28] [29] [30] . The number of images of each class was 364. Fig. 1 shows sample X-ray images of the dataset with their corresponding marked lesions. 

Deep learning using CNN has obtained great success in diagnosing disease using medical image analysis [31] [32] . The diagnosis of COVID-19 using CNN has become a popular research technique due to the extraction of powerful features. In medical image classification for rare or emerging diseases, enough labeled images are not available to train a CNN model from scratch. In such cases, ImageNet pre-trained CNN models can be used [33] [34] [35] [36] [37] .

In this study, the pre-trained CNN model is used as feature extractor to construct the image feature space. The extracted features from different types of pre-trained CNN models generally characterize different description of images. Therefore, we throw the images into different pre-trained CNN models to construct the multi-view feature space for each image by considering the extracted features from each model as a view.

Then, the features of all views are concatenated and fed into the presented multi-view feature learning method to achieve more comprehensive features.

Given a training data matrix from classes, where and is the number of samples.

is the label vector in which only the element [ 1 , 2 ,…, ] T ∈ ℝ × ∈ ℝ associated with the assigned class is 1 and all the others are -1.

The optimization problem of least square regression (LSR) can be written as follows:

where is a transformation matrix, is an all one vector, is an intercept vector, and is a trade-

Considering the label matrix as regression target is too strict and inappropriate for classification, since it leads to wrong ± 1 penalization for correct classification that are far from [38] [39] . In order to solve this issue, the regression target is reformulated ± 1 as where is a non-negative adjustment matrix, and is a Hadamard product operator. This strategy can + ⊙ ∈ ℝ × ⊙ enlarge the distance between the true and the false classes. The optimization problem (1) can rewritten as follows:

The optimization problem (2) can be extended to the multi-view scenario, where different views of the data are reflected by different types of features. Multi-view learning provides a mechanism for exploiting multiple types of features [40] [41] [42] . Let = denote a data matrix from the -th view, where is the feature space dimension for the -th view. The

optimization problem for multi-view learning scenario is expressed as follows:

where is a set of positive weight parameters. For simplicity, the weight parameter is merged into the transformation matrix 

.

The optimization problem (4) is convex due to the convexity of both terms. The convexity implies that the optimal solution for the parameters , , , and exists. Therefore, an effective procedure is developed to iteratively update parameters , , , and to obtain the optimal solution [39] . The algorithm is detailed as follows:

Updating the parameters and during the -th iteration: With the fixed parameters and , the optimization (4) can be rewritten as ) . .

where and . The optimal and during the -th iteration can be derived by setting the partial (0) = { (0) } = 1 = 1 ( + 1) deviations of the objective function to zero. The optimal solution with respect to is ( , )

Also, the optimal solution with respect to is

feature in during the -th iteration, and in which is an identity matrix and is an all one = -

Updating the adjustment matrix during the -th iteration: When the parameters , , and are fixed, the ( + 1) (4) can be rewritten as

The optimal solution with respect to is

where denotes the predicted labels, computes element-wise absolute values of a matrix, and = T ( + 1) +

Updating the adaptive weight parameter during the -th iteration: With the fixed parameters , , and , ( + 1)

the optimization problem (4) can be rewritten as

where is the view index. The optimal solution with respect to is

The obtained weight parameters can be further used to calculate new weighted transformation matrix in the next

iteration. The algorithm is terminated when , where

[ 1 , 2 ,…, ] T Once the weighted transformation matrix and intercept vector have been learned, the new representation for the training dataset can be achieved using (12) , which is used to train a classifier.

In the testing phase, the weighted transformation matrix and intercept vector are used to compute the new representation of the test dataset, and the trained classifier is then used to classify the test dataset. In this algorithm, the Covid-19 detection is done based on multi-view deep feature learning, so we named it MV-COVIDet. In this study, we use extreme learning machine (ELM) [43] as classifier due to its fast learning procedure and remarkable generalization performance.

We first introduce the experimental setup and then perform experiments on the X-ray images data to assess the effectiveness of the MV-COVIDet method in detecting COVID-19, normal, and pneumonia cases. Finally, the results of our method are compared with previous studies.

The five types of pre-trained models are considered to extract deep learning based features (e.g. =5). They are AlexNet, GoogleNet, ResNet50, SqueezeNet, and VGG19. Deep features are extracted from the fully connected layer "fc8" in the AlexNet and VGG19 model, while the fully connected layer "loss3-classifier" is used in the GoogleNet model, the fully connected layer "fc1000" is used in the ReNet50 model, and the pooling layer "pool10" is used in the SqueezeNet model. The input image size of all models is 224 224, except for the AlexNet and SqueezeNet models in which the input image size is 227 227. The training of × × each model is realized in 50 epoch, and the mini-batch size is 64. The stochastic gradient descent (SGD) optimization algorithm with an initial learning rate of is used as solver. 10 -5

In the experiments, the number of hidden neurons of ELM is empirically set as and the sigmoid function is adopted as = 220 an activation function. The trade-off parameter is tuned using a grid search technique from the set .

{10 -3 ,10 -2 ,10 -1 ,1,10,10 2 ,10 3 }

The termination parameter is set as . = 0.005

The 5-fold cross-validation technique is used to evaluate the performance of the MV-COVIDet method. The performance metrics used for the analysis of the experimental results are sensitivity (Se), specificity (Sp), precision (Pre), F-score, accuracy (Acc), and overall accuracy (Overall Acc). The metrics are formulated as follows: 

In the first set of experiments, the extracted deep features from AlexNet, VGG19, GoogleNet, ResNet50, and SqueezeNet models are applied individually as the input to MV-COVIDet method to assess the performance of the method in single-view classification. Besides, the classification results of ELM classifier applied on each extracted deep feature is taken as the baseline. The experimental results are shown in Tables 1 and 2. Tables 1 and 2 show the performance of MV-COVIDet method and ELM classifier in terms of sensitivity, specificity, precision, F-score, accuracy, and overall accuracy, respectively. As we can see in Table 1 , the MV-COVIDet method fed with deep features yielded promising results. VGG19 features achieved the highest overall accuracy score of 98.99%, while the AlexNet features achieved an overall accuracy score of 98.81% as the second best score, and the SqueezeNet features achieved an overall accuracy score of 98.08% as the third best score. The percentage of success for classifying COVID-19 cases using VGG19, AlexNet, and SqueezeNet features are 99.63%, 99.63%, and 99.27%, respectively. Table 2 shows that the overall accuracy scores achieved by VGG19, AlexNet, and SqueezeNet features are 98.53%, 96.61%, and 92.86%, respectively. It is observed that the MV-COVIDet method provides successful enhancement in detecting COVID-19, normal, and pneumonia cases than applying ELM classifier on extracted deep features. The higher performance of MV-COVIDet method is related to the nature of learned feature representation.

The confusion matrix obtained using the MV-COVIDet method for the VGG19 features is shown in Fig. 2 . As observed in Fig. 3 , MV-COVIDet method produced the top three best-performed results on extracted deep features from VGG19, AlexNet, and SqueezeNet models. Therefore, in the second set of experiments, the focus is on the concatenation of these types of deep features with each other to construct multi-view data. The analysis results of MV-COVIDet method for multi-view data are shown in Table 3 . The reported metric values in Table 3 indicates that the MV-COVIDet method using multiple types of features generally performs better compared to using single feature type. This confirms the logic of combining multiple types of features. It is seen that increasing the number of types of features improves the classification performance. The MV-COVIDet method on the concatenation of AlexNet, SqueezeNet, and VGG19 features ensured satisfactory results with an overall accuracy of 99.73%.

In the third set of experiments, we examined the effect of constructing multi-view data with top four feature types on the performance of the MV-COVIDet method. The analysis results of MV-COVIDet method on four-view data are shown in Table 4 . From Table 4 , it can be observed that increasing the number of types of features from 3 to 4 results in a slight performance improvement for MV-COVIDet method, thus indicating that the method is nearly saturated. Therefore, we do not use more feature types to construct multi-view data. Fig. 4 shows the confusion matrix obtained using the MV-COVIDet method for the concatenated features of AlexNet, ResNet50, SqueezeNet, and VGG19. From Fig. 4 , it can be observed that only 2 pneumonia samples are misclassified and the other samples are correctly classified, which signifies the robustness of the method. In the fourth set of experiments, the performance metrics of the MV-COVIDet method with and without adaptive weights are compared to evaluate the effectiveness of the adaptive weights. In this scenario, we used the concatenation of AlexNet, ResNet50, SqueezeNet, and VGG19 features. The experimental results are shown in Fig. 6 . It can be seen that the MV-COVIDet method with the adaptive weights performs better than the one without the adaptive weights. It demonstrates the effectiveness of the adaptive weights. 

Finally, the MV-COVIDet performance is compared with previous studies performed on X-ray images. It is important to know that a fair comparison of results is not possible due to differences in datasets, methods, and validation techniques. The comparison results are shown in Table 5 .

It can be observed from Table 5 that the MV-COVIDet method outperforms other methods in terms of overall accuracy score, validating the importance of effectively exploiting the multiple types of deep features. Since the dataset we used in this study is the same as the dataset in the study of [20] , a one-to-one comparison is performed only with this study. In [20] , the image contrast enhancement algorithm (ICEA) was performed on X-ray images to provide better quality ones. As can be observed, [20] yielded an overall accuracy score of 99.38% while our method reached an overall accuracy score of 99.82% without any use of image processing technique. 

The outbreak of COVID-19 has put a major pressure on health centers, preventing them from providing effective treatment without the risk of infection. In this study, a method called MV-COVIDet was presented for detecting COVID-19 based on X-ray images which can help medical clinicians in making appropriate decisions for diagnosis. This method leverage multiple types of deep features extracted from X-ray images to learn an efficient feature representation and feed them into the ELM classifier to classify the data into COVID-19, normal, and pneumonia cases. The MV-COVIDet method was evaluated with different types of deep features. Simulation results showed that our method achieved the highest overall accuracy score of 99.82% on the concatenated features of AlexNet, ResNet50, SqueezeNet, and VGG19. The accuracy scores of COVID-19, normal, and pneumonia classes were 100%, 99.82%, and 99.82%, respectively. The comparison of our method with the previous ones indicated its superiority in terms of overall accuracy score. The limitation of the MV-COVIDet method is that it cannot properly handle a set of X-ray and CT images. In the future, we plan to expand the image collection to include X-ray and CT images of lung diseases and assess the performance of the presented method on it. In addition, we intend to extend this method for a scenario where only a limited number of data is labeled.

Application of deep learning techniques for detection of COVID-19 cases using chest X-ray images: A comprehensive study

A review on deep learning techniques for the diagnosis of novel coronavirus (covid-19)

Detection of COVID-19 in X-ray images by classification of bag of visual words using neural networks

Review on COVID-19 diagnosis models based on machine learning and deep learning approaches

Covid-19 infection: origin, transmission, and characteristics of human coronaviruses

World Health Organization, Coronavirus disease (COVID-19) pandemic

a diagnostic challenge

Sensitivity of chest CT for COVID-19: comparison to RT-PCR

Chest CT findings in Coronavirus Disease-19 (COVID-19): relationship to duration of infection

Chest CT for typical 2019-nCoV pneumonia: relationship to negative RT-PCR testing

Deep learning approaches for COVID-19 detection based on chest X-ray images

Automated detection of Covid-19 disease using deep fused features from chest radiography images

COVIDetectioNet: COVID-19 diagnosis system based on X-ray images using features selected from pre-learned deep features ensemble

Classification of COVID-19 in chest X-ray images using DeTraC deep convolutional neural network

CoroNet: A deep neural network for detection and diagnosis of COVID-19 from chest x-ray images

COVID-19 detection using deep learning models to exploit Social Mimic Optimization and structured chest X-ray images using fuzzy color and stacking approaches

COVIDetection-Net: A tailored COVID-19 detection from chest radiography images using deep learning

Ensemble-based bag of features for automated classification of normal and COVID-19 CXR images

Covid-19: automatic detection from x-ray images utilizing transfer learning with convolutional neural networks

Diagnosis of COVID-19 using deep neural networks and meta-heuristic-based feature selection on X-ray images

Covid-19 classification by FGCNet with deep feature fusion from graph convolutional network and convolutional neural network

COVID-19 classification by CCSHNet with deep fusion using transfer learning and discriminant correlation analysis

COVID-19 anomaly detection and classification method based on supervised machine learning of chest X-ray images

COVID-DeepNet: hybrid multimodal deep learning system for improving COVID-19 pneumonia detection in chest X-ray images

COVID-CheXNet: hybrid deep learning framework for identifying COVID-19 virus in chest X-rays images

A comprehensive investigation of machine learning feature extraction and classification methods for automated diagnosis of COVID-19 based on X-ray images

Artificial intelligence-based solution for sorting COVID related medical waste streams and supporting data-driven decisions for smart circular economy practice

COVID-19 image data collection

Can AI help in screening viral and COVID-19 pneumonia?

Labeled Optical Coherence Tomography (OCT) and Chest X-Ray Images for Classification

A survey on deep learning in medical image analysis

Computer-aided detection of COVID-19 from X-ray images using multi-CNN and Bayesnet classifier

ImageNet classification with deep convolutional neural networks, In: Advances in neural information processing systems

Very deep convolutional networks for large-scale image recognition

Deep residual learning for image recognition

Going deeper with convolutions

SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and < 0.5 MB model size

Discriminative least squares regression for multiclass classification and feature selection

Adaptive-weighting discriminative regression for multi-view classification

Deep multi-view learning methods: a review

Survey on deep multi-modal data analytics: collaboration, rivalry, and fusion

Diagnosis of coronavirus disease 2019 (COVID-19) with structured latent multi-view representation learning

Extreme learning machine: theory and applications

☒ The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.☐ The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: