key: cord-0824500-li9znxe5
authors: Hasan, Nahian Ibn
title: A Hybrid Method of Covid-19 Patient Detection From Modified CT-Scan/Chest-X-Ray Images Combining Deep Convolutional Neural Network And Two- Dimensional Empirical Mode Decomposition
date: 2021-07-23
journal: Computer Methods and Programs in Biomedicine Update
DOI: 10.1016/j.cmpbup.2021.100022
sha: 4dc4c97cdecbe10a1675d405d7240b361ea44bbd
doc_id: 824500
cord_uid: li9znxe5

The outbreak of the SARS-CoV-2/Covid-19 virus in 2019-2020 has made the world look for fast and accurate detection methods of the disease. The most commonly used tools for detecting Covid patients are Chest-X-ray or Chest-CT-scans of the patient. However, sometimes it’s hard for the physicians to diagnose the SARS-CoV-2 infection from the raw image. Moreover, sometimes, deep-learning-based techniques, using raw images, fail to detect the infection. Hence, this paper represents a hybrid method employing both traditional signal processing and deep learning technique for quick detection of SARS-CoV-2 patients based on the CT-scan and Chest-X-ray images of a patient. Unlike the other AI-based methods, here, a CT-scan/Chest-X-ray image is decomposed by two-dimensional Empirical Mode Decomposition (2DEMD), and it generates different orders of Intrinsic Mode Functions (IMFs). Next, The decomposed IMF signals are fed into a deep Convolutional Neural Network (CNN) for feature extraction and classification of Covid patients and Non-Covid patients. The proposed method is validated on three publicly available SARS-CoV-2 data sets using two deep CNN architectures. In all the databases, the modified CT-scan/Chest-X-ray image provides a better result than the raw image in terms of classification accuracy of two fundamental CNNs. This paper represents a new viewpoint of extracting preprocessed features from the raw image using 2DEMD.

The recent outbreak of SARS-CoV-2/ Covid-19/ Corona-Virus has affected people of 215 countries [1] all-over the world. SARS-CoV-2 affects people in different physiological manners. It is supposed to be very much lethal for geriatrics, children, and people with other significant physical conditions. Some of the common symptoms of SARS-CoV-2 are diarrhea, fever, tiredness, cough, etc. Most of the time, physicians diagnose prospective SARS-CoV-2 patients using CT-scan images and Chest-X-ray images. It is one of the quickest methods for the detection of pneumonia patients. The early diagnosis of Covid patients is very much necessary for efficient treatment. Bai et al. [2] has reported that some radiologists around the world classified SARS-CoV-2 from common viral pneumonia cases on chest CT-scan images with moderate sensitivity but high specificity. Hence, the automatic detection of SARS-CoV-2 patients is needed.

Consequently, automatic detection based on CNN networks and machine learning algorithms are necessary with higher accuracy and lower false-negative rate.

Deep learning-based Covid-patients detection has gained the attraction of the scientific community. Some of the reported methods involve direct usage of images through different convolutional neural network architectures. However, some other methods involve gathering features from traditional signal processing techniques and then train those features through CNN to classify the patients.

Jaiswal et al. [3] proposed a Densenet201-based transfer learning model to classify a Covid patient from a non-Covid patient. Chen et al. [4] has used a massive database of 46,096 anonymous images to train through deep learning models for Covid-19 patient detection and classification. On top of that, Barstugan et al. [5] used Grey Level Co-occurrence Matrix, Length Matrix, Local Directional Pattern, Size Zone Matrix, and Discrete Wavelet Transform as primary feature extraction methods, which are then trained through SVM machines for efficient detection of SARS-CoV-2 patients. On the other hand, Wang et al. [6] introduced COVID-Net, a deep learning based classifier of SARS-CoV-2 patients from Chest-X-ray images. They have designed and fine-tuned the architecture for their database. At the same time, Narin et al. [7] used different existing network architectures (i.e. ResNet50, Inception-ResNetV2, and InceptionV3) for classification purpose. Recently, Kareem et al. [8] has used ML models like Naive Bayes (NB), Random Forest (RF), and Support Vector Machine (SVM) for diagnosing Covid patients. Also, Waisy et al. [9] introduced a hybrid CheXNet model for detecting Covid patients by utilizing image enhancement and noise removal with pre-trained deep-learning models. The author also showed in another report [10] that the same enhancement and noise removal techniques can work as well with deep belief networks.

Apart from these, Rajaraman et al. [11] used a new strategy to localize the region of interest (ROI) in the Chest-X-ray images, which are then passed through CNN (VGG16 architecture) for final prediction and classification. Furthermore, Ozturk et al. [12] has reported the usage of Darknet and You-onlylook-once (YOLO) (an object detection system) for both binary and multi-class classification of SARS-CoV-2 patients. However, Sun et al. [13] have proposed a deep forest algorithm based on adaptive feature selection criteria for classification. They also used a deep forest model for a high-level representation of features. Besides, Bai et al. [14] compared the results of a deep learning model and radiologists. For this purpose, they first segmented the lung for excluding non-pulmonary regions of the CT. Ko et al. [15] have developed a Fast Track SARS-CoV-2 Classification Network (FCONet) to classify SARS-CoV-2 cases from CT-scan image. They have used pre-trained models like VGG16, ResNet-50, Inception-v3, or Xception to classify Covid patients. However, Hu et al. [16] has proposed a weakly supervised deep learning model for weakly labeled CT-scan images to classify SARS-CoV-2 patients. Apart from these, Mahmud et al. [17] has introduced a new neural network architecture, named 'CovXNet', which utilizes depth-wise convolution and different dilation rates. They used the new architecture for differentiating SARS-CoV-2 Chest-X-ray images from viral pneumonia, bacterial pneumonia, and normal patients. Moreover, Harmon et al. [18] has reported a similar approach to [14] , where they first segmented the lung and then used 3D deep learning models to classify the SARS-CoV-2 patients from CT-scan images.

In all of the deep-learning-based studies, either the raw input image or any lightly preprocessed image has been used as input data to the deep learning network. However, we wanted to explore this preprocessing task, where the raw image is processed by the mode decomposition technique so that it is easier for the network to learn the inherent features in a more effective way. In this study, two-dimensional Empirical Mode Decomposition (2DEMD) has been presented as a new strategy to extract features from CT-scan and Chest-X-ray images.

2DEMD has never been used as a preprocessing technique for deep-learningbased methods. The primary goal of the classification is to show that a modified image from 2DEMD performs better as input data for the deep CNN while classifying Covid patients. The paper represents that training a fundamental deep neural network with a modified CT-scan/Chest-X-ray (through 2DEMD) shows better performance than training with a raw image. The modified image acts as a better candidate than the raw image irrespective of the complexity and performance of the CNN and the variation of data in the databases. The proposed method has been validated on three publicly available SARS-CoV-2 databases -two comprised of CT-scan images, and the third consists of Chest-X-ray images.

This section represents the overview of the proposed method and a detailed discussion of the feature extraction methodology and CNN training. Next, the 2DEMD algorithm decomposes the image into separate IMFs, which are simply summed together to form a modified version of the real medical image. The modified image is then passed through the CNN for feature learning and classification.

2D EMD is an extension of single-dimensional EMD. EMD decomposes a signal into separate modes which are also known as Intrinsic Mode Functions (IMFs) [19] [20] . Each IMF has similar lengths and an equal number of zerocrossing and extrema points. The envelopes of the decomposed signals serve as oscillatory modes. The IMFs are non-orthogonal, but they can describe the signal adequately. Single dimensional EMD is effective for natural signals (i.e. Electrocardiogram signals) because it can track the non-linearity and nonstationarity of these signals. EMD signal conveys intrinsic features of a signal.

In this description, we, first, consider the analysis of single-dimensional EMD. For a 1D EMD, the signal values are specified with respect to time (t).

Digital representation of the signal will convert the time (t) parameter with the number of samples (n). However, for simplicity, we consider a time (t) varying signal. The minima and maxima points of a signal f (t) can be joined together to find the the lower envelope curve (f l (t)) and the upper envelope curve (f u (t)), respectively. The average value of the two signals is

This subtraction of mean value from the original signal (f (t)) results into the first proto-IMF signal, W 1 (t):

The proto-IMF signal, W 1 (t) then goes through a sifting process subsequently up to a threshold point. At this threshold point, the conditions for an Intrinsic Mode Function (IMF) are fulfilled [19] . These steps provide the very first IMF signal (s 1 ). The corresponding residue signal, (r 1 (t)), after the first step is,

Next, applying the sifting process to the first residue signal (r 1 (t)) in a similar fashion results in second, third and etc IMFs. Generally,

Here, N denotes the number of IMFs. Finally a residue signal (r N (t)) and

certain number of IMFs (s 1 (t), s 2 (t), ..., s N (t)) are obtained such that,

here, s j (t) is the j th order IMF. The lower-order IMFs correspond to highfrequency modes and higher-order IMFs resemble the low-frequency modes [19] .

In the uni-directional EMD, the method achieves uni-directional IMFs, in the case of two-dimensional EMD (2DEMD), there are 2D IMFs. The algorithm for 2DEMD is discussed in [21] . Morphological reconstruction extracts the 2D

IMFs during the sifting process of EMD. It helps in detecting the radial basis function (RBF) and image extrema to compute the surface interpolation. In this paper, the sifting process is allowed to increase up to 3000 th iteration for achieving lower error in achieving the IMFs. The whole sifting algorithm is specified in detail in [20] . According to Havlicek et al. [22] , a two-dimensional IMF is a zero-mean 2D AM-FM component. The image AM-FM decomposition is a separate algorithm that is based on partial unsupervised features. In unidirectional EMD, the process is fully unsupervised. During the 2DEMD process, first, morphological reconstruction identifies the extrema of the image, which is based on the geodesic operators. Next, Similar to the case of EMD, the envelope (in 2DEMD, this is a 2D envelope) is generated with an RBF. The local mean (Y(2D)) is calculated by averaging the maxima and minima envelope. Then subtraction of the mean (Y(2D)) from the original image results into the proto-2D-IMF signal. The sifting process is repeatedly applied to the proto-2D-IMF signal at every step until the conditions for a 2D-IMF are fulfilled.

Next, the same procedure is applied to the residual signal after the first 2D-IMF is achieved. The mechanism of extrema detection and the parameter specifications of 2DEMD are elaborately described in [21] .

In this paper, the 2DEMD is applied on CT-scan/Chest-X-ray images and the maximum number of IMFs to be extracted is set to 6. That means any image undergoing the 2DEMD process, can generate a varying number of IMFs, but all of them are considered for forming a modified CT-scan/Chest-X-ray image by simple summation. Some of the modified images are shown in Fig.4 and Fig.   5 . The decomposed 2DIMFs capture the texture profile of the image. 2DEMD

creates scopes for the extraction of spatial frequency components from coarsest to finest scales.

Two fundamental neural network architectures, (VGG16 [23] and VGG19 [23] ), have been used for the training of decomposed and modified image. VGG16

consists of 5 stages of convolutional operations, which requires an input image size of 224 × 224. Similarly, VGG19 also consists of 5 stages of convolutional operations, but with a higher number of operations per stage. The kernels of the CNN layers are initialized with the pre-trained 'imagenet' [24] weights of VGG16, VGG19, and Resnet152 models [23] . The learning rate is similar for all of the layers. A TESLA K80 Graphics Processing Unit has been utilized for the training purpose on a system of intel core-i9 CPU with 4 GB of memory. The training loss function is 'Categorical Cross-entropy', and 'softmax' activation is applied at the final output classification layer.

This section provides a detailed description of the databases, different training procedures, learning parameters, evaluation metrics, and training results.

The proposed method is evaluated on three publicly available databases of SARS-CoV-2. The first database was collected from [25] . on this database can be found in [26] [27] [28] [29] [30] [31] [32] [33] .

The second dataset was collected from [34] . The dataset contains in total of 746 CT-scan images of SARS-CoV-2 and non-SARS-CoV-2 patients. Among them, 349 images are from 216 SARS-CoV-2-positive patients. Several classification results have been mentioned in [35] [36] [37] [38] [39] [40] based on this database.

The third dataset was collected from [41] . This dataset consists of 813 Chest-X-ray images from multiple sources and multiple patients. Researchers have reported classification and segmentation results on this dataset in [12, [42] [43] [44] [45] .

The data distribution within the training-set and the testing-set is illustrated in Fig. 2 

The training procedure incorporates a learning rate scheduler for better and efficient learning of image features (generated in distributed CNN). The scheduler used for training is 'cosine-annealing'. Generally, such learning rate schedulers with restarting mechanism are also known as the stochastic gradient descent with warm restarts (SGDR) [46] . But this paper uses the restarting mechanism with the RMSProp optimizer. This restart technique frees the optimization from local minima over the optimization space at any time during the training. 'Cosine annealing with warm restart' consists of two parts. The first one is the 'cosine function' that acts as the learning rate annealing function.

The second part is the 'warm-restart' that makes the learning rate scheduler restart again from the initial point. The purpose of using such a scheduler is to maximize the probability of converging to the global minimum cost location and also to minimize the probability of being stuck at a local minimum cost point. For the purpose of this paper, an initial learning rate of 0.00001 has been used as the maximum learning-rate. Besides, there are 10 cycles accommodated within the range of training epochs. As a result, the minimum learning rate achieved was only 0.00000006155. Within the i th epoch, the learning rate is decayed using the following function specified in [46] .

where, η i max and η i min are the limits of the learning rate, T cur denotes the number of epochs that have passed since the last restart, T i denotes the periodicity of the restart (for example, after T i epochs, the learning rate is restarted again from the initial rate). Fig. 3 represents the schematic diagram of the 'cosineannealing' scheduler throughout the training epochs.

Several evaluation metrics have been incorporated into the training result analysis to convey the performance of the proposed method from varying perspective. This section provides a brief discussion of these evaluation metrics. 

This parameter is also known as the 'miss-rate'.

It denotes how often the predicted class matches the true class. In the proposed method, there are databases with two classes, which involve binary classification accuracy. However, the proposed method also reports the classspecific accuracy, which helps to explain the class-specific performance of the models. The classification accuracy can be expressed in terms of TP, TN, FP, FN as follows-

'Precision' denotes what proportion of positively infected patients were accurately predicted. The following equation describes the metric-

'Recall' indicates what proportions of originally infected people were correctly classified. The following formula expresses the idea of recall-

'Specificity' signifies the number of patients who were not infected, and the model also correctly classified them. The following formula describes specificity-

The F β score combines the precision and recall. If β = 1, this is known as F 1 score and if β = 2, it is known as F 2 score.. The formula is expressed as -

F 1 score provides equal importance to precision and recall. F 2 score provides weights recall higher than 'precision'.

AUC measures the area under the Receiver Operating Characteristic curve (ROC curve). It renders an overall performance of the model. AUC does not depend on the classification threshold. However, it is also a scale-invariant parameter. The higher the value of AUC (0 < AU C < 1), the better the performance of the model.

In this section, some sample analysis of the 2DEMD is presented. 2DEMD extracts the texture profile of an image. The overview is mentioned in the previous section. Here, the residual part of the 2DEMD is discarded and the IMFs are simply added together.

illustrates the effect of 2DEMD on Chest-X-ray images. However, the 2DEMD intensifies the inner-view of the lung. Also, discarding the residual signal from 2DEMD makes a better scope for visual inspection. In the modified images, the colors seem to differentiate different tissue types (i.e. liquid/empty space from bones/tissues). For example, most of the liquid and air-filling pores/cavities in the lung are colored blue, whereas most of the soft tissues and bones are colored red/yellow/orange. Also, the color helps to differentiate the densities of mucus in the lung in Covid patients. This bonus visualization benefit emerged while adding different IMFs.

This sections presents the classification result on all of the three databases using the proposed method in Table 1, Table 2, and Table 3 for database 1, 2, and 3, respectively. The evaluation metrics have been calculated per image (not per subject). Similarly, when the model is trained using VGG19 architecture and modified CT-scan, there is a similar pattern of a steady increase in these three metrics.

For example, in this case, the accuracy improves from 86.71% to 91.87%, the AUC increases from 96.64% to 97.43%, and the F1 score increases from 85.77%

to 91.94%. All other metrics like FNR, precision, recall, specificity, F2 score, etc. follow a random fluctuating pattern. Such steady and fluctuating patterns are verified through a second database which also contains CT-scan images. The results for database 2 are shown in Table 2 . Table 2 , it is seen that the FNR, Recall, and F2 score also increases when the models are trained with modified CT-scan images. Moreover, precision and specificity decrease when modeled with the modified CT-scan image. Therefore, a generalized pattern is followed by classification accuracy, AUC, and F1 score irrespective of different databases. But that is not the case for other metrics like FNR, precision, recall, specificity, and F2 score. Next, the proposed method is applied to a database containing Chest-X-ray images to see whether the same generalization can be deduced like CT-scan images. Table 3 shows the resultant metrics for database 3. From the modified image is a better tool for training than the raw image. Fig. 6 and Fig. 7 show the class-specific performance metrics for all of the three databases. mance in all evaluation metrics simultaneously. In medical diagnosis (i.e. the classification of Covid and non-Covid patients), the most important evaluation metric is the classification accuracy (shown in Fig. 6(a) ), which our proposed method achieves in all databases. We represented reports on other metrics as well to show the overall performance and to show that in achieving the best classification accuracy, what effects the training did impose on other metrics. It is evident that we had to trade off the performance in some metrics.

On average, the time it takes for the 2DEMD to analyze a single image in the dataset is less than 30 seconds. A single prediction from the CNN model takes less than 5 seconds. Hence, the method can be readily implemented.

In the recent advancements of SARS-CoV-2 detection and classification, most of the automatic detection algorithms work with the raw image of the CT-scan/Chest-X-ray image. But in this report, a hybrid method is utilized comprising traditional signal processing and deep-learning methodology. First, On the other hand, for database 3, the FNR (c) and Recall (e) decreases when trained with the modified Chest-X-ray image. However, F2 score (b) decreases for database 1 but increases for database 2 when the model is trained with modified CT-scan. Moreover, for database 2, the Precision (d) and

Specificity (f) decreases. Besides, in database 2, the FNR decreases with modified CT-scan.

the raw CT-scan/Chest-X-ray image is converted to a single channel (gray-scale)

image, and then 2DEMD is applied to the image. Next, the residual part of the decomposition is discarded, and all of the Intrinsic Mode Functions (IMFs)

are synthesized together to form a modified CT-scan/Chest-X-ray image. This modification represents the image with greater scope for visual inspection, and it also extracts the texture profile of the image. After that, the modified image is trained through a deep Convolutional Neural Network (CNN), and a final fully connected layer classifies the image either as a 'Covid-Positive' or 'Covid-Negative'. The method is applied to CT-scan and Chest-X-ray images of three publicly available databases. In all of the cases, the models trained with modified CT-scan/Chest-X-ray images provide superior performance to the model that is trained with the raw images from the perspective of accuracy and F1

score. However, the models also show some database-specific steady and fluctuating patterns in other types of resultant metrics. Two fundamental CNN architectures are used for validation. One of the main purposes of this paper is not to achieve the best classification result, but to compare the performance of raw image and 2DEMD-based modified image training while maintaining the same set of hyper-parameter and learning methodology. The analysis in this paper makes it possible for the 2DEMD-based modified CT-scan/Chest-X-ray image to be used as a performance-boosting criterion in the deep learning-based classification of SARS-CoV-2 patients. In summary, this paper represents a new viewpoint for deep-learning engineers to utilize the feature extraction power of 2DEMD. One of the limitations of this work is that it has not been applied to a large set of chest-X-Ray images. It seems like the method performs better on chest CT scans rather than on chest-X-Ray images. The reason might be due to chest ribs and diaphragms in the chest-X-Ray image in higher quantity than in a ct-scan. Besides, instead of taking all of the decomposed IMFs, how will the result be affected if only several IMFs are combined to form a modified image, is still an area of possible future work

Declarations of interest: none. This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

☒ The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

☐The authors declare the following financial interests/personal relationships which may be considered as potential competing interests:

Performance of radiologists in differentiating covid-19 from viral pneumonia on chest ct

Classification of the covid-19 infected patients using densenet201 based deep transfer learning

Deep learning-based model for detecting 2019 novel coronavirus pneumonia on high-resolution computed tomography: a prospective study

Coronavirus (covid-19) classification using ct images by machine learning methods

Covid-net: A tailored deep convolutional neural network design for detection of covid-19 cases from chest x-ray images

Automatic detection of coronavirus disease (covid-19) using x-ray images and deep convolutional neural networks

Realizing an effective covid-19 diagnosis system based on machine learning and iot in smart hospital environment

Covid-chexnet: hybrid deep learning framework for identifying covid-19 virus in chest xrays images

Coviddeepnet: Hybrid multimodal deep learning system for improving covid-19 pneumonia detection in chest x-ray images

Visualization and interpretation of convolutional neural network predictions in detecting pneumonia in pediatric chest radiographs

Automated detection of covid-19 cases using deep neural networks with x-ray images

Adaptive feature selection guided deep forest for covid-19 classification with chest ct

Ai augmentation of radiologist performance in distinguishing covid-19 from pneumonia of other etiology on chest ct

Covid-19 pneumonia diagnosis using a simple 2d deep learning framework with a single chest ct image: Model development and validation

Weakly supervised deep learning for covid-19 infection detection and classification from ct images

Covxnet: A multi-dilation convolutional neural network for automatic covid-19 and other pneumonia detection from chest x-ray images with transferable multi-receptive feature optimization

Artificial intelligence for the detection of covid-19 pneumonia on chest ct using multinational datasets

Ecg denoising based on the empirical mode decomposition

An adaptive data analysis method for nonlinear and nonstationary time series: the empirical mode decomposition and hilbert spectral analysis

Image analysis by bidimensional empirical mode decomposition

Multidimensional quasieigenfunction approximations and multicomponent am-fm models

Very deep convolutional networks for largescale image recognition

2009 IEEE conference on computer vision and pattern recognition

Sars-cov-2 ct-scan dataset: A large dataset of real patients ct scans for sars-cov-2 identification

A fully automated deep learningbased network for detecting covid-19 from a new and large lung ct scan dataset

Covid-ct-md, covid-19 computed tomography scan dataset applicable in machine learning and deep learning

A deep learning and grad-cam based color visualization approach for fast detection of covid-19 cases using chest x-ray and ct-scan images

Automatic screening of covid-19 using an optimized generative adversarial network

Covid-19: Automatic detection of the novel coronavirus disease from ct images using an optimized convolutional neural network

Contrastive cross-site learning with redesigned net for covid-19 ct classification

A new approach for computer-aided detection of coronavirus (covid-19) from ct and x-ray images using machine learning methods

Applications of artificial intelligence in battling against covid-19: a literature review

Covid-ct-dataset: a ct scan dataset about covid-19

Review of artificial intelligence techniques in imaging data acquisition, segmentation and diagnosis for covid-19

Inf-net: Automatic covid-19 lung infection segmentation from ct images

Jcs: An explainable covid-19 diagnosis system by joint classification and segmentation

Artificial intelligence in the battle against coronavirus (covid-19): a survey and future research directions

Sampleefficient deep learning for covid-19 diagnosis based on ct scans

Iteratively pruned deep learning ensembles for covid-19 detection in chest x-rays

Covid-19 image data collection: Prospective predictions are the future

Automatic detection of coronavirus disease (covid-19) using x-ray images and deep convolutional neural networks

Covid-19: automatic detection from x-ray images utilizing transfer learning with convolutional neural networks

Classification of covid-19 in chest x-ray images using detrac deep convolutional neural network

Detection of coronavirus disease (covid-19) based on deep features and support vector machine

Sgdr: Stochastic gradient descent with warm restarts