key: cord-0727849-jo5frak7
authors: Zhang, Yu-Dong; Satapathy, Suresh Chandra; Liu, Shuaiqi; Li, Guang-Run
title: A five-layer deep convolutional neural network with stochastic pooling for chest CT-based COVID-19 diagnosis
date: 2020-11-03
journal: Mach Vis Appl
DOI: 10.1007/s00138-020-01128-8
sha: 7453b950b162b2c170bb4c21aa430d07dc2512b0
doc_id: 727849
cord_uid: jo5frak7

Till August 17, 2020, COVID-19 has caused 21.59 million confirmed cases in more than 227 countries and territories, and 26 naval ships. Chest CT is an effective way to detect COVID-19. This study proposed a novel deep learning model that can diagnose COVID-19 on chest CT more accurately and swiftly. Based on traditional deep convolutional neural network (DCNN) model, we proposed three improvements: (i) We introduced stochastic pooling to replace average pooling and max pooling; (ii) We combined conv layer with batch normalization layer and obtained the conv block (CB); (iii) We combined dropout layer with fully connected layer and obtained the fully connected block (FCB). Our algorithm achieved a sensitivity of 93.28% ± 1.50%, a specificity of 94.00% ± 1.56%, and an accuracy of 93.64% ± 1.42%, in identifying COVID-19 from normal subjects. We proved using stochastic pooling yields better performance than average pooling and max pooling. We compared different structure configurations and proved our 3CB + 2FCB yields the best performance. The proposed model is effective in detecting COVID-19 based on chest CT images.

tion by radiologists is tedious and easy to be influenced by fatigue, emotion, and other factors. A smart diagnosis system via computer vision and artificial intelligence can benefit patients, radiologists, and hospitals.

Traditional artificial intelligence (AI) and modern deep learning (DL) methods have achieved excellent results in analyzing medical images, e.g., Lu [6] proposed a radialbasis-function neural network (RBFNN) to detect pathological brains. Yang [7] presented a kernel-based extreme learning classifier (K-ELM) to create a novel pathological brain detection system. Their method was robust and effective. Lu [8] proposed a novel extreme learning machine trained by the bat algorithm (ELM-BA) approach. Jiang [9] used a six-layer convolutional neural network to recognize sign language fingerspelling. Their method is abbreviated as 6L-CNN-F, here F means fingerspelling. Szegedy et al. [10] presented the GoogLeNet. Yu and Wang [11] suggested the use of ResNet18 for mammogram abnormality detection. Two references provide systematic reviews of machine learning techniques in detecting COVID-19 [12, 13] . Besides, there are some successful applications in other industrial and academic fields using traditional AIs [14] [15] [16] [17] [18] .

This study used deep convolutional neural network (DCNN) as the backbone. To make our algorithm effective in detecting COVID-19, we proposed three improvements, (i) We introduced stochastic pooling (SP) to replace traditional average pooling and maximum pooling methods; (ii) We created conv block (CB) by combining conv layer and batch normalization, and (iii) we created fully connected block (FCB) by combining dropout layer and fully connected layer.

Those three improvements help enrich the performance of the basic DCNN, and we name our proposed algorithm as "5-layer DCNN with stochastic pooling for COVID-19 (5L-DCNN-SP-C) algorithm." Sections 2, 3, 4, and 5 present the dataset, methodology, results, and conclusions, respectively.

We enrolled 142 COVID-19 subjects and 142 healthy controls (HCs) from local hospitals. CCT was performed on all subjects, and three-dimensional volumetric images were obtained. Slice level selection (SLS) method was used: For COVID-19 pneumonia patients, the slice showing the largest size and number of lesions was selected. For healthy controls, any level of the image can be selected. Use this slice level selection method, we extract 320 images (resolution: 1024 × 1024) from both COVID-19 patients and HC subjects, respectively. The demographics of our image set are offered in Table 1 . Table 2 shows the abbreviation list for easy reading. 

Let us set the original CCT image set to be S 1 , which is composed of n CCT images as

First, we compress the three-channel color image to gray image, and get the grayscale image set S 2 as

Second, the histogram stretching (HS) method was firstly employed to increase the image's contrast. For i-th image s 2 (i), the new histogram stretched image s 3 (i) was obtained as the minimum value of CCT image s 2 (i). ε max 2 (i) means the maximum value of image s 2 (i).

In all, we get the histogram stretched dataset S 3 as

Third, we crop the images to remove the texts at the margin area, and the checkup bed at the bottom area. Thus, we get the cropped dataset S 4 as Fourth, we downsampled the image s 4 (i) to size of [ , ], and we now get the resized image set S 5 as (7) where means downsampling operation.

128 in this study. Figure 1 shows the above four preprocessing steps. Table 3 compares the size and storage per image at every preprocessing step. We can see here after the five-step preprocessing procedure, each image will only cost about 0.52% of its original storage. The byte compression ratio (BCR) was calculated as: BCR byte(s 5 ) ÷ byte(s 1 ) 65, 536 ÷ 12, 582, 912 0.52%. Figure 2 shows two samples of our collected and preprocessed dataset S 5 , from which we can clearly observe the clinical biomarkers of COVID-19. Cui et al. [19] reported the preliminary CT findings of COVID-19 in their publication. Tuncer et al. [20] used chest CT images, and then developed a local binary pattern and iterative ReliefF algorithm. There are more open publications that show it is feasible to develop effective AI systems based on CCT images.

Deep convolutional neural network (DCNN) is a king of new artificial neural network. Its main feature is to use multiple layers to build a deep neural network. Generally, DCNN is composed of conv layers (CLs), pooling layers (PLs), and fully connected layers (FCLs) [21] [22] [23] [24] [25] . Figure 3 presents a simplistic instance consisting of 2 CLs, 2 PLs, and 2 FCLs. On the right part of Fig. 3 , The blue rectangle means FCL block, and red rectangle means the softmax function. DCNNs could reach better performances than old-dated AI methods, because they learn the feature from the data during the training procedure. There is no need to consume much time in feature engineering.

The essential operation in DCNN is convolution. The CL performed 2D convolution along the width and height directions. Note that the weights in CNN are initialized with random, and then learnt from data itself by network training. Figure 4 illustrates the pipeline of input feature maps passing across a CL. Assume there is an input matrix, J kernels (K 1 , 

, and an output O, with theirs sizes S defined as

where (W , X , C) represent the size of height, width, and channels of the matrix, respectively. Subscript I, K, and O represent input, kernel, and output, respectively. J denotes total number of filters. Note that

which means the channel of input C I should equal the channel of kernel C K , and the channel of output C O should equal the number of filters J.

Assume those filters move with padding of B and stride of A, we can get their relationship by simple math as:

where . represents the floor function. Afterward, CL's outputs are hurled into a nonlinear activation function (NLAF) σ , that usually selects the rectified linear unit (ReLU) function.

ReLU is preferred to traditional NLAFs such as hyperbolic tangent (HT) and sigmoid (SM) function

The activation maps (AMs) after each block within DCNN are usually too large, i.e., the size of their width, length, and channels are too large to handle, which will cause (i) overfitting of the training and (ii) large computational costs. Pooling layer (PL) is a form of nonlinear downsampling (NLDS) method to solve above issue. Further, PL can provide invariance-to-translation property to the AMs. For a 2 × 2 region, suppose the pixels within the region ϕ are

The average pooling (AP) calculates the mean value in the region ϕ. Assume the output value after NLDS is z, we can have

where |ϕ| means the number of elements of region ϕ.

Here, |ϕ| 4 if we use a 2 × 2 NLDS pooling. Using Fig. 5 as an example, and assuming the regioṅ ϕ at 2 nd row 1 st column of the input AM, I is chosen, i.e.,φ I (row 2, col 1); thus, we have z AṖ ϕ average(φ) (4 + 4 + 3 + 9) ÷ 4 20 ÷ 4 5 . The max pooling (MP) operates on the region ϕ and selects the max value. Note that both AP and MP work on every slice separately.

In Fig. 5 , z M Ṗ ϕ max(φ) max(4 + 4 + 3 + 9) 9 . In practice, scholars observed that the AP did not work well, because all pixels in the region ϕ are within the arguments of the NLDS function; hence, it could down-weight (DW) intense activation owing to numerous near-zero pixels.

For example, in our regionφ, the strongest value 9 DM → 5. On the other hand, MP deciphers above DW problem; however, it simply overfits the training set and causes the lack-ofgeneralization (LoG) problem.

The stochastic pooling (SP) was introduced to conquer the DW, overfitting, and LoG problems caused by MP and AP. Instead of computing the average or the max, the output of the SP z SP is calculated via sampling from a multinomial distribution generated from the activations of each region ϕ. Three steps of SP are depicted below:

(1) Estimate the probability θ i, j ∈ of each entry ϕ i, j , i, j 1, 2 within the region ϕ.

in which, (i, j) is the element index of region ϕ. In matrix format, equation (17a) can be rewritten as

(2) Select a location β within ϕ in accordance with the probability θ i, j .

β ∼ Prob θ 1,1 , θ 1,2 , θ 2,1 , θ 2,2 .

(3) The output is the value at location β. (21b)

Using the probability map, we randomly select the position β (2, 2) associates with probability of θ 2, 2 (φ) 0.45. Thus, the SP output ofφ is z SṖ ϕ φ β φ (2, 2) 9. In all, SP uses non-maximal activations from the region ϕ, instead of outputting the greatest value.

The motivation of batch normalization transform (BNT) is the so-called internal covariant shift (ICS), which means the effect of randomness of the distribution of inputs to internal DCNN layers during training. The phenomenon of ICS will worsen the DCNN's performance.

This study introduced BNT to normalize those internal layer's inputs A {a i } over every mini-batch (suppose its size is m), in order to guarantee the batch normalized output 

The empirical mean μ and empirical variance σ 2 over training set A can be calculated as

The input a i ∈ A was first normalized to (25) where in denominator in Eq. (25) is to enhance the numerical stability. The value of is a small constant. 10 −5 in this study. Now has zero-mean and unit-variance characteristics. In order to have a more expressive deep neural network [26] , a transformation is usually carried out as (26) where the parameters C and D are two learnable parameters during training. The transformed output b i ∈ B is then passed to the next layer and the normalized remains internal to current layer.

In the inference stage, we do not have mini-batch anymore. So instead of calculating empirical mean and empirical variance, we will calculate population mean μ and population variance σ 2 , and we have the output b i at inference stage as

We proposed to use convolution block (CB) to be one of the building blocks of our DCNN. The CB consists of one conv layer and one batch normalization layer.

In traditional DCNN, the fully connected layer (FCL) serves the role of classifier. We plan to replace FCL with fully connected block (FCB), which will include one dropout layer (DOL) and one FCL layer. Srivastava et al. [27] proposed the concept of dropout neurons (DON) and DOL by randomly drop neurons and set to zero their neighboring weights s from the DCNN during training.

The neuron's incoming and outgoing connections are freezing, after it is dropped out. Figure 6 illustrates the illustration of neurons in DOL. The selections of dropout are random with a retention probability (θ r p ). 

where θ r p 0.5, ands means the weights of dropped out neurons.

During inference, we run the entire DCNN without dropout, but the weights of FCLs of FCBs are downscaled (viz., multiplied) by θ r p . Figure 7 shows a toy DCNN example with four FCL layers. Suppose we have N (k) neurons at k-th layer, and assume N (1) 12, N (2) 10, N (3) 8, N (4) 4. Thus, we have in total 4 k 1 N (k) 34 nodes. Suppose we do not consider incoming and outgoing weights, and do not consider the number of biases, the size of learnable weights S b (i, j) as number of weights between layer i and layer j before dropout, roughly calculating, can be written as S b (1, 2) 12 × 10 120, S b (2, 3) 10 × 8 80, S b (3, 4) 8 × 4 32. In total, we have the total number of learnable weights before dropout as S b

232. Using θ r p 0.5, the size of learnable weights after dropout between layer i and layer j is symbolized as S a (i, j), and we can calculate the total number of learnable weights as S a (3, 4) 30 + 20 + 8 58.

The compression ratio of learnable weights (CRLW), roughly, can be calculated by 58/232 0.25, which is the squared value of retention probability θ r p .

where S a and S b means the number of learnable weights after and before dropout, respectively.

We create a new five-layer DCNN with stochastic pooling for COVID-19 detection (5L-DCNN-SP-C) with three CBs and two FCBs. The structure of proposed 5L-DCNN-SP-C is shown in Fig. 8 , where SP is added after each activation map. The reason why set three CBs and two FCBs are by manual trial-and-error method. In the experiment, we will compare this setting (3 CBs + 2 FCBs) against other setting. The hyperparameters of each layer/block of proposed 5L-DCNN-SP-C are listed in Table 4 , where (αβ × β/γ ) means α filters with size of β × β, followed by pooling layer with pooling size of γ . Meanwhile, W and B represent the size of weight matrix and bias vector, respectively. The last column in Table 4 shows the activation map (AM).

Ten runs of tenfold cross-validation were employed. Suppose confusion matrix C is defined as

where (c 11 , c 12 , c 21 , c 22 ) represent TP, FN, FP, and TN, respectively. k is the index of trial (in each trial, onefold was used as test, and all the other folds were used as training), and r is the index of run. Note that C will be calculated based on each test fold, and summarized across all 10 trials. Then, we get the C(r ) (32) Now we can calculate six indicators η(r ) based on the confusion matrix over r-th run C(r ). η 2 (r ) , . . . , η 6 (r )] where η 1 is sensitivity, η 2 is specificity, η 3 is precision, and η 4 is accuracy. Ignoring variable r, we have:

η 3 c 11 c 11 + c 21 (34c) η 4 c 11 + c 22 c 11 + c 12 + c 21 + c 22 (34d) η 5 is F1 score.

and η 6 is Matthews correlation coefficient (MCC) (36) Fig. 9 Error bar of different pooling methods

The mean and standard deviation (SD) of all six measures η will be calculated over all ten runs.

where 1 ≤ m ≤ 6 represents the index of measures.

The results of 10 runs η of SP were compared against AP and MP. We compared three pooling methods on test set. The results of all three pooling methods are listed in 

We set the number of CBs as γ CB and the number of FCB as γ FCB . We set γ CB 3 and γ FCB 2 by trial-and-error method. Suppose we all use SP, and we create five different structure configurations (SC) setting as in Table 6 . The results of cognate performances on test set η are shown in Table 7 , where we can observe the SC-V performs the best results, which corresponds to our optimal SC setting: γ CB 3 and γ FCB 2

We compare our method "5L-DCNN-SP-C" with other COVID-19 classification approaches: RBFNN [6] , K-ELM [7] , ELM-BA [8] , 6L-CNN-F [9] , GoogLeNet [10] , ResNet-18 [11] . The results η on ten runs over test set are presented in Table 8 . It is easily observed that our proposed 5L-DCNN-SP-C smashes all the other six comparison baseline methods in all indicators. Particularly, 6L-CNN-F [9] also used convolutional neural network method, and they used more layers (6 layers) than layers used in our model (5 layers).

The reason why our five-layer model is better than that six-layer model [9] is threefold: (i) We choose SP to improve the performance of our deep learning model; (ii) We fine-tune the hyperparameters (such as γ CB , γ FCB , number of filters at each CB, number of neurons at each FCB); (iii) Our model was particularly designed for detecting COVID-19, while the 6L-CNN-F [9] was designed for fingerspelling recognition. In the future, we shall try to use clustering techniques [28, 29] to help improve the performance. Figure 10 shows the comparison bar plot of all seven methods.

This study proposed a novel 5L-DCNN-SP-C framework, that combines deep convolutional neural network and stochastic pooling for COVID-19 diagnosis. We added batch normalization transform and dropout layers, and proposed two new blocks (convolution block and fully connected 10 Comparison to state-of-the-art approaches block). In our test, we proved three CBs and two FCBs structure can give the best performance. There are several shortcomings of our method: (i) The dataset is somewhat small. We shall seek to collect more datasets. (ii) Some new network technologies would be tried in our future studies, such as the recent transfer learning pretrained models.

A review on COVID-19: origin, spread, symptoms, treatment, and prevention

Development of a protective device for RT-PCR testing SARS-CoV-2 in COVID-19 patients

Multiple assays in a real-time RT-PCR SARS-CoV-2 panel can mitigate the risk of loss of sensitivity by new genomic variants during the COVID-19 outbreak

Correlation of chest CT and RT-PCR testing in coronavirus disease 2019 (COVID-19) in China: a report of 1014 cases

Coronavirus disease 2019 (COVID-19): role of chest CT in diagnosis and management

A pathological brain detection system based on radial basis function neural network

A pathological brain detection system based on kernel based ELM

A pathological brain detection system based on extreme learning machine optimized by bat algorithm

Chinese sign language fingerspelling recognition via sixlayer convolutional neural network with leaky rectified linear units for therapy and rehabilitation

Going deeper with convolutions

Abnormality diagnosis in mammograms by transfer learning based on ResNet18

Role of biological data mining and machine learning techniques in detecting and diagnosing the novel coronavirus (COVID-19): a systematic review

Coronavirus Disease (COVID-19): A Machine Learning Bibliometric Analysis

Tea category identification using computer vision and generalized eigenvalue proximal SVM

Multiple sclerosis detection based on biorthogonal wavelet transform, RBF kernel principal component analysis, and logistic regression

Multi-objective path finding in stochastic networks using a biogeography-based optimization method

Application of stationary wavelet entropy in pathological brain detection

RGB-D image-based detection of stairs, pedestrian crosswalks and traffic signs

Preliminary CT findings of coronavirus disease 2019 (COVID-19)

An automated residual exemplar local binary pattern and iterative ReliefF based COVID-19 detection method using chest X-ray image

Classification of cerebral microbleeds based on fullyoptimized convolutional neural network

Sensorineural hearing loss identification via ninelayer convolutional neural network with batch normalization and dropout

Fruit category classification via an eight-layer convolutional neural network with parametric rectified linear unit and dropout technique

Classification of Alzheimer's disease based on eightlayer convolutional neural network with leaky rectified linear unit and max pooling

Cerebral micro-bleed detection based on the convolution neural network with rank based average pooling

Dropout versus batch normalization: an empirical study of their impact to deep learning

Dropout: a simple way to prevent neural networks from overfitting

KNN-BLOCK DBSCAN: fast clustering for large-scale data

Fast density peak clustering for large scale data based on kNN

Publisher's Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.Prof. Yu