key: cord-0185626-m90jdogp
authors: He, Kelei; Zhao, Wei; Xie, Xingzhi; Ji, Wen; Liu, Mingxia; Tang, Zhenyu; Shi, Feng; Gao, Yang; Liu, Jun; Zhang, Junfeng; Shen, Dinggang
title: Synergistic Learning of Lung Lobe Segmentation and Hierarchical Multi-Instance Classification for Automated Severity Assessment of COVID-19 in CT Images
date: 2020-05-08
journal: nan
DOI: nan
sha: 9ab6ec8904663d9022d8b23471007b7cd5320d3a
doc_id: 185626
cord_uid: m90jdogp

Understanding chest CT imaging of the coronavirus disease 2019 (COVID-19) will help detect infections early and assess the disease progression. Especially, automated severity assessment of COVID-19 in CT images plays an essential role in identifying cases that are in great need of intensive clinical care. However, it is often challenging to accurately assess the severity of this disease in CT images, due to variable infection regions in the lungs, similar imaging biomarkers, and large inter-case variations. To this end, we propose a synergistic learning framework for automated severity assessment of COVID-19 in 3D CT images, by jointly performing lung lobe segmentation and multi-instance classification. Considering that only a few infection regions in a CT image are related to the severity assessment, we first represent each input image by a bag that contains a set of 2D image patches (with each cropped from a specific slice). A multi-task multi-instance deep network (called M$^2$UNet) is then developed to assess the severity of COVID-19 patients and also segment the lung lobe simultaneously. Our M$^2$UNet consists of a patch-level encoder, a segmentation sub-network for lung lobe segmentation, and a classification sub-network for severity assessment (with a unique hierarchical multi-instance learning strategy). Here, the context information provided by segmentation can be implicitly employed to improve the performance of severity assessment. Extensive experiments were performed on a real COVID-19 CT image dataset consisting of 666 chest CT images, with results suggesting the effectiveness of our proposed method compared to several state-of-the-art methods.

. Typical cases of two non-severe (left) and two severe (right) patients with COVID-19, where infections often occur in small regions of the lungs in CT images. The similar imaging biomarkers (e.g., ground glass opacities, mosaic sign, air bronchogram and interlobular septal thickening) of both cases (denoted by red boxes) make the non-severe and severe images difficult to distinguish.

the performance of severity assessment. Extensive experiments were performed on a real COVID-19 CT image dataset consisting of 666 chest CT images, with results suggesting the effectiveness of our proposed method compared to several state-of-the-art methods. golden standard for COVID-19 is the Reverse Transcription-Polymerase Chain Reaction (RT-PCR) test, the effectiveness of those imaging-base applications is limited. Moreover, approximately 80% of patients with COVID-19 have only mild to moderate symptoms [9] , while the remaining patients have severe symptoms. Based on previous studies [5] , [10] , the imaging-based characters of COVID-19 patients are distinct to related diseases, e.g., viral pneumonia. Therefore, the severity assessment of the disease is of high clinical value, which helps effectively allocate medical resources such as ventilator. Among various radiological examinations, chest CT imaging plays an essential role in fighting this infectious disease by helping early identify lung infections and assess the severity of the disease. Previous studies show that computed tomography (CT) has the ability to provide valuable information in the screening and diagnosis [10] . In this work, CT help the clinicians to evaluate the condition of the patients in advance, by which necessary measures or treatments could better proceed, especially for severe patients in time.

However, automatically assess the severity of COVID-19 in CT images is a very challenging task. First, infections caused by COVID-19 often occur in small regions of the lungs and are difficult to identify in CT images, as shown in Fig. 1 . Second, imaging biomarkers of COVID-19 patients caused by an infection are similar in some severe and non-severe cases are similar, including ground-glass opacities (GGO), mosaic sign, air bronchogram, and interlobular septal thickening (Fig. 1 ). In addition, there are large inter-case variations in CT images of COVID-19 patients (Fig. 2) , because these images are usually acquired by multiple imaging centers with different scanners and different scanning parameters.

Several recent methods have been proposed for the diagnosis of COVID-19 [2] [3] [4] , with only some specifically designed for severity assessment of the disease. In several studies [5] [6] [7] , segmentation of lung or lung lobe is used as a prerequisite procedure for diagnosis purposes.

However, most of these methods treat the lung lobe segmentation and disease diagnosis as two separate tasks, ignoring their underlying association. Note that the segmentation of lung lobe can provide rich information regarding spatial locations and tissue types in CT images. Therefore, it is intuitively reasonable to jointly perform lung lobe segmentation and severity assessment/prediction, where the context information provided by segmentation results can be used to improve the prediction performance. The joint learning scheme is obviously faster than the two-stage framework, since detecting and cropping the lung field are not needed. Besides, as demonstrated by related works of class activation maps (CAMs) [11] , the classification task raises high signal responses in lung lobe area, where the infection patterns of lung lobe in disease progression could also provide useful guidance for the segmentation of lung lobes.

Moreover, most of the previous works are based on 2D image slices [2] [3] [4] . However, the annotation of 2D CT slices is a heavy workload for radiologists. It is interesting to directly employ 3D CT images for automated severity assessment of COVID-19, which is desired for real-world clinical applications.

To this end, in this paper, we propose a synergistic learning framework for automated severity assessment of COVID-19 in 3D CT images, by jointly performing severity assessment of COVID-19 patients and lung lobe segmentation. The proposed method extends the previous conference work [12] to deal with 3D images, and makes substantial methodological improvements for the task of COVID-19 severity assessment. Specifically, considering that only a few slices in CT images are related to severity assessment, each input CT image is represented by a bag of 2D image patches, each of which is randomly cropped from a specific slice. Furthermore, each slice is represented by a bag of infection regions represented by intermediate embedding features. With each bag as input, a multi-task multi-instance deep neural network (called M 2 UNet) is developed, including 1) a shared patchlevel encoder, 2) a segmentation sub-network for lung lobe segmentation, and 3) a classification sub-network for severity assessment of COVID-19 patients (i.e., severe or non-severe) using a hierarchical multi-instance learning strategy. Here, the segmentation results are used to provide context information of input CT images to boost the performance of severity assessment. Extensive experiments have been performed on a real-world COVID-19 dataset with 666 chest CT images, with the results demonstrating the effectiveness of the proposed method compared to several state-of-the-art methods.

The contributions of this work are three-fold:

• A multi-task multi-instance learning framework is proposed to jointly assess the severity of the COVID-19 patients and segment lung lobes in chest CT images, where the segmentation task provides context information to aid the task of severity assessment in chest CT image. • A unique hierarchical multi-instance learning strategy is developed to predict the severity of patients in a weakly supervised manner. • We evaluate the proposed method on a real clinical dataset with 666 3D CT images of COVID-19 patients, achieving promising results in severity assessment compared to several state-of-the-art methods. The rest of the paper is organized as follows. In Section 2, we introduce the related works for the segmentation and diagnosis of CT images of COVID-19 patients, as well as related studies on deep multi-instance learning. Then, we introduce the proposed method in Section 3. In Section 4, we present the materials, experimental setup, and experimental results. Finally, we conclude this paper and present several future research directions in Section 5.

In this section, we briefly review the most relevant studies from the following three aspects: 1) lung segmentation of CT images with COVID-19, 2) automated diagnosis of COVID-19 patients, and 3) deep multi-instance learning.

Segmentation of lung or lung lobe has been used as a common pre-requisite procedure for automatic diagnosis of COVID-19 based on chest CT images. Several deep learning methods have been proposed for the segmentation of lung in CT images with COVID-19. For instance, U-Net et al . [13] has been widely used for segmentation of both lung regions and lung lesions in COVID-19 applications [7] , [14] [15] [16] . Qi et al . [7] use U-Net to delineate the lesions in the lung and extract radiometric features of COVID-19 patients with the initial seeds given by a radiologist for predicting hospital stay. Also, several variants of U-Net have been applied to the diagnosis or severity assessment of COVID-19. Jin et al . [6] design a two-stage pipeline to screen COVID-19 in CT images, and they utilize U-Net++ [17] to detect the whole lung region and to separate lesions from lung regions. Besides, V-Net [18] is also used in various segmentation applications. Shan et al . [5] integrates human-in-the-loop strategy into the training process of VB-Net (a variant of V-Net). The human-aided strategy is an intuitive way to address the issue of lacking manual labels during segmentation in CT images.

Both X-rays [19] and CT images [10] can provide effective information for the computer-assisted diagnosis of COVID-19. Compared with X-rays, chest CT imaging contains hundreds of slices, which is clearer and more precise but has to take more time for specialists to diagnose. Therefore, there is a great demand to use CT images for automated diagnosis of COVID-19. In general, the existing methods for COVID-19 diagnosis based on CT images can be roughly divided into two categories: 1) classification; 2) severity assessment. In the former category, many studies have been conducted to determine whether patients are infected with COVID-19 disease. For example, Chen et al . [2] exploits a UNet++ based segmentation model to segment COVID-19 related lesions in chest CT images of 51 COVID-19 patients and 55 patients with other diseases, and finally determine the label (COVID-19 or non-COVID-19) of each image based on the segmented lesions. Ying et al . [3] propose a CT diagnosis system, namely DeepPneumonia, which is based on the ResNet50 model to identify patients with COVID-19 from bacteria pneumonia patients and healthy people. In the second category, Tang et al . [4] proposed to first adopt VB-Net to separate the lung into anatomical sub-regions, and then use these sub-regions to compute quantitative features for training a random forest (RF) model for COVID-19 severity assessment (with labels of being non-severe or severe).

The scenario of multi-instance learning (MIL) [20] [21] [22] or learning from weakly annotated data [23] arises when only a general statement of the category is given, but multiple instances can be observed. MIL aims to learn a model that can predict the label of a bag accurately, and many recent studies have focused on implementing MIL via deep neural networks. For instance, Oquab et al . [23] train a deep model with multiple image patches of multiple scales as input, and aggregate the prediction results of multiple inputs by using a max-pooling operation. Besides, many studies [24] [25] [26] propose to formulate image classification as a MIL problem so as to address the weakly supervised problem. Moreover, MIL is particularly suitable for problems with only a limited number (e.g., tens or hundreds) of training samples in various medical image-based applications, such as computer-assisted disease diagnosis [27] [28] [29] [30] . For instance, Yan et al . [29] propose a two-stage deep MIL method to find discriminative local anatomies, where the first-stage convolutional neural network (CNN) is learned in a MIL fashion to locate discriminative image patches and the second-stage CNN is boosted using those selected patches. More recently, a landmark-based deep MIL framework [30] CT image is first pre-processed, and multiple 2D image patches (with each patch from a specific slice) are then extracted to construct an instance bag for representing each input CT scan. This bag is then fed into the proposed multi-task multi-instance UNet (M 2 UNet) for joint lung lobe segmentation and severity assessment of COVID-19, consisting of a shared patch-level encoder, a segmentation sub-network, and a classification sub-network for severity assessment. Here, the segmentation task can provide location and tissue guidance for the task of severity assessment that employs a hierarchical multi-instance learning strategy.

Since there are only a limited number of cases at hand, it is desirable to employ the multi-instance learning strategy for severity assessment of COVID-19 patients in chest CT images.

The framework of the proposed method is illustrated in Fig. 3 , where the input is the raw 3D CT image and the output is the lung segmentation and severity assessment of COVID-19 patients (i.e., severe or non-severe). Specifically, each 3D CT image is processed via several image pre-processing steps. Then, a set of 2D image patches is randomly cropped from the processed image to construct an instance bag, and each bag represents a specific input CT image. This bag is regarded as the input of the proposed multi-task multi-instance U-Net (M 2 UNet). The M 2 UNet is designed to learn two tasks jointly, i.e., severity assessment of a COVID-19 patient and segmentation of the lung lobe.

As shown in Fig. 3 , in M 2 UNet, an encoding module is first used for patch-level feature extraction of 2D patches in each input bag, followed by two sub-networks for joint severity assessment and lung lobe segmentation. Specifically, in the classification sub-network, these extracted patch-level features are fed into a feature embedding module and an image-level feature learning module to capture the local-toglobal volume representation of the input CT image. With the learned volume features, a classification layer is finally used to assess the severity of each COVID-19 patient (i.e., severe or non-severe). In the segmentation sub-network, those patchlevel features are fed into a decoding module to perform lung lobe segmentation for each patch in the input bag. Since these two sub-networks are trained jointly with a shared patch-level encoder, the context information provided by the segmentation results can be implicitly employed to improve the performance of severity assessment.

To eliminate the effect of the background noise in each raw 3D CT image, we crop each scan to only keep the region containing the human body, by using a threshold-based processing method. Specifically, we first binarize the image using the threshold of zero, through which the human tissues and the gas regions will be separated. Then, the human body region is cropped according to the binary mask. Each body region image has a size of at least 256 × 256 for the axial plane in this work.

While image resampling is commonly used in many deep learning methods for segmentation and classification [31] , [32] , we do not resample the raw CT images in order to preserve their original data distributions. Since our method is clinical-oriented with inconsistent imaging qualities, CT images used in this study are not as clean as those in benchmark datasets [33] . For example, the physical spacing of our data has large variation, e.g., from 0.6 mm to 10 mm between slices, because of the use of different CT scanners and scanning parameters. Using a common interpolation method (e.g., tri-linear interpolation) to resample a CT image into 1 mm, one will introduce heavy artifacts to the image. Besides, only a few infection regions in each CT image are related to severity assessment. To this end, we employ the weakly supervised multi-instance learning (MIL) strategy for handing these inconsistent CT images. Specifically, for each preprocessed CT image, we randomly crop a set of 2D patches sampled from 2D slices (with each patch from a specific slice) in each image to construct an instance bag, and each bag is used to represent a specific CT image and treated as the input of the subsequent M 2 UNet. In this way, the inter-slice/patch relationships can be implicitly captured by our M 2 UNet. In addition, this MIL strategy represents each 3D image through a set of 2D image patches rather than sequential slices. This can partially alleviate the problem of data inconsistency, so our method has high practical value in real-world applications.

As shown in Fig. 3 , using each bag (consisting of a set of 2D image patches) as the input data, the proposed M 2 UNet first employs an encoding module for patch-level feature extraction. Based on these features, the classification and segmentation sub-networks are then used to jointly perform two tasks, respectively, i.e., 1) severity assessment of the patients, and 2) segmentation of lung lobes in each patch. Specifically, the classification sub-network uses a unique hierarchical MIL strategy to extract the local-to-global representation of each input image, with an embedding-level MIL module, an imagelevel MIL module, and a classification module. The segmentation sub-network contains a decoding module to segment lung lobes of 2D image patches in each bag.

The detailed network architecture is listed in Table I . The combination of the encoder and decoder is U-Net like, with four down-sampling blocks in the encoder and four upsampling blocks in the decoder. The outputs of the same level blocks in the encoder and decoder are concatenated and fed into the next block of the decoder. Limited by computational resources, all the convolutional layers in the encoder and decoder have the same number (i.e., 64) of kernels, except the last block in the encoder. The last block of encoder outputs 512 dimensional features to help build a more robust classification for severity assessment. The decoder outputs the corresponding segmentation mask of five types of lung lobes for each image patch.

While infection regions of the lung, related to COVID-19 (e.g., nodule and GGO) are usually located in regions of the CT image, the category of each CT image is labeled at the entire image level, rather than the region-level. That is, many regions are actually unrelated to the classification task for severity assessment.

Multi-instance learning (MIL) provides a useful tool to solve such a weakly supervised problem. Conventional MIL represents a 2D image as a bag, and each bag consists of multiple regions of the input image (i.e., instances). Their overall prediction is made at the bag-level by roughly two kinds of methods, i.e., the embedding-level methods and the instance-level methods. The former learns the relationship among instances by projecting them into a new embedding space. The latter directly generates the bag-level predictions by performing voting on the instance predictions. However, both methods are inapplicable for the classification of 3D images, as 3D images contain multiple 2D image slices, and the class labels are also related to some local regions of the slices. In this work, based on the previous study on pathological images [12] , we propose a hierarchical MIL strategy in the classification sub-network of our M 2 UNet to perform severity assessment of COVID-19 in 3D CT images, as shown in Fig. 3 . As mentioned in Section III-B, we represent each input 3D volumetric image as a bag consisting of a set of 2D image patches, and these patches are regarded as the instances in the MIL problem settings. Formally, we first construct a bag with n 2D patches cropped from the regional slices to represent each input CT image. Denote the i-th and the j-th 3D CT image as X i and X j , respectively, where X i = {φ ins i1 , φ ins i2 , · · · , φ ins ini } and X j = {φ ins j1 , φ ins j2 , · · · , φ ins jnj }. Here, φ ins kl ∈ R d (k = 1, 2, · · · , n k ) indicates the l-th instance of the k-th image. Then, these 2D patches (size: height × width) are fed into the encoding module for patch/instance-level feature extraction. These instance-level features are further fed into an embedding-level MIL module, which will be introduced later. After obtaining the instance-level features, the bag/image-level feature Φ i are then generated by our proposed global contrast pooling (GCP) layer in the image-level MIL module.

As illustrated in Fig. 4 , the proposed GCP layer aims to make the instance features closer to the relevant concepts, and also push those irrelevant instance features and concepts away from each other. In this work, the term 'concept' denotes the to-be-learned feature of GCP layer that is discriminative for severity assessment. Theoretically, the concept is a normalized weight to map features in instance feature space to an ordered embedding space. Specifically, in the GCP layer, we assume the bag-level feature Φ i is represented by the relationship between instance features and p concepts. Here, these concepts are learned to reveal the data structure in a global perspective. The bag-level feature is then denoted as a p dimensional feature vector, with each dimension denoting the maximum similarity between one concept and all instance features. We use the cosine function to measure such relationships. Thus, the bag feature and the similarity can be written as

where s im (m = 1, · · · , p) is the maximum similarity between the instance features of the i-th bag and the m-th image-level concept w m . R(·) denotes the commonly used regularization term used in deep networks. With Eqs. (1)-(2), one can observe that the proposed GCP layer can automatically learn the concepts that are related to those discriminative instances, thus reducing the influence of those irrelevant instances. Note such a GCP layer can be also used in other weakly supervised problems, where only a small portion of regions in an image are related to the task at hand (such as MRI-based brain disorder diagnosis [30] ).

We further use an embedding-level MIL module (with a GCP layer) to learn embedding-level representations, by regarding each image patch as a bag and the intermediate patch-level features produced by the encoder as instances. In this way, the relationships among small regions in each patch can be modeled by our method. Based on the embedding-level features, an image-level MIL module (with a GCP layer) is further used to generate the volume features. Based on the volume features, we use a fully-connected layer followed by a cross-entropy loss to predict the severity score (i.e., severe or non-severe) of each input CT image. The final loss function in the proposed hierarchical MIL network for severity assessment can be formulated as

where f c(·) denotes the mapping function of the fullyconnected layer, and y denotes the severity type confirmed by clinicians.

The segmentation task is supervised by the aggregation of cross-entropy loss and Dice loss as follows

wherep c n and l c n denote the predicted and ground-truth segmentation masks for the n-th patch in the c-th category. In this work, we segment c = 7 categories, including five parts of lung lobes and the background. It is worth noting that most of the cases in our dataset do not have groundtruth segmentation masks. For these cases, we simply avoid calculating the segmentation loss for them.

Finally, the losses in Eqs. 3 and 4 are trained simultaneously in a multi-task learning manner, and the overall loss of the proposed method is written as

where λ is the trade-off parameter used to balance the contributions of these two tasks. In this work, λ is empirically set to 0.01.

The proposed method is implemented based on the opensource deep learning library Pytorch. The training of the network is accelerated by four NVidia Tesla V100 GPUs (each with 32 GB memory). For feasible learning of the lung region images, we clamp the intensities of the image into [−1200, 0], which indicates that we use the width of 1200 and the level of −600 for the pulmonary window. Then, the data is normalized to the value of [0, 255], as used by other deep learning methods. The dataset is highly imbalanced as the number of severe patients is much fewer than the nonsevere patient. The ratio of the severe patient is less than 20% in our dataset. Therefore, we augmented the data by directly duplicated the severe cases in the training set. This can be done because the proposed method uses a random cropping strategy to construct the inputs. This makes the duplicated cases not the same to each other for the training of the network. We also use the random cropping strategy in the testing stage, by assuming that the data distribution is already well learned in training. Other cropping strategies, e.g., center cropping, may not be suitable here, as the center of pulmonary is dominated by the trachea and other tissues.

In both training and testing stage, we randomly crop 200 image patches from each input 3D CT image to construct the image-level bag (i.e., with the bag size of n = 200). And we use the output of the encoder to construct the embeddinglevel bag that contains 8 × 8 feature maps. We train M 2 UNet using the learning rate of 0.01 with a decay strategy of 'Poly' (with the power of 0.75). The network is optimized by a standard Stochastic Gradient Descent (SGD) algorithm with 100 epochs. And the weights are decayed by a rate of 1×10 −4 with the momentum of 0.9.

In this section, we first introduce the materials, competing methods, and experimental setup. We then present the experimental results achieved by our method and several state-ofthe-art methods. We finally investigate the influence of two major strategies used in our method. More results on parameter analysis can be found in the Supplementary Materials.

The real COVID-19 dataset contains a total of 666 3D chest CT scans acquired from 242 patients who are confirmed with COVID-19 (i.e., RT-PCR Test Positive). These CT images are collected from seven hospitals with a variety of CT scanners, including Philips (Ingenuity CT iDOSE4), GE (Bright speed S), Siemens (Somatom perspective), Hitachi (ECLOS), and Anke (ANATOM 16HD). The images are of large variation in terms of the image size of 512 × (512 ∼ 666) × (23 ∼ 732), and the spatial resolution of 0.586 ∼ 0.984 mm, 0.586 ∼ 0.984 mm and 0.399 ∼ 10 mm. Obviously, diagnosis based on these images is a very challenging task. The severity of the patient is confirmed by clinicians, following the guideline of 2019-nCoV (trail version 7) issued by the China National Health Commission. The severity of the patient is categorized into four types, i.e., mild, moderate, severe, and critical. In clinical practice, patients are often divided into two groups with different treatment regimens, i.e., severe and non-severe. The segmentation of 152 out of 666 images were delineated by an AI-based software and confirmed by experienced radiologists. In this work, we employ this partitioning strategy. That is, mild and moderate are treated as non-severe, while severe and critical are regarded as severe. Therefore, the task of severe assessment is formulated into a binary classification problem. Therefore, the dataset is partitioned into 51 severe and 191 non-severe patients.

We first compare the proposed M 2 UNet with four stateof-the-art methods in [4] , [34] [35] [36] for severity assessment of COVID-19 patients. The first two methods [4] , [36] are both based on hand-crafted features of CT images, while the last two [34] , [35] are deep learning-based methods that can learn imaging features automatically from data. Specifically, Tang et al . [4] first segment the lung, lung lobe and lesions in CT images. Then, the quantitative features of COVID-19 patients, e.g., the infection volume and the ratio of the whole lung, are calculated based on the segmentation results. The prediction is done by a random forest method. Yang et al . [36] propose to aggregate infection scores calculated on 20 lung regions for severity assessment. Besides, the ResNet50+Max method [34] is also compared for patch-wise classification. ResNet50+Max is a non-MIL method, which has a ResNet-50 network architecture and performs image-level classification through max-voting. In addition, we apply the Gated Att. MIL method proposed in [35] on our dataset, which is a one-stage MIL method with an attention mechanism. For fair comparison, this method shares the same multi-instance pool as our M 2 UNet.

We further compare our method with two state-of-the-art methods for lung lobe segmentation, including 1) UNet [13] , and 2) UNet++ [17] . The parameter settings for these five competing methods are the same as those in their respective papers.

To evaluate the influence of the proposed multi-task learning and hierarchical MIL strategies used in M 2 UNet, we further compare M 2 UNet with its two variants: 1) M 2 UNet with only the classification sub-network (denoted as Clas. Only), 2) M 2 UNet with only the segmentation sub-network (denoted as Seg. Only).

A five-fold cross-validation (CV) strategy is used in the experiments for performance evaluation. Specifically, the whole dataset is first randomly partitioned into five subsets (with approximately equal sample size of subjects). We treat one subset as the testing set (20%), while the remaining four subsets are combined to construct the training set (70%) and validation set (10%). The validation set here is used for selecting the hyper-parameters. This process is iterated until each subsets serve as a testing set once. The final results are reported on the test set. Two tasks are included in the proposed method, i.e., classification of severity assessment, and 2) segmentation of the lung lobe. For performance evaluation, two sets of metrics are used in these two tasks, with the details given below.

1) Metrics for Classification: We use five commonly used metrics to evaluate the classification performance achieved by different methods in the severity assessment task, i.e., Accuracy, Precision, Recall, F1 Score, and the area under the receiver operating characteristic curve (AUC).

P recision = T P T P + F P , Recall = T P T P + F N ,

where TP, TN, FP and FN denote true positive, true negative, false positive and false negative, respectively.

2) Metrics for Segmentation: We use three metrics, i.e., Dice Similarity Coefficient (DSC), Positive Predict Value (PPV) and Sensitivity (SEN), to evaluate the segmentation performance of different methods, with the definitions given below.

where V gt and V seg represent the ground-truth and predicted segmentation maps for each scan, respectively. D. Comparison with State-of-the-art Methods 1) Results of Severity Assessment: We first report the performance of six different methods in the task of severity assessment for COVID-19 patients, with the results shown in Table II . Note that the results from the competing methods are directly obtained from the respective papers. As can be seen, four deep learning-based methods usually outperform two hand-crafted feature-based methods in most cases. For some specific metrics, the method in [4] achieves the Recall of 0.933, which is significantly better than the non-MIL method ResNet50+Max. The conventional MIL-based method in [35] gets a performance improvement in terms of accuracy by 8%. Three MIL methods (i.e., [35] , Cls. Only, and M 2 UNet) yield satisfying performance, and the proposed M 2 UNet achieves the best results (e.g.., the accuracy of 98.5% and F1 Score of 99.1%). However, the proposed method with multiple instances in multi-instance learning achieves the accuracy of 98.5% and F1 Score of 99.1%. The receiver operating characteristic (ROC) curves of six competing methods are illustrated in Fig. 6 . Note that this ROC curve is plotted based on the results on one fold testing data, which is slightly different from the average performance on five-folds in Table II. Table II and Fig. 6 clearly suggest that our M 2 UNet generates the overall best performance in the task of severity assessment of COVID-19 patients based on chest CT images.

2) Results of Lung Lobe Segmentation: We then report the results of lung lobe segmentation achieved by four different methods in Table III . Comparing Seg. Only and the conventional U-Net, the former dramatically reduces the parameter from 131.71MB to 14.37MB. As a consequence, the performance in terms of DSC and PPV is also decreased by 1.7% and 4.9%, respectively. By using multi-task learning, M 2 Net improves the performance, from 0.759 to 0.785 in terms of DSC, which also outperform the performance of conventional U-Net, with a decreasing of parameters, from 131.71 to 15.32. The proposed M 2 UNet also achieves a slightly higher performance compared with U-Net++.

The visualization of segmentation results achieved by three different methods on two subjects is shown in Fig. 5 . From this figure, we can see that M 2 UNet generates the overall best segmentation masks, while U-Net and U-Net++ usually yield under-segmentation results on these cases. These results further show the advantage of our M 2 UNet.

We further evaluate the influence of two major strategies used in our M 2 UNet, i.e., 1) the hierarchical MIL strategy for classification, and 2) the multi-task learning strategy for joint severity assessment and lung lobe segmentation.

1) Influence of Hierarchical MIL Strategy: To evaluate the effectiveness of the hierarchical MIL strategy, we compare the variant of the proposed M 2 UNet (i.e., Cls. Only without the segmentation subnetwork) with a non-MIL method (i.e., ResNet50+Max) and a one-stage MIL method (i.e., Gated Att. MIL [35] ). The classification results of these three methods in the task of severity assessment are reported in Table IV . As shown in Table IV , two MIL methods (i.e., Gated Att. MIL 2) Influence of Multi-task Learning Strategy: Our M 2 UNet can jointly learn the segmentation task and the classification task in a multi-task learning manner. Here, we also investigate the influence of such a multi-task learning paradigm, by comparing M 2 UNet with its two single-task variants, i.e., 'Cls. Only' for classification and 'Seg. Only' for segmentation. The performance comparison in two tasks for severity assessment and lung lobe segmentation are reported in Table V and  Table VII , respectively. Table V suggests that, compared with Cls. Only, the multi-task learning paradigm used in M 2 UNet helps to improve the classification accuracy by 1.6%, while increasing the precision score by over 5% and the F1 Score by 2.5%. Notably, the F1 and precision of the Cls. Only method are already higher than 90%, which are hard to be improved. This is more valuable in this classification scheme, as the F1 score is more representative in evaluating such an imbalanced classification task.

As can be observed from Table VII , although M 2 UNet is not specifically designed for lung lobe segmentation, it still improves the overall segmentation performance in terms of three metrics, compared with its single-task variant (i.e., Seg. Only). This implies that the proposed multi-task learning strategy is useful in boosting the learning performance of both tasks of severity assessment and lung lobe segmentation.

We further investigate the performance of our method using different size of bags, and the results are shown in Table VI . Specifically, we vary the bag size within {50, 80, 100, 150, 200}. As shown in the table, the performance of M 2 UNet for classification changes along with the bag size. The proposed method achieves the best performance with the bag size of 200. Another observation is that, the table suggests the performance of the proposed AI-based severity assessment model is not sensitive with the bag size when larger than 100, indicating that at least 100 patches are required for the proposed method to achieve an acceptable result.

V. CONCLUSION AND FUTURE WORK In this paper, we propose a synergistic learning framework for automated severity assessment and lung segmentation of COVID-19 in 3D chest CT images. In this framework, we first represent each input image by a bag to deal with the challenging problem that the severity is related to local infected regions in the CT image. We further develop a multitask multi-instance deep network (called M 2 UNet) to assess the severity of COVID-19 patients and segment the lung lobe simultaneously, where the context information provided by segmentation can be employed to boost the performance of severity assessment. A hierarchical multi-instance learning strategy is also proposed in M 2 UNet for severity assessment. Experimental results on a real COVID-19 CT image dataset demonstrate that our method achieves promising results in severity assessment of COVID-19 patients, compared with several state-of-the-art methods.

In the current work, the severity assessment of COVID-19 only relies on one time-point data, without considering longitudinal imaging biomarkers. It is interesting to perform a longitudinal study to investigate the progression of the disease, which is one of our future work. Since annotations for lung lobe in 3D CT images are usually tedious and error-prone, only a small number of subjects in our dataset have groundtruth segmentation. Therefore, it is highly desired to develop automated or even semi-automated image annotation methods, which will also be studied in the future.

COVID-19 CORONAVIRUS PANDEMIC

Deep learning-based model for detecting 2019 novel coronavirus pneumonia on high-resolution computed tomography: A prospective study

Deep learning enables accurate diagnosis of novel coronavirus (COVID-19) with CT images

Severity assessment of coronavirus disease 2019 (COVID-19) using quantitative features from chest CT images

Lung infection quantification of COVID-19 in CT images with deep learning

Ai-assisted CT imaging analysis for COVID-19 screening: Building and deploying a medical AI system in four weeks

Machine learning-based CT radiomics model for predicting hospital stay in patients with pneumonia associated with SARS-CoV-2 infection: A multicenter study

Review of artificial intelligence techniques in imaging data acquisition, segmentation and diagnosis for COVID-19

Estimates of the severity of coronavirus disease 2019: a model-based analysis

Chest ct for typical 2019-ncov pneumonia: relationship to negative rt-pcr testing

Learning deep features for discriminative localization

Midcn: A multiple instance deep convolutional network for image classification

U-net: Convolutional networks for biomedical image segmentation

Deep learning-based detection for COVID-19 from chest CT using weak label

Longitudinal assessment of COVID-19 using a deep learning-based quantitative CT pipeline: Illustration of two cases

Serial quantitative chest CT assessment of COVID-19: Deep-learning approach

Unet++: A nested u-net architecture for medical image segmentation

V-net: Fully convolutional neural networks for volumetric medical image segmentation

Frequency and distribution of chest radiographic findings in covid-19 positive patients

Solving the multiple instance problem with axis-parallel rectangles

Show, attend and tell: Neural image caption generation with visual attention

Joint classification and regression via deep multi-task multi-channel learning for Alzheimer's disease diagnosis

Weakly supervised object recognition with convolutional neural networks

Deep miml network

Multiple instance learning convolutional neural networks for object recognition

Deep multiple instance learning for image classification and auto-annotation

Multiple instance learning for classification of dementia in brain MRI

Weakly supervised histopathology cancer image segmentation and classification

Multi-instance deep learning: Discover discriminative local anatomies for bodypart recognition

Landmark-based deep multiinstance learning for brain disease diagnosis

Pelvic organ segmentation using distinctive curve guided fully convolutional networks

Hierarchical fully convolutional network for joint atrophy localization and Alzheimer's disease diagnosis using structural MRI

Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the BRATS challenge

Inception-v4, inception-resnet and the impact of residual connections on learning

Attention-based deep multiple instance learning

Chest CT severity score: An imaging tool for assessing severe COVID-19