key: cord-0820539-r7ez09va authors: Islam, M. T.; Fleischer, J. W. title: Distinguishing L and H phenotypes of COVID-19 using a single x-ray image date: 2020-05-03 journal: nan DOI: 10.1101/2020.04.27.20081984 sha: 0ed35755ab7f427bc62d83859af5a3415e9378b5 doc_id: 820539 cord_uid: r7ez09va Recent observations have shown that there are two types of COVID-19 response: an H phenotype with high lung elastance and weight, and an L phenotype with low measures. H-type patients have pneumonia-like thickening of the lungs and require ventilation to survive; L-type patients have clearer lungs that may be injured by mechanical assistance. As treatment protocols differ between the two types, and the number of ventilators is limited, it is vital to classify patients appropriately. To date, the only way to confirm phenotypes is through high-resolution computed tomography. Here, we identify L- and H-type patients from their frontal chest x-rays using feature-embedded machine learning. We then apply the categorization to multiple images from the same patient, extending it to detect and monitor disease progression and recovery. The results give an immediate criterion for coronavirus triage and provide a methodology for respiratory diseases beyond COVID-19. It is commonly thought that COVID-19 presents as other coronavirus diseases, with Acute Respiratory Distress Syndrome (ARDS) caused by pneumonia-like thickening of the lungs 4 . However, recent observations suggest that half of all patients present differently, with near-normal breathing mechanics, clearer lungs, and hypoxia caused by poor oxygen transfer to the capillaries 1,2,5 . The ARDS-like H phenotype is characterized by high lung elastance, high weight, and high recruitment 2 (potential to reexpand collapsed lung tissue, e.g. closed alveoli 6 ); these patients require mechanical ventilation to survive 7 . The corresponding measures for the L phenotype are low 2 ; applying high-pressure ventilation in these cases can injure lungs and precipitate a rapid worsening to ARDS 3, 5 . It is vital to classify patients appropriately. To date, the only way to confirm the phenotypes is through high-resolution computed tomography (CT, Fig. 1 ) 2 . This is inhibitory for rapid triage and impossible for hospitals with limited resources. Here, we introduce a machine-learned early detection system to identify COVID-19 and differentiate its L and H phenotypes using conventional chest x-rays. That this is possible comes from the origin of CT images: they are made from a collection of x-rays taken at different angles, each of which contains information on lung size, shape, and density. Resolution of CT images improves with the number of different views; higher resolution removes artifacts and provides cross-sections that allow visual separation of internal organs (Figs. 1a-c) . The significance of this can be seen by comparing CT scans with individual x-ray images (Figs. 1e-h): each x-ray picture is a 2D projection of the 3D body image, so that all internal features are superimposed. This overlap complicates diagnosis considerably, especially when disease forms around the heart and behind the breastbone. Nevertheless, frontal x-rays are the most frequent radiological test performed in intensive care units 8 and are used often at the initial point of care to diagnose respiratory distress, e.g. pneumonia and chronic obstructive pulmonary disease 9 . The images in Figs. 1d-h, taken from the publicly available COVIDx dataset 10 , show representative chest x-rays from normal, pneumonia-positive 11 , and COVID-19-positive patients 12 . Compared with a normal image, a pneumonia x-ray shows a diffuse pattern characteristic of fluid-filled lungs. Known as ground-glass opacities when translucent and consolidation when fully opaque, these regions form when an overstimulated immune system fills the alveoli with cells and fluid 13, 14 . Consequently, the location and density of these regions give many indications of the type and stage of disease. Similar patterns are seen in COVID-19 patients 15 , particularly as the illness progresses (Figs. 1b,d,f). Figure 2 shows the results of three different numerical approaches for embedding features from the xray images. Each reduces the high-dimensional image space to a 2D plot of the samples. All are unsupervised, in the sense that data labels are known but only provided afterward for visualization; they are not seen or used by the algorithms. Figure 2a shows the output from Principal Component Analysis (PCA), which is a linear technique that rearranges the input data to maximize variance. It preserves large distances between points, so that samples that are distinct from each other end up far apart. It gives three clusters of x-rays with significant overlap among the points. Projection along the main principal component (Fig. 2b) gives a Gaussian profile for COVID-19 patients, a direct result of the broad statistics of the underlying data. In contrast, Stochastic Neighbor Embedding (SNE) is a nonlinear mapping that emphasizes similarity between sample points, e.g. the joint probability of having features in common 16 . Figures 2c,d show the results of SNE using Student's t-test as a metric (Gaussian distributions for relative probability) 17 . Remarkably, the network organizes the COVID-19 patients into two distinct clusters: one that resembles pneumonia and one that has a more normal presentation. Unfortunately, the COVID image dataset 12 has no labels for the L and H phenotypes. However, most chest x-rays have radiologist and physician notes on the stage of disease and response to treatment. This metadata, discussed below and in the Supplementary Information, supports the labeling of t-SNE data according to phenotype. One problem with t-SNE clustering is that the mapping only gives information on similarity within a group, with no measure of dissimilarity between them. Neither the size of the clusters nor the distance between them has any physical meaning 18 . A more sophisticated clustering algorithm that overcomes these issues is Uniform Manifold Approximation and Projection (UMAP), which categorizes objects using Riemannian geometry and algebraic topology 19 . It is a k-neighbor graph-based algorithm in the same class as t-SNE and again gives two clusters (Figs. 2e,f). Compared with the t-SNE plot, the UMAP plot has tighter groupings and sharper boundaries between the x-ray types. As a consistency check, we use the L and H labels from the t-SNE clusters to color the UMAP (again, this is for visualization purposes only and . CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted May 3, 2020. . https://doi.org/10.1101/2020.04. 27.20081984 doi: medRxiv preprint was not used by the algorithm). This confirms that UMAP separates the same points as t-SNE, but with a better mapping. A provocative feature of both mappings is the isolation of the H phenotype from the pneumonia samples, indicating a unique signature of COVID-19 infection. Similarly, the placement of L-type patients between the reference ones indicates identifiable lung damage in the x-ray image, reconciling the simultaneous observations of normal breathing and low oxygenation levels 1, 5 . Future work will look for signs of susceptibility to ventilation-induced lung injury 3,20 (a diagnostic not currently available 20 ) and test the hypothesis that COVID-19 is a different class of coronavirus from those that cause MERS and SARS 21 . As UMAP gives quantitative distance metrics, and accommodates new data better than t-SNE 19 , we focus analysis on the UMAP mapping. As a first step, we observe that the groupings in Fig. 2e are nearly orthogonal; re-plotting the data in terms of these major axes gives the representation in Fig. 3a . The two clusters of data are bound by distinct ellipses, each containing 95% of their identified x-rays. The vertical L cluster is broader and more disperse than the horizontal H cluster, with external points on both sides of its major axis. From the patient metadata 10 , it was found that the isolated point in the upper left is from the youngest patient in the dataset. The outlying point in the upper right has a partner x-ray image inside the L ellipse, taken from the same patient at an earlier time. The "misclassified" point in the H cluster also has an earlier partner inside the L ellipse. The remaining external point, bisecting the ellipses, has ambiguous radiology and no clear marker suggesting outlier status. The algorithms treat each x-ray independently and have no knowledge that multiple images could belong to the same patient. When clustering images within a set, the two methods have different priorities: t-SNE emphasizes commonality, while UMAP takes into account both locally similar features and globally different ones. This difference explains the appearance of an "L" point in the H cluster and, combined with a distance metric, suggests that UMAP can provide a scale of disease severity. While the limited data prevents its development here, patient progression and recovery are captured: worsening condition appears as movement from left to right in parameter space (normal-to pneumonia-like), while improvement goes in the opposite direction. Significantly, this ordering is preserved when UMAP is applied to new test data as well. Details are given in the Supplementary Information. . CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 3, 2020. 4i,j). These distributions are consistent with observations of disease severity 25 and re-affirm the distinct epidemiology of COVID-19 from other viral pandemics, such as the 1918 flu 26 . With the current data, it would be beneficial to repeat the training for more specific regions of interest and to use a wider range of metrics, e.g. structural similarity 27 and image statistics 28 . Likewise, it would help to leverage an additional viewpoint, such as the lateral (side) x-ray images commonly taken at the same time as the frontal view. Lateral x-rays are particularly useful for assessing the air spaces behind the breastbone and heart 13 and provide an extreme form of limited-angle tomography 29 , giving much-needed perspective for volume estimation. Finally, it is important to correlate the x-ray images with data from other observations as well 30 , such as cardiovascular 24 and renal 31 metrics, the improvement of many patients via prone positioning (laying on the stomach) 32 , and acoustic signatures of wheezing, gasping, and coughing 33 . Far more data and clinical studies are needed, but the classification of patients should start now. . CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 3, 2020. . We used the publicly available COVIDx dataset 10 , which is a combination of COVID-19 image data collection 12 and the RSNA pneumonia dataset 11 . To date, the training set includes posteroanterior (PA) xray images from 7966 pneumonia-positive patients and 8066 "normal" patients (pneumonia-negative and COVID-19-negative patients) but only 98 images of COVID-19-positive patients. The test set consists of 100 normal, 100 pneumonia, and 10 COVID-19 patients. For the t-SNE and UMAP analysis, we employed 308 normal, 308 pneumonia, and 98 COVID-19 x-ray images. Images are resized such that the lowest dimension contains 256 pixels and then center-cropped for feature analysis. For fine-tuning the network, we augmented the data with random jitter in brightness, random crops, and horizontal and vertical flips. We use a DenseNet-121 architecture 34 trained on the ImageNet database 35 , performed using the PyTorch deep learning library 36 . The network uses Adam optimization 37 with an initial learning rate of 0.0001. It is then trained for a total of 10 epochs, with the learning rate decreasing by a factor of 0.1 every 3 epochs. For evaluation, we employ a five-crop criterion in which we take four corner crops and one center crop of the image and make decisions based on majority rule. The fine-tuned network showed a sensitivity of 80% for COVID-19 detection and 93% for pneumonia detection. Image features output from the pre-trained network are grouped using the t-SNE 17 and UMAP 19 clustering algorithms. To confirm consistency, the t-SNE perplexity value was varied from 20 to 100 and the number of iterations was varied from 1000 to 5000; for the final results, the perplexity value is set to 50 and the number of iterations is set to 1000. For the UMAP analysis, spread values of 1-3 were considered; for the final results, the value is set to 2. To obtain a consistent topology, the UMAP distance parameter is tuned to 0.1. The authors have no conflict of interest relevant to this article. is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 3, 2020. . CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 3, 2020. . . CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 3, 2020. Both t-SNE and UMAP reduce the high-dimensional dataset X to a low-dimensional space Y (typically, the target space is two-dimensional). In t-SNE, image pairs are assigned a joint probability and grouped by similarity 1 . The algorithm is non-parametric, i.e. it does not learn a mapping : → . Consequently 1,4 , 1) there may be no meaning to the placement of clusters; 2) there may be no meaning to the distance between clusters; and 3) the algorithm cannot simply add new data points to an existing plot; it must be re-run on the expanded set when new data points are added. The first two properties mean that multiple runs of t-SNE may result in multiple output geometries. In our case, we observe two different manifestations, depending on the (random) initial condition chosen to start the algorithm: half the time it places a COVID-19 cluster between those of pneumonia and normal x-rays (Fig. S1a) , half the time it places the COVID-19 cluster above the normal one (Fig. S1b) . The third property can be overcome by parameterizing t-SNE to accommodate new data 5 . A more modern algorithm better suited for generalization is UMAP 2 . This algorithm provides a distance metric between points (typically Riemannian) and gives a more consistent topology over a wide range of parameters and initial conditions (Figs. 1c,d) . Accordingly, we use UMAP for analysis and prediction. . CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 3, 2020. In the original UMAP mapping, there was an extra point labeled H that appeared in the ellipse for the L phenotype. This point, shown in red in Fig. S2a , later revealed itself as an error in the COVIDx dataset 3,6 : it was labeled as a frontal (PA) chest x-ray in the physician notes, when in fact it is a lateral (side) x-ray image (Fig. S2b) . We removed this data point in subsequent analysis. We present some representative samples from the UMAP plot given in the main text (Fig. 3a) . Shown are points within each cluster, at the boundaries, and in the outlying regions. Results for the L phenotype are given first, followed by those for the H phenotype. Under each x-ray image, we also give the available radiologist and physician notes 6 . Figure S3 . Typical x-ray image within the L cluster. . CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 3, 2020. . CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 3, 2020. . Figure S13 : Typical x-ray image within the H cluster. Figure S14: x-ray image on the boundary of the H cluster. The patient has lymphatic leukemia, which would put him in the high-risk H category regardless of imaging. Figure S15 : Mislabeled x-ray image in the H cluster, near the point in Fig. S14 . Label from t-SNE, placement from UMAP. As before, disease worsening progresses from left to right in parameter space. . CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 3, 2020. Covid-19 x-ray data is very limited and does not allow for significant predictive testing of the network. Nevertheless, we saved a small number of x-rays from the COVIDx dataset 6 to analyze performance on new patients, viz. 100 normal x-ray images, 100 pneumonia x-ray images, and 10 COVID-19 x-ray images. Results are shown in Figs. S18-22. As seen in previous applications of UMAP for phenotyping 7 , the new data points appear on the periphery of the previously learned mapping. As before, some patients have multiple chest x-rays, allowing comparison of different stages of illness. . CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted May 3, 2020. . S19. Progression of illness for H-type patient. As before, the x-ray becomes more pneumonialike as the condition worsens. . CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted May 3, 2020. . https://doi.org/10.1101/2020.04. 27.20081984 doi: medRxiv preprint Figure S20 . Slow progression of L-type patient. Figure S21 . New x-ray image placed near origin of L-H axes. Figure S22 . New x-ray image placed near origin of L-H axes. . CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted May 3, 2020. . https://doi.org/10.1101/2020.04. 27.20081984 doi: medRxiv preprint Covid-19 does not lead to "typical" acute respiratory distress syndrome COVID-19 pneumonia: different respiratory treatments for different phenotypes? Ventilator-induced lung injury: lessons from experimental studies Evaluation and Treatment Coronavirus (COVID-19). StatPearls [Internet Alveolar recruitment in acute lung injury Lung Recruitment in Patients with the Acute Respiratory Distress Syndrome Routine chest x-rays in intensive care units: a systematic review and meta-analysis Revisiting signs, strengths and weaknesses of Standard Chest Radiography in patients of Acute Dyspnea in the Emergency Department COVID-Net: A Tailored Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from RSNA pneumonia detection challenge Covid-19 Image Data Collection Imaging of diffuse lung disease Ground-glass opacity of the lung parenchyma: a guide to analysis with high-resolution CT Imaging Profile of the COVID-19 Infection: Radiologic Findings and Literature Review Proceedings of the 15th International Conference on Neural Information Processing Systems Visualizing data using t-SNE UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction Emerging concepts in ventilation-induced lung injury. F1000Res 9, F1000 Faculty Rev-1222 COVID-19, SARS and MERS: are they closely related? Sex determination from measurements of the sternum and fourth rib using multislice computed tomography of the chest Morphometric analysis of variation in the ribs with age and sex COVID-19 and the cardiovascular system Estimates of the severity of coronavirus disease 2019: a model-based analysis. The Lancet Infectious Diseases The great influenza : the story of the deadliest plague in history Image quality assessment: from error visibility to structural similarity When deep denoising meets iterative phase retrieval Flow-scanning optical tomography Clinical features of patients infected with 2019 novel coronavirus in Wuhan Kidney disease is associated with in-hospital death of patients with COVID-19 Does prone positioning improve oxygenation and reduce mortality in patients with acute respiratory distress syndrome? Cough Sound Analysis Can Rapidly Diagnose Childhood Pneumonia 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR ImageNet Large Scale Visual Recognition Challenge Advances in Neural Information Processing Systems A Method for Stochastic Optimization Visualizing data using t-SNE UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction COVID-Net: A Tailored Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from How to use t-SNE effectively Covid-19 Image Data Collection Visualization of Emergency Department Clinical Data for Interpretable Patient Phenotyping First case of Coronavirus Disease 2019 (COVID-19) pneumonia in Taiwan CC-BY-NC 4.0 International license It is made available under a The authors thank DARPA and the AFOSR for general support. This Supplementary Information analyzes the data in more detail. Part 1 discusses the dimensionreduction methods of t-SNE 1 and UMAP 2 . Part 2 describes the discovery by UMAP of a mislabeled xray image in the COVIDx dataset 3 . Part 3 provides a deeper discussion of existing points in the UMAP classification and Part 4 explores predictive testing by UMAP for new data points.