key: cord-0949229-nf6knurp
authors: Xiong, Yibai; Ma, Yan; Ruan, Lianguo; Li, Dan; Lu, Cheng; Huang, Luqi
title: Comparing different machine learning techniques for predicting COVID-19 severity
date: 2022-02-17
journal: Infect Dis Poverty
DOI: 10.1186/s40249-022-00946-4
sha: 53d8d480246f79239ebc56e87c0384a8085ba514
doc_id: 949229
cord_uid: nf6knurp

BACKGROUND: Coronavirus disease 2019 (COVID-19) is still ongoing spreading globally, machine learning techniques were used in disease diagnosis and to predict treatment outcomes, which showed favorable performance. The present study aims to predict COVID-19 severity at admission by different machine learning techniques including random forest (RF), support vector machine (SVM), and logistic regression (LR). Feature importance to COVID-19 severity were further identified. METHODS: A retrospective design was adopted in the JinYinTan Hospital from January 26 to March 28, 2020, eighty-six demographic, clinical, and laboratory features were selected with LassoCV method, Spearman’s rank correlation, experts’ opinions, and literature evaluation. RF, SVM, and LR were performed to predict severe COVID-19, the performance of the models was compared by the area under curve (AUC). Additionally, feature importance to COVID-19 severity were analyzed by the best performance model. RESULTS: A total of 287 patients were enrolled with 36.6% severe cases and 63.4% non-severe cases. The median age was 60.0 years (interquartile range: 49.0–68.0 years). Three models were established using 23 features including 1 clinical, 1 chest computed tomography (CT) and 21 laboratory features. Among three models, RF yielded better overall performance with the highest AUC of 0.970 than SVM of 0.948 and LR of 0.928, RF also achieved a favorable sensitivity of 96.7%, specificity of 69.5%, and accuracy of 84.5%. SVM had sensitivity of 93.9%, specificity of 79.0%, and accuracy of 88.5%. LR also achieved a favorable sensitivity of 92.3%, specificity of 72.3%, and accuracy of 85.2%. Additionally, chest-CT had highest importance to illness severity, and the following features were neutrophil to lymphocyte ratio, lactate dehydrogenase, and D-dimer, respectively. CONCLUSIONS: Our results indicated that RF could be a useful predictive tool to identify patients with severe COVID-19, which may facilitate effective care and further optimize resources. GRAPHICAL ABSTRACT: [Image: see text] SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s40249-022-00946-4.

Coronavirus disease 2019 (COVID-19) has been a severe public health event and is still ongoing spreading globally [1, 2] . The main manifestations of COVID-19 include fever, fatigue, and dry cough, most COVID-19 cases are non-severe with favorable outcome [3, 4] . However, patients with severe COVID-19 tend to progress rapidly and experience respiratory failure, respiratory distress syndrome, and septic shock or even death within a short period of time [5] , with a high mortality of 53% [6] . Previous studies observed that male, aged over 65, smoking, elevated lactate dehydrogenase, elevated D-dimer, and chest imaging findings associated with COVID-19 severity [7] [8] [9] [10] . However, the studies didn't demonstrate indicators' contribution to COVID-19 severity on admission. Hence, it is essential to assess importance of features to disease severity on admission for earlier and more targeted care, further reduce severe disease and prioritize medical resource.

Machine learning as an effective and innovative tool has been reported applications in diabetes [11] , cardiovascular diseases [12] , cancer [13, 14] , sepsis [15] , and depression [16] , etc., and showed favorable performance. Moreover, machine learning was used to predict COVID-19 diagnosis and treatment outcome, etc. [17] [18] [19] [20] [21] . However, different machine learning techniques were used to predict COVID-19 severity have not well been reported. Our study aimed to compare different machine learning techniques including random forest (RF), support vector machine (SVM), and logistic regression (LR) to predict COVID-19 severity based on clinical, laboratory, and chest computed tomography (C-CT) features, further identify feature importance to COVID-19 severity. The results can provide reference for clinicians to diagnose and treat patients timely and effectively, and further optimize medical resources, especially in resource-limited areas.

A retrospective study was conducted in the JinYinTan Hospital from January 26 to March 28, 2020. Data of demographic, clinical, laboratory, C-CT features, and treatment outcome were reviewed from 287 patients of hospital information system, all cases were laboratoryconfirmed by real-time reverse transcriptase polymerase chain reaction assay.

In the present study, 30 features were remained after the initial 86 features were screened by LassoCV in the study. Subsequently, we carried out spearman's rank correlation

Comparing different machine learning techniques for predicƟng COVID-19 severity Feature importance ROC curves of RF, SVM, and LR The heat map of correlaƟon between features to further reduce confounding, features with correlation coefficients greater than 0.4 (r > 0.4)were excluded. For example, white blood cell (WBC) was correlated with neutrophil to lymphocyte ratio (NLR); red blood cell (RBC) and hematocrit (HCT) were both correlated with hemoglobin concentration (HB); albumin (ALB) is associated with cholinesterase (CHE), etc. Additionally, combined with expert opinions, and literature evaluation 7 features were excluded from 30 features, namely WBC, RBC, HCT, ALB, r-glutamyl transpeptidase(r-GT), CHE, creatine kinase (CK), then 23 features were remained ( Fig. 1) , correlation coefficient between features were less than 0.4 (Fig.2) , which were entered into models for further analysis.

Twenty-three features included chest computed tomography (C-CT), fever, malignant tumor(MT), heart rate (HR), systolic blood pressure (SBP), hemoglobin concentration (HB), neutrophil-to-lymphocyte ratio (NLR), red cell volume distribution width (RDW-CV), immature granulocyte (IMG), interleukin-6 (IL-6), erythrocyte sedimentation rate (ESR), indirect bilirubin (IBIL), alkaline phosphatas (ALP), cystatin C (CysC), lactate dehydrogenase (LDH), Ca 2+ , α-fucosidase (AFU), amylase (AMS), retinol-Binding protein (RBP), hypersensitive troponin (Hs-cTn), myoglobin (MB), prothrombin time (PT), D-dimer.

Both dichotomous features are containing precisely two distinct values in the form of 0 and 1, for the three classification features such as C-CT, 0, 1 and 2 as three categorical variables were used to represent no lesions, unilateral lesions, and bilateral lesions, respectively; continuous variables were expressed with unit standardization. The 23 features were processed by RF, SVM, and The heat map of correlation between features. Color indicates the value of the correlation coefficient (r). The color intensity is proportional to the correlation coefficient (r), with positive correlations (r > 0) shown and negative correlations (r < 0), lower intensive color indicates lower correlations, in the study, 23 features selected by machine learning techniques, each feature is weakly correlated with each other (r < 0.4). Twenty-three features including C-CT, fever, MT, HR, SBP, HB, NLR, RDW-CV, IMG, IL-6, ESR, IBIL, ALP CysC, LDH, Ca 2+ , AFU, AMS, RBP, Hs-cTn, MB, PT(s), D-dimer. C-CT chest computed tomography, MT malignant tumor, HR heart rate, SBP systolic blood pressure, NLR neutrophil-to-lymphocyte ratio, HB hemoglobin concentration, RDW-CV red cell volume distribution width, LDH lactate dehydrogenase, IBIL indirect bilirubin, PT prothrombin time, ESR erythrocyte sedimentation rate, AFU α-fucosidase, RBP retinol-Binding protein, IL-6 interleukin-6, Hs-cTn hypersensitive troponin, AMS amylase, CysC cystatin C, IMG immature granulocyte, ALP alkaline phosphatas, MB myoglobin LR modeling to compare the three models in predicting COVID-19 severity on admission.

RF is a classifier that contains multiple decision trees, and the output is determined by the voting results of multiple trees. We used a grid search method to determine the number of classifiers, maximum depth, and minimum number of branch samples in random forest modeling. SVM based on statistical learning theory, is a class of generalized linear classification technique that classifies data binary in a supervised learning manner, and correctly classifies samples by looking for decision boundaries. LR is a generalized linear regression analysis model that uses different kernel functions to estimate data conditions. Similar to random forest, we use grid search method to determine the selection of key parameters of SVM and LR, thereby reducing the training error in modeling.

COVID-19 severity was determined based on the Diagnosis and Treatment Protocol for COVID-19 (seventh edition) [22] . In our study, given the small sample of 287 patients analyzed from single center, therefore, COVID-19 cases were categorized as non-severe and severe cases. Mild and moderate cases were classified as non-severe cases, severe and critical were classified as severe cases. The outcome variable was coded as binary 0 or 1, and the predictive output variable was defined as severe type (coded as 1) and non-severe type (coded as 0).

Of three models, 10-fold cross-validation was used to determine best appropriate data set to further better accuracy on predictions. Firstly, we split the all data into 10 number of folds. Each model has been trained and tested 10 times. For every iteration we selected 1-fold as test data and rest all as training data and ensured that different sets of data were used for model training and testing each time. Finally, we summed up the score for every fold and took the average to get the overall score.

The model's performance was evaluated respectively to compare the sensitivity, specificity, accuracy and area under curve (AUC) of receiver operating characteristic (ROC) of different models for severe COVID-19. The cut-off value was determined by the point on ROC curve where the Youden index was highest.

In order to quantify the importance of clinical, laboratory, and C-CT features on the predicted outcomes, the best performance model was used to provide relative feature importance by computing the contribution of each input feature toward the predictive outcome.

Categorical and continuous variables were expressed as n (%) and median (interquartile range, IQR), respectively. The Mann Whitney U test, χ 2 test, or Fisher's exact test were used to compare differences between severe and non-severe cases, as appropriately. The LR, SVM, and RF models were all performed, and AUC were calculated by using python 3.8.3 (Python Software Foundation, Delaware, USA) and anaconda 4.10.1 (Red Hat, Raleigh, North Carolina, USA). Main python module was shown in Additional file 1.

A total of 287 patients enrolled in our study. The median age was 60.0 years (IQR: 49.0-68.0 years) with 63.0 years (IQR: 55.5-70.0 years) for severe cases and 57.0 years (IQR: 46.7-67.0 years) for non-severe cases. A higher proportion (56.4%) was male, and the proportion of severe and non-severe cases were 36.6% and 63.4%, respectively. The most common symptoms reported were cough (61.3%) and fever (30.7%), 23 features of COVID-19 patients in the cohort are demonstrated in Table 1 . 

We further analyzed the predictive performance of RF, SVM and LR with AUC, sensitivity, specificity, and accuracy ( Table 2, Fig. 3 ), in terms of predictive performance among the three models, we observed that the overall better performance by AUC of 0.970 were RF for predicting COVID-19 severity at admission compared to SVM with AUC of 0.948 and LR with AUC of 0.928. Moreover, RF also achieved a favorable sensitivity of 96.7%, specificity of 69.5%, and accuracy of 84.5%. SVM had sensitivity of 93.9%, specificity of 79.0%, and Table 1 Twenty-three features of COVID-19 patients in the cohort Data were presented median (IQR) or n (%). P values were estimated by comparing variables between severe and non-severe cases a "0" denotes" no lung lesions", "1" denotes "unilateral lung lesions", and "2" denotes "bilateral lung lesions"

MT Malignant tumor, HR Heart rate, SBP Systolic blood pressure, NLR Neutrophil-to-lymphocyte ratio, HB Hemoglobin concentration, RDW-CV Red cell volume distribution width, LDH Lactate dehydrogenase, IBIL Indirect bilirubin, PT Prothrombin time, ESR Erythrocyte sedimentation rate, AFU α-fucosidase, RBP Retinol-Binding protein, IL-6 Interleukin-6, Hs-cTn Hypersensitive troponin, AMS Amylase, CysC cystatin C, IMG Immature granulocyte, ALP Alkaline phosphatas, MB Myoglobin, C-CT Chest computed tomography 

We further analyzed the top 23 features ranked by relative importance to COVID-19 severity by using RF with best performance. C-CT had highest importance to COVID-19 severity, and the following were NLR, LDH count, and D-Dimer, respectively (Fig. 4 ).

Previous studies observed that RF, LR, SVM, gradient boosted decision tree, and neural network to predict COVID-19 [23] [24] [25] . Schwab et al. demonstrated that RF had highest predictive performance for predicting hospital admissions for COVID-19 patients, and critical care admissions for COVID-19 cases in terms of AUC compared to LR, SVM, neural network, and gradient boosting [25] . In the present study, we found that RF could predict COVID-19 severity with best performance compared to SVM and LR. The results highlight the possibility that high-performance of RF are able to predict disease severity, the findings can provide reference to monitor parameters effectively and further optimize medical resource.

Chest CT is a widely used imaging test, which can be useful for initial triage of suspected COVID-19 patients. Previous studies demonstrated that chest CT or X-rays findings were associated with COVID-19 severity based on machine learning [26] [27] [28] [29] [30] , our data observed that chest CT findings with highest importance scores was for diagnosis of severe COVID-19 on admission by RF model, which can help clinicians a fast diagnosis and severity assessment of COVID-19 even before molecular test results are available or laboratory testing is limited. Wang et al. reported that COVID-19 clinical severity could be assessed based on the proposed chest lung-lesion automated deep-learning model [26] , given deep learning was likely to be over-fitting for small sample, therefore, which was not used in the study. Following the C-CT ranked feature importance scores were NLR, LDH, D-Dimer, Ca 2+ , ESR, and AFU, respectively, were confirmed in previous studies [30] [31] [32] . Ponti et al. reported that NLR was closely in relationship with coagulation cascades in disseminated intravascular coagulation and acute respiratory distress syndrome in severe COVID-19 patients [33] . LDH and D-dimer are consistently reported among hospitalized severe COVID-19 patients [7] , while gradual development during the disease course is particularly related to disease worsening in adults and children [34, 35] . However, compared with previous researches, AFU and HB are rarely mentioned in COVID-19 severity. AFU is essential in viral and bacterial infections including the cell-cell adhesion processes that mediate inflammation [36, 37] . Studies illustrated that AFU may be potentially upregulated by chemokines in the later stages of inflammation, such as COVID-19 [37, 38] . We also found that AFU is a significant lab indicator to distinguish severe illness, which may be a possible therapeutic target in the future. In other studies, hemoglobin and red blood cell count were found lower in COVID-19 patients, however, with no significant difference between survivors and non-survivors [39] . COVID-19 patients might suffer from hyper-metabolic states, such as septic shock and multiple organ dysfunction syndromes, which needs an increasing capability of hemoglobin to support the peripheral tissue demands for oxygen [40] . The ability to meet the oxygen demands of the peripheral tissues caused by the COVID-19 infection becomes the leading role to decide the prognosis, according to recent evidence [41] .

Our study has several limitations. Firstly, the small sample size from a single study site is a major limitation of the present study, which may undermine reliability of our analysis. Secondly, the data was retrospective collected, some data were incomplete or missing (such as smoking and drinking, and educational level), which were not further analyzed in the study. Further large sample, multi-center studies should be conducted to confirm our results.

Our study indicates that RF may be a prioritization tool in predicting COVID-19 severity with favorable overall performance at admission. The prediction model can potentially assist clinicians to facilitate severe COVID-19 patients' identification promptly and apply treatment timely, further optimize potentially healthcare resources. 

COVID-19 pandemic in BRICS countries and its association with socio-economic and demographic characteristics, health vulnerability, resources, and policy response

• fast, convenient online submission • thorough peer review by experienced researchers in your field • rapid publication on acceptance • support for research data, including large and complex data types • gold Open Access which fosters wider collaboration and increased citations maximum visibility for your research: over 100M website views per year submit your research ? Choose BMC

Risk factors for developing severe COVID-19 in China: an analysis of disease surveillance data

Clinical course and risk factors for mortality of adult inpatients with COVID-19 in Wuhan, China: a retrospective cohort study

Characteristics of SARS-CoV-2 and COVID-19

Risk factors for severity and mortality in adult COVID-19 inpatients in Wuhan

Risk factors of critical & mortal COVID-19 cases: a systematic literature review and meta-analysis

A longitudinal cohort study using a modified child-pugh score to escalate respiratory support in COVID-19 patients

Factors defining the development of severe illness in patients with COVID-19: a retrospective study

Risk factors for adverse clinical outcomes with COVID-19 in China: a multicenter, retrospective, observational study

Use and performance of machine learning models for type 2 diabetes prediction in community settings: a systematic review and meta-analysis

Machine learning prediction in cardiovascular diseases: a metaanalysis

FOLFOX treatment response prediction in metastatic or recurrent colorectal cancer patients via machine learning algorithms

Radiomic and genomic machine learning method performance for prostate cancer diagnosis: systematic literature review

Machine learning for the prediction of sepsis: a systematic review and meta-analysis of diagnostic test accuracy

Applications of machine learning algorithms to predict therapeutic outcomes in depression: A meta-analysis and systematic review

Artificial intelligence algorithm for predicting mortality of patients with acute heart failure

Model-based prediction of critical illness in hospitalized patients with COVID-19

Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal

Development of a clinical decision support system for severity risk prediction and triage of COVID-19 patients at hospital admission: an international multicentre study

Using machine learning of clinical data to diagnose COVID-19: a systematic review and meta-analysis

Diagnosis and Treatment Protocol for Novel Coronavirus Pneumonia (Trial Version 7)

Machine learning-based CT radiomics method for predicting hospital stay in patients with pneumonia associated with SARS-CoV-2 infection: a multicenter study

Machine learning based early warning system enables accurate mortality risk prediction for COVID-19

Clinical predictive models for COVID-19: systematic study

A deep-learning pipeline for the diagnosis and discrimination of viral, non-viral and COVID-19 pneumonia from chest X-ray images

Using artificial intelligence to detect COVID-19 and community-acquired pneumonia based on pulmonary CT: evaluation of the diagnostic accuracy

Artificial intelligence-driven assessment of radiological images for COVID-19

Performance of a severity score on admission chest radiography in predicting clinical outcomes in hospitalized patients with coronavirus disease (COVID-19)

A multimodality machine learning approach to differentiate severe and nonsevere COVID-19: Model development and validation

Neutrophil-to-lymphocyte ratio predicts critical illness patients with 2019 coronavirus disease in the early stage

Risk factors for mortality due to ventilator-associated pneumonia in a Chinese hospital: a retrospective study

Biomarkers associated with COVID-19 disease progression

Hematological findings and complications of COVID-19

From SARS to COVID-19: what we have learned about children infected with COVID-19

Development of fucosyltransferase and fucosidase inhibitors

Development of a UHPLC-MS method for inhibitor screening against α-l-1,3-fucosidase

Leukocyte extravasation: an immunoregulatory role for alpha-l-fucosidase?

Anemia and iron metabolism in COVID-19: a systematic review and meta-analysis

Epidemiology and immune pathogenesis of viral sepsis

COVID-19 does not lead to a "typical" acute respiratory distress syndrome

We thank Prof. Songwang Wang and Yewu Zhang, statistic experts from China CDC, for statistics guidance. We also acknowledge the work and contribution of all participants in the study.

The online version contains supplementary material available at https:// doi. org/ 10. 1186/ s40249-022-00946-4.

Authors' contributions YB-X and YM drafted the manuscript. YM and DL participated in the design of the study and performed the statistical analysis. RL-G and the National Traditional Chinese Medicine Medical Team carried out the verification of data and the patients' enrollment. LC and LQ-H conceived of the study, and revised the manuscript. All authors read and approved the final manuscript. 

The datasets generated and/or analyzed during the present study are not publicly available due to sensitivity of the patients' personal information but are available from the corresponding author on reasonable request.

Ethics approval and consent to participate Ethics approval was granted by China Academy of Chinese Medical Sciences (N. CACMS-IRB2020-001-1).

Not applicable.

The authors declare that they have no competing interests.