key: cord-1054361-g7wbhz3o authors: Naseem, M.; Arshad, H.; Hashimi, S. A.; Irfan, F.; Ahmed, F. S. title: Predicting mortality in SARS-COV-2 (COVID-19) positive patients in the inpatient setting using a Novel Deep Neural Network date: 2020-12-14 journal: nan DOI: 10.1101/2020.12.13.20247254 sha: 513dec303c3c587ae048679a6c25c3ff1d863c30 doc_id: 1054361 cord_uid: g7wbhz3o Background The second-wave of CVOID-19 pandemic is anticipated to be worse than the initial one and will strain the healthcare systems even more during the winter months. Our aim was to develop a machine learning-based model to predict mortality using the Neo-V framework. We hypothesized this novel machine learning approach could be applied to COVID-19 patients to predict mortality successfully and high accuracy. Methods The current Deep-Neo-V model is built on our previously statistically rigorous machine learning framework [Fahad-Liaqat-Ahmad Intensive Machine (FLAIM) framework] that evaluates the statistically significant risk factors, generate new combined variables and then supply these risk factors to deep neural network to predict mortality in RT-PCR positive COVID-19 patients in the inpatient setting. We analyzed adult patients ([≥]18 years) admitted to the Aga Khan University Hospital with a working diagnosis of COVID-19 infection (n=1228). We excluded patients that were negative on COVID-19 on RT-PCR, had incomplete or missing health records. The first phase selection of risk factor was done using Cox-regression univariate and multivariate analyses. In the second phase, we generated new variables and tested those statistically significant for mortality and in the third and final phase we applied deep neural networks and other traditional machine learning models like Decision Tree Model, k-nearest neighbor models and others. Results A total of 1228 cases were diagnosed as a COVID-19 infection, we excluded 14 patients after the exclusion criteria and (n=)1214 patients were analyzed. We observed that several clinical and laboratory-based variables were statistically significant for both univariate and multivariate analyses while others were not. With most significant being septic shock (hazard ratio [HR], 4.30; 95% confidence interval [CI], 2.91-6.37), supportive treatment (HR, 3.51; 95% CI, 2.01-6.14), abnormal international normalized ratio (INR) (HR, 3.24; 95% CI, 2.28-4.63), admission to the intensive care unit (ICU) (HR, 3.24; 95% CI, 2.22-4.74), treatment with invasive ventilation (HR, 3.21; 95% CI, 2.15-4.79) and laboratory lymphocytic derangement (HR, 2.79; 95% CI, 1.6-4.86). Machine learning results showed our DNN (Neo-V) model outperformed all conventional models (Neo-V) and Deep-FLAIM models with test set accuracy of 99.53%, sensitivity of 89.87%, and specificity of 95.63%; positive predictive value, 50.00%; negative predictive value, 91.05%; and area under the curve of the receiver-operator curve of 88.5. Conclusion Our novel Deep-Neo-V model outperformed all other machine learning models. The model is easy to implement, user friendly and with high accuracy. Severe acute respiratory syndrome coronavirus 2 (SARS-COV 2) has caused 60 million infections and 1.4 million deaths worldwide (1) and 800, 000 deaths in the United States (2) . Despite strict measures being deployed and special instructions given to the mass public, the second wave is anticipated to be far worse than the first one (3) . Some countries have started to ease those earlier restrictions because of economic implications from the initial lockdown, which may create a further deepening of the current crisis, as cases continue to rise. This could overwhelm the already strained healthcare systems across the United States and the world. Machine learning has been extensively used in the automotive, defense and fin-tech industry over the past couple of years with great success. The use of these systems to predict health outcomes have been limited. Epidemic Renormalization Group (eRG) has used a machine learning framework to predict the time evolution of the first and second wave based on the data from the first wave in Europe (4) . We have previously developed algorithms that predict mortality in the clinical setting and performed better than most clinical scales utilized currently to predict mortality (5-7) During the current COVID-19 pandemic crisis, the aim was to develop a mortality prediction tool that can predict death in COVID-19 patients at admission. This would help the already strained healthcare systems and physicians around the world in crucial clinical decision making, resource management and family-counselling. We hypothesize that machine learning, specifically deep-learning could be applied to COVID-19 patients with high accuracy. Using deep-learning to predict mortality in these patients may assist in clinical decision making, risk stratification and planning strategies in future for such pandemics at a larger scale. Not much work has been done in mortality prediction in COVID- 19 patients in with lower socio-economic countries (LMIC) or developed countries. Our hypothesis is that machine learning could be applied to COVID-19 patients with high accuracy. Hence, predicting mortality and clinical outcomes using ML algorithms may assist in clinical decision making, risk stratification and planning strategies in future for such pandemics at a larger scale. Clinical data was conducted at the Aga Khan University Hospital (AKUH). All patients' records were completely anonymous, and the data collected has received Institutional Review Board/Ethical Review Committee (IRB/ERC) approval from Aga Khan University Hospital (AKUH), Pakistan. The dataset was de-identified and our study complied with the ethical principles recommended by Helsinki declaration (1964) and its amendments. We retrospectively collected data from AKUH -electronic medical record (EMR) that were admitted with a primary diagnosis of COVID-19 infection to the hospital between February 2020 and September 2020. We included adult patients (>18 years of age) that were admitted to the hospital with a diagnosis of COVID-19 or were tested positive during their admission on reverse transcriptase polymerase chain reaction (RT-PCR) based on Center for Disease Control and Prevention (CDC) and College of American Pathologist (CAP) guidelines (8, 9) . Data was collected on demographics, and comorbidities at admission, the first 24-hours of laboratory investigations (hematological and blood biochemistry, ( Table 1. ), imaging and complete clinical characteristics (history, examination, treatment, hospital course and outcomes). We excluded all patients that had RT-PCR negative tests for COVID-19 and incomplete records or inaccurate medical record information. Neo-V is a tri-phase bio-statically rigorous machine learning that builds on our previous framework that had better accuracy then the currently used clinical scoring systems in predicting mortality in the intensive care unit (ICU) patients (5, 6) . Phase I: Also known as the statistical-phase; in which data was analyzed by univariate and multivariate Cox-regression analysis (X) using IBM SPSS (version 24.0.0.0) (X) for outcome assessment with hazard ratio and confidence intervals. A p-value of <0.05 was considered statistically significant. We also did demographic data frequency analysis. Statistical analysis was carried out on all the variables included in Table 1 . Phase II: In contrast to our previously published model we created new variables for the existing dataset called neo-variables. These variables included a combination of two clinically relevant labs that were significant in both the univariate and the multivariate analysis (Table 1. ). These variables also underwent univariate and multivariate analysis for outcome assessment with hazard ratio and confidence intervals Phase III: Biological datasets are highly imbalanced with respect to the outcomes (i.e. more people were survived, then those who didn't) and machine learning models are very sensitive to imbalanced data and can produce variable and non-reproducible results. To address this, we optimized the dataset using Synthetic Minority Over-sampling Technique (SMOTE) algorithm during the training process (10) . In the machine learning phase, we used all variables that were statistically significant in phase I and II in both the univariate and multivariate analysis (non-significant risk factors were excluded). The final dataset was randomized and divided into a training and testing set with a 70/30 percent split respectively (30% data left out to test the models). After partitioning the data, we allocated feature vectors of the training instances by X_train with corresponding outcome label as Y_train. Similarly, for the test set we allocated X_test and Y_test as testing vector instances and corresponding outcomes, respectively. The models trained on X_train and Y_train. The models tried to learn the behavior/distribution of the data and generate a hypothesis/fitting function. Once the training is concluded the model will then test the X_test and produce an output (prediction) called Y_pred. A comparison is done between Y_pred and Y_test. We had previously discussed that reduction of the number of irrelevant risk factors can produce better performances and significantly improve classifications. In this study we used conventional FLAIM framework only has phase I and III and we used it to compare it with Neo-V Framework. The Deep-FLAIM model is a 4 layered model and details have been reported previously (5) . Performance of all models was evaluated by comparing their accuracies and area under the receiver-operator curves (AUROC). Primary outcomes included sensitivity and specificity, while secondary outcomes included positive predictive values (PPV) and negative predictive values (NPV). From a total of 1228 patients we selected 1214 patients that were adult patients with complete data and RT-PCR proven COVID-19 infections. Demographics of this population showed a All rights reserved. No reuse allowed without permission. preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. shows results from univariate and multivariate analysis (hazard ratios, confidence intervals and p-values) of all the clinical and laboratory data. In phase II (new variable phase) we created 11 new variables that included; total number of comorbidities (range 0-6, HR=1.30, 95%CI=1.14-1.49) , more than 2 comorbidities (n=499, Table 2 . The new variables in phase II referred to as 'neo-variables' were statistically significant in the uni and the multivariate analysis except the total number of symptoms. The performance of our previously designed Deep-FLAIM model was compared to the Neo-V framework (including Deep-Neo-V and other conventional machine learning algorithms) see Table 3 . Performance results show Deep-FLAIM (training accuracy = 86.7%, testing All rights reserved. No reuse allowed without permission. preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. As the second wave of COVID-19 has started to unfold the already stained healthcare systems globally are being pushed to the limit with hospital and intensive care unit (ICU) beds reaching full-capacity. Impact of the virus has been global with developed countries even struggling with infection rates and hospitalization (11) . The second wave is anticipated All rights reserved. No reuse allowed without permission. preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this this version posted December 14, 2020. ; to be much tougher than the first one (12) . With new vaccines on the horizon infection rates in the United states have skyrocketed to 1.34 million cases being diagnosed in the second week of December 2020 and more than 1100 deaths (13) . The vaccines have just received FDA approval for widespread use of the vaccine for prevention of the disease while the actual logistics and distribution plans are still unknown (14). However, there is a need for the existence of clinical biomarkers and predictive models for mortality in these patients. machine learning has been used to predict mortality in cancer (15) , cardiac disease (16); while our own work on mortality prediction has been on trauma patients, postoperative ileus cases in the ICU (5, 6) and diverticulitis in the inpatient setting (17) (25), AB+ Blood group (26) and recurrent admission to the ICU. Hematological labs that were associated with mortality (previously presented in the results) were also seen in other studies (27) . Biochemical laboratory All rights reserved. No reuse allowed without permission. preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this this version posted December 14, 2020. ; abnormalities like creatinine, blood urea nitrogen (28) , INR and PT (29) were also associated with mortality. Patients managed with invasive ventilation (30) and non-invasive ventilation (31) which actually signifies that the patients that were not able to maintain normal respiratory physiology had worse outcomes. Having fever (32) (34) . However, the Deep-Neo-V model underperformed in terms of sensitivity and slightly with negative predictive value. The Deep-Neo-V will continue to improve and develop and will potentially be replaced by a model with better performance parameters (accuracy, PPV and NPV). This All rights reserved. No reuse allowed without permission. preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The Deep-Neo-V model has some limitations in terms of the available dataset, retrospective nature of the dataset and data form a single hospital, analyzed at admission and day-one data, other observational study confounders may exist and are unaccounted for. In the immediate future we actively look to validate these findings in an external dataset. In the longer term we will continue to develop an algorithm built on the Neo-V Framework approach that has the potential to be implemented, initially in future pandemics because of its ability to accurately predict outcomes using smaller datasets. Deep-Neo-V is a statistically robust machine learning model that is developed for clinical use to predict mortality risk in patients admitted with RT-PCR proven COVID-19 infection. The mortality prediction was modeled based on clinically relevant variables (patient associated risk factors and the first 24-hours labs. Our experimental results show that with a high All rights reserved. No reuse allowed without permission. preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this this version posted December 14, 2020. ; accuracy and specificity it has the potential to develop as a test of choice for predicting mortality in COVID-19 patients. These findings need further external validation. All rights reserved. No reuse allowed without permission. preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this this version posted December 14, 2020. ; Maleeha Naseem: Study design, methodology, protocol design, IRB/ERC approval, data collection supervision, data collection tool and quality control, statistical analysis review, medical literature review and article writing. Table 1 . Demographics with Univariate and multivariate analysis of clinical variables as part of Phase I of the Neo-V and FLAIM machine learning frameworks. All rights reserved. No reuse allowed without permission. preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this this version posted December 14, 2020. ; preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this this version posted December 14, 2020. ; T a b l e 1 . D e m o g r a p h i c s w i t h U n i v a r i a t e a n d m u l t i v a r i a t e a n a l y s i s o f c l i n i c a l v a r i a b l e s a s p a r t o f P h a s e I o f t h e N e o -V a n d F L A I M m a c h i n e l e a r n i n g f r a m e w All rights reserved. No reuse allowed without permission. preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this this version posted December 14, 2020. All rights reserved. No reuse allowed without permission. preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The effect of large-scale anti-contagion policies on the COVID-19 pandemic An interactive web-based dashboard to track COVID-19 in real time Protect the NHS" in the United Kingdom Second wave COVID-19 pandemics in Europe: a temporal playbook A statistically rigorous deep neural network approach to predict mortality in trauma patients admitted to the intensive care unit A hybrid machine learning framework to predict mortality in paralytic ileus patients using electronic health records (EHRs) Exploring the Potential of Artificial Intelligence and Machine Learning to Combat COVID-19 and Existing Opportunities for LMIC: A Scoping Review Guidance for COVID-19 Testing for CAP-Accredited Laboratories. CAP Guidelines, caporg CDC's Diagnostic Test for COVID-19 Only and Supplies. CDC Guidelines, cdcgov SMOTE: Synthetic Minority Over-sampling Technique COVID-19: Emergence, Spread, Possible Treatments, and Global Burden Beware of the second wave of COVID-19 Tracker: Maps, charts, and data provided by the CDC. wwwcdcgov. 2020. 14. (FDA) FDA. COVID-19 Vaccines: FDA has rigorous scientific and regulatory processes in place to facilitate development and ensure the safety, effectiveness and quality of COVID-19 vaccines Machine Learning Approaches to Predict 6-Month Mortality Among Patients With Cancer Machine learning for prediction of all-cause mortality in patients with suspected coronary artery disease: a 5-year multicentre prospective registry analysis Machine Learning Can Predict Deaths in Patients with Diverticulitis During their Hospital Stay Effect of chronic obstructive pulmonary disease and smoking on the outcome of COVID-19 Mortality analysis of COVID-19 infection in chronic kidney disease, haemodialysis and renal transplant patients compared with patients without kidney disease: a nationwide analysis from Turkey Description and Proposed Management of the Acute COVID-19 Cardiovascular Syndrome High incidence and mortality of pneumothorax in critically Ill patients with COVID-19. Heart Lung Acute respiratory failure in COVID-19: is it "typical Coronavirus Disease Dyspnea rather than fever is a risk factor for predicting mortality in patients with COVID-19 Prediction model and risk scores of ICU admission and mortality in COVID-19 Blood type A associates with critical COVID-19 and death in a Swedish cohort Hematologic, biochemical and immune biomarker abnormalities associated with severe illness and mortality in coronavirus disease 2019 (COVID-19): a meta-analysis Kidney disease is associated with in-hospital death of patients with COVID-19 A novel severity score to predict inpatient mortality in COVID-19 patients Mortality in COVID-19 is not merely a question of resource availability. The Lancet Respiratory medicine Mortality of Patients With Severe COVID-19 in the Intensive Care Unit: An Observational Study From a Major COVID-19 Receiving Hospital Risk factors for predicting mortality in elderly patients with COVID-19: A review of clinical data in China Association Between Administration of Systemic Corticosteroids and Mortality Among Critically Ill Patients With COVID-19: A Meta-analysis Deep-learning artificial intelligence analysis of clinical variables predicts mortality in COVID-19 patients