key: cord-259329-8pta6o6a
authors: Haimovich, Adrian; Ravindra, Neal G.; Stoytchev, Stoytcho; Young, H. Patrick; PerryWilson, Francis; van Dijk, David; Schulz, Wade L.; Taylor, R. Andrew
title: Development and validation of the quick COVID-19 severity index (qCSI): a prognostic tool for early clinical decompensation
date: 2020-07-21
journal: Ann Emerg Med
DOI: 10.1016/j.annemergmed.2020.07.022
sha: 
doc_id: 259329
cord_uid: 8pta6o6a

Abstract Objective The goal of this study was to develop a prognostic tool of early hospital respiratory failure among emergency department (ED) patients admitted with COVID-19. Methods This was an observational, retrospective cohort study from a nine ED health system in the United States of admitted adult patients with SARS-CoV-2 (COVID-19) and a ≤ 6 L/min oxygen requirement. We sought to predict respiratory failure within 24 hours of admission as defined by oxygen requirement of ≥ 10 L/min, any high-flow device, non-invasive or invasive ventilation, or death. Predictive models were compared to the Elixhauser comorbidity index, quick serial organ failure assessment (qSOFA), and the CURB-65 pneumonia severity score. Results During the study period from March 1 to April 27, 2020, 1,792 patients were admitted with COVID-19, 620 (35%) of whom had respiratory failure in the ED. Of the remaining 1,172 admitted patients, 144 (12.3%) met the composite endpoint within the first 24 hours of hospitalization. Using area under receiver-operating characteristic curves, we compared the performance of a novel bedside scoring system, the quick COVID-19 severity index (qCSI) composed of respiratory rate, oxygen saturation, and oxygen flow rate (mean [95% CI]) (0.81 [0.73-0.89]), a machine- learning model, the COVID-19 severity index (0.76 [0.65-0.86]), to the Elixhauser mortality index (0.61 [0.51-0.70])), CURB-65 (0.50 [0.40-0.60]), and qSOFA (0.59 [0.50-0.68]). A low qCSI score (≤ 3) had a sensitivity of 0.79 [0.65- 0.93] and specificity of 0.78 [0.72-0.83] in predicting respiratory decompensation with a less than 5% risk of outcome in the validation cohort. Conclusions A significant proportion of admitted COVID-19 patients progress to respiratory failure within 24 hours of admission. These events are accurately predicted using bedside respiratory exam findings within a simple scoring system.

. A low qCSI score was associated with a less than 5% risk of respiratory decompensation in the validation cohort. Conclusions: A significant proportion of admitted COVID-19 patients progress to respiratory failure within 24 hours of admission. These events are accurately predicted using bedside respiratory exam findings within a simple scoring system.

Background SARS-CoV-2 (COVID-19) is a global pandemic with millions of cases and hundreds of thousands of deaths. 1, 2 Despite initial reports of patient characteristics and risk factors for critical illness, there is little evidence-based guidance available to aid provider decision-making in safely dispositioning patients with COVID-19. 3, 4 Inappropriate inpatient dispositions lead to increased provider contacts in the form of rapid response teams and the utilization of multiple care areas at a time when hospital capacities are limited. 5, 6 More significantly, in other domains of emergency care, undertriage of patients is associated with worse morbidity and mortality than patients directly admitted to higher levels of care. 7, 8 Given the high morbidity and mortality associated with COVID-19 and uncertainty around the disease process and prognosis, there is great urgency in developing and validating effective clinical risk stratification tools for COVID-19 patients.

Expert recommended admissions guidelines do not risk stratify among patients with severe COVID-19. 9 International definitions of severe COVID-19 are evolving, but typically include respiratory rate ≤ 30/min, SpO 2 ≥ 93%, PaO 2 :FiO 2 ≤ 300 mmHg, and infiltrates of ≥ 50% of lungs. 9, 10 Critical COVID-19 exists on a spectrum with severe illness and involves organ failure, often leading to prolonged mechanical ventilation. 9 In a large cohort of COVID-19 patients, severe and critical illness represented almost 20% of the studied population. 10 In most institutions, dispositions for patients with critical respiratory failure (e.g., ventilated patients or those on non-rebreather masks) are largely apparent and determined by system protocols and capacity. Rapid progression from severe to critical illness, however, is a common problem and presents a prognostic challenge for ED providers determining admissions.

For this reason, we focus on patients for whom critical respiratory illness is not universally apparent in the EDnamely those requiring at most 6 L/min nasal cannula. In our health system, 6 L/min is typically the maximum flow rate delivered by nasal cannula. Over 90% of patients on ≤ 6 L/min are admitted to the floors, but of those, over 10% were observed to have increased oxygen requirements within 24 hours. Conversely, among these patients admitted to higher levels of care, about 70% did not progress above 6 L/min nasal cannula. These data suggest potential to improve our ability to risk-stratify ED patients prior to admission.

The objective of this study was to derive a risk stratification tool to predict 24 hour respiratory decompensation in admitted patients with COVID-19. Here, we expand on previous efforts describing the course of critical COVID-19 illness in three ways. First, we focus on ED prognostication by studying patient outcomes within 24 hours of admission using data available during the first four hours of presentation. 11 While critical illness often occurs later in hospitalization, the relevance of these later events to ED providers is less clear. We emphasize oxygen requirements and mortality, rather than intensive care unit placement, because we have observed the latter to have highly variable criteria depending on total patient census. 12 Second, to aid healthcare providers in assessing illness severity in COVID-3 19 patients, we present predictive models of early respiratory failure during hospitalization and compare them to three benchmarks accessible using data in the electronic health record: the Elixhauser comorbidity index, 13 the quick sequential organ failure assessment (qSOFA), 14, 15 and the CURB-65 pneumonia severity score. 16 While many clinical risk models exist, these benefit from wide clinical acceptability and relative model parsimony as they require minimal input data for calculation. The Elixhauser comorbidity index was derived to enable prediction of hospital death using administrative data. 13 The qSOFA score was included in SEPSIS-3 guidelines and can be scored at the bedside as it includes respiratory rate, mental status, and systolic blood pressure. 14 The CURB-65 pneumonia severity score has been well-validated for hospital disposition, but its utility in both critical illness and COVID-19 is unclear. 16, 17 Finally, we make the prognostic tool, the quick COVID-19 Severity Index (qCSI), available to the public via a web interface.

This was a retrospective observational cohort study to develop a prognostic model of early respiratory decompensation in patients admitted from the emergency department with COVID-19. The healthcare system is comprised of a mix of suburban community (n = 6), urban community (n = 2), and urban academic (n = 1) EDs. Data from eight EDs were used in the derivation and cross-validation of the predictive model, while data from the last urban community site was withheld for independent validation. We adhered to the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) checklist (Supplementary Materials). 18 This study was approved by our local institutional review board (IRB# 2000027747).

Patient demographics, summarized past medical histories, vital signs, outpatient medications, chest x-ray (CXR) reports, and laboratory results available during the ED encounter were extracted from our local Observational Medical Outcomes Partnership (OMOP) data repository and analyzed within our computational health platform. 19 Data were collected into a research cohort using custom scripts in PySpark (version 2.4.5) that were reviewed by an independent analyst.

Non-physiologic values likely related to data entry errors for vital signs were converted to missing values based on expert-guided rules (Table S1) . Laboratory values at minimum or maximum thresholds and encoded with "<" or ">" were converted to the numerical threshold value and other non-numerical values were dropped. Past medical histories were generated by using diagnoses prior to the date of admission to exclude potential future information in modeling. Outpatient medications were mapped to the First DataBank Enhanced Therapeutic Classification System. 20 CXR reports were manually reviewed by two physicians and categorized as "no opacity", "unilateral opacity", or "bilateral opacities". One hundred x-ray reports were reviewed by both physicians to determine inter-rater agreement with weighted kappa. Oxygen devices were similarly extracted from OMOP (Table S2) .

We define critical respiratory illness in the setting of COVID-19 as any COVID-19 patient meeting one of the following criteria: oxygenation flow rate ≥ 10 L/min, high-flow oxygenation, noninvasive ventilation, invasive ventilation, or death (Table S2) . We do not include intensive care unit admission in our composite outcome because at the start of the COVID-19 pandemic, ICU admissions were protocolized to include even minimal oxygen requirements. A subset of outcomes were manually reviewed by physician members of the institutional computational healthcare team as part of a system-wide process to standardize outcomes for COVID-19 related research.

Data included visits from March 1, 2020 through April 27, 2020 as our institution's first COVID-19 tests were 5 ordered after March 1, 2020. This study included admitted COVID-19 positive patients as determined by test results ordered between 14 days prior to and up to 24 hours after hospital presentation. We included delayed testing because institutional guidelines initially restricted testing within the hospital to inpatient wards. Testing for COVID-19 was performed at local and/or reference laboratories by nucleic acid detection methods using oropharyngeal (OP), nasopharyngeal (NP), or a combination OP/NP swab. We excluded patients less than 18 years of age and those who required more than 6 L/min or otherwise met our critical illness criteria at any point within four hours of presentation. The latter was intended to exclude patients for whom critical illness was nearly immediately apparent to the medical provider and for whom a prediction would not be helpful. Patients who explicitly opted out of research were excluded from analysis (n < 5). Data were extracted greater than 24 hours after the last included patient visit so that all outcomes could be extracted from the electronic health record.

We generated comparator models using Elixhauser comorbidity index, qSOFA, and CURB-65 (Supplementary Materials). ICD-10 codes from patient past medical histories were mapped to Elixhauser comorbidities and indices using H-CUP Software and Tools (hcuppy package, version 0.0.7). 21, 22 qSOFA was calculated as the sum of the following findings, each of which were worth one point: GCS < 15, respiratory rate ≥ 22, and systolic blood pressure ≤ 100. CURB-65 was calculated as the sum of the following findings, each of which were worth one point:

GCS < 15, BUN > 19 mg/dL respiratory rate ≥ 30, systolic blood pressure < 90 mmHg or diastolic ≤ 60 mmHg, and age 65 years. Baseline models were evaluated on the training and internal validation cohort using logistic regression on the calculated scores.

Samples from eight hospitals were used in model generation and internal validation with the remaining large, urban community hospital serving as an independent site for validation. All models were fit on patient demographic and clinical data collected during the first four hours of patient presentation and predictions are made using the most recently available data at the four hour time point unless otherwise noted. We used an ensemble technique to identify and rank potentially important predictive variables based on their occurrence across multiple selection methods: univariate regression, random forest, logistic regression with LASSO, Chi-square testing, gradient boosting information gain, and gradient boosting Shapley additive explanation (SHAP) interaction values (Supplementary Materials). [23] [24] [25] We counted the co-occurences of the the top 30, 40, and 50 variables of each of the methods prior to selecting features for a minimal scoring model (qCSI) and machine learning model (CSI) using gradient boosting.

For the qCSI, we used a point system guided by logistic regression (Supplementary Materials). The gradientboosting CSI model was fit using the XGBoost package and hyperparameters were set using Bayesian optimization with a tree-structured Parzen estimator (Supplementary Materials). 26, 27 All analyses were performed in Python (version 3.8.2).

We report summary statistics of model performance in predicting the composite outcome between four and 24 hours of hospital arrival. We used bootstrapped logistic regression with ten-fold cross-validation to generate performance benchmarks for the Elixhauser, qSOFA, CURB-65, and qCSI models and bootstrapped gradient boosting with tenfold cross validation for the CSI model. Where necessary, data were imputed using training set median values of bootstraps. We report area under receiver-operating characteristic (AU-ROC), accuracy, sensitivity and specificity at Youden's index, area under precision-recall curve (AU-PRC), 28 Brier score, F1, and average precision (Supplementary Materials). Similarly, to evaluate model performance on the independent validation cohort, means and confidence intervals were calculated from bootstrap iterations of the test set using sampling with replacement. We report 95% confidence intervals derived from the percentiles of the bootstrapped distribution or Welch's two-sample t-test for statistical comparisons of model performance. 29 

Between March 1, 2020 and April 27, 2020, there were a total of 1,792 admissions for COVID-19 patients meeting our age criteria. Of these, 620 patients (35%) were excluded by meeting critical respiratory illness endpoints within four hours of presentation. Of the included patients, 144 (12.3%) had respiratory decompensation within the first 24 hours of hospitalization: 101 (8.6%) requiring >10 L/min oxygen flow, 112 (9.6%) on a high-flow device (Table S2) Table 1 . Study patient flow is shown in Figure 1 and patient characteristics for the development and validation populations are shown in Table S3 -S4.

Our full dataset included 713 patient variables available during the first four hours of the patient encounters (Table   S5 ). These included demographics, vital signs, laboratory values, comorbidities, chief complaints, outpatient medications, tobacco use histories, and CXR. Radiologist evaluated CXRs were classified into three categories with strong inter-rater agreement (κ = 0.81). Associations between CXR findings and outcomes are shown in Table S6 . We preferentially selected variables available at bedside for derivation of the qCSI. Our ensemble approach identified three bedside variables as consistently important across the variable selection models: nasal cannula requirement, minimum recorded pulse oximetry, and respiratory rate ( Figure S1 ). These three features appeared in at least five of the six variable selection methods.

We divided each of these three clinical variables into value ranges using clinical experience and used logistic regression to derive weights for the qCSI scoring system ( Table 2) . Normal physiology was used as the baseline category, and the logistic regression odds ratios were offset to assign normal clinical parameters zero points in the qCSI (Supplementary Materials). The qCSI range is from 0 to 12.

We identified an additional twelve features from the predictive factor analysis for use in a machine learning model (CSI) with gradient boosting (Table 2, Figure S1 ). These variables were selected by balancing the goals of model parsimony, minimizing highly correlated features (i.e., various summaries of vital signs), and predictive performance.

We used SHAP methods to understand the importance of various clinical variables in the CSI (Figure 2) . 25, [30] [31] [32] values are an extension of the game-theoretic Shapley values that seek to describe variable impacts on model output, as defined as the contribution of a specific variable to the prediction itself. 30 The key advantage of the related SHAP values is that they add interpretability to complex models like gradient boosting, which otherwise provide opaque outputs. SHAP values are dimensionless and represent the log-odds of the marginal contribution a variable makes on a single prediction. In the case of our gradient boosting CSI model, we employ an isotonic regression step for model calibration, so the SHAP values reflect a relative weighting of contributions. 33 The rank-order of average absolute SHAP values across all variables in a model suggests the most important variables in assigning modeled risk. For the CSI these were flow rate by nasal cannula, followed by lowest documented pulse oximetry, and aspartate aminotransferase (AST) (Figure 2a) . Consistent with prior studies, we also observed utility of inflammatory markers, ferritin, procalcitonin, and CRP. We then explored how ranges of individual feature values affected model output (Figure 2b ). For example, low oxygen flow rates (blue) are protective as indicated by negative SHAP values, as are high pulse oximetry values (red). To better investigate clinical variable effects on predicted patient risk, we generated individual variable SHAP value plots (Figure 3 ). Age displayed a nearly binary risk distribution with an inflection point between 60 and 70 years of age (Figure 3a) . Younger patients displayed a higher risk of 24 hour critical illness than did older patients. We also observed that elevated AST, alanine aminotransferase (ALT), and ferritin were associated with elevated model risk, but the SHAP values reached their asymptotes well before the maximum value for each of these features (Figure 3b-d) . AST and ALT SHAP values reached their maximum within 9 normal or slightly elevated ranges for these laboratory tests. The inflection point in risk attributable to ferritin levels, however, was close to 1000 ng/mL, above institutional normal range for this test (30-400 ng/mL).

Across the cohort, 72% of patients did not have a GCS documented. On cross-validation, the qCSI had an AU-ROC 

The qCSI is available at covidseverityindex.org. The qCSI calculator includes selection boxes for each of the three variables which are summed to generate a score and prediction as estimated using the independent validation cohort.

The data in this study were observational data provided from a single health system and so may not be generalizable based on local testing and admissions practices. Our data were extracted from an electronic health record, which is associated with known limitations including propagation of old or incomplete data. There are important markers of oxygenation which were out of the scope of our study, including alveolar-arterial gradients. Due to data availability, no signs or symptoms or provider notes were included as candidate predictor variables.

Retrospective observational studies lack control of variables so prospective studies will be required to assess validity of the presented models and the specificity of the features we identify as important to COVID-19 progression. Due to the retrospective nature of this study and the use of electronic health records, data imputation and assumptions about missingness were required which introduce biases into our results. We assume a GCS of 15 unless documented otherwise, which may underestimate severity in qSOFA and CURB-65. Likewise, comorbidities were populated from prior in-system diagnoses -patients without system visits are likely to have lower Elixhauser indices than those whose care were integrated within the health system. In the qCSI calculations, nasal cannula flow rate was imputed if nasal cannula was documented without a flow rate. In the CSI, no specific imputations were required as gradient boosting natively handles missing values. Chest x-ray interpretation was done manually using radiology reports, but without reviewing the radiography, which introduces subjectivity as reflected in the inter-rater agreement metric.

There are limitations in model performance with confidence intervals reflective of moderate study size. We additionally did not compare the models to unstructured provider judgment and thus one cannot make conclusions as to whether this tool has utility above and beyond clinical gestalt. Most significant, however, is that management of COVID-19 is evolving, so it may be possible that future clinical decisions may not match those standards used in the reported clinical settings.

1Consistent with clinical observations, we noted a significant rate of progression to critical respiratory illness within the first 24 hours of hospitalization in COVID-19 patients. We used six parallel approaches to identify a subset of variables for the final qCSI and CSI models. The qCSI ultimately requires only three variables, all of which are accessible at the bedside.

We propose that a qCSI score of 3 or less be considered low-likelihood for 24 hour respiratory critical illness with a mean outcome rate of 4% in the independent validation cohort ( Figure 4 ) and a LR-of 0.27 [0. 26-0.28 ]. This score is achievable under the following patient conditions: respiratory rate ≤ 28, minimum pulse oximetry reading of ≥ 89%, and oxygen flow rate of ≤ 2L. In the validation cohort, a qCSI cutoff above 3 had a sensitivity of 0.79 (0.65-0.93) in predicting progression of respiratory failure. We note, however, that few patients in the validation cohort had qCSI of patients. In alignment with current hypotheses about COVID-19 severity, we note that multiple variable selection techniques identified inflammatory markers including CRP and ferritin as potentially important predictors. More striking however, was the importance of AST and ALT in CSI predictions as calculated with SHAP values. 35, 36 Interestingly, the transition point where the SHAP value analysis identified model risk associated with liver chemistries was at the high end of normal, consistent with previous observations that noted that normal to mild liver dysfunction among COVID-19 patients. We hypothesize that the asymptotic quality of the investigated variables with respect to CSI risk contributions reflects our moderate study size. We expect that scaling CSI training to larger cohorts will further elucidate the impacts of more extreme lab values. While our dataset included host risk factors including smoking history, obesity, and BMI, these did not appear to play a prominent role in predicting acute deterioration. Here, we recognize two important considerations: first, that predictive factors may not be mechanistic or causative factors in disease, and second that these factors may be related to disease severity without providing predictive value for 24 hour decompensation.

We include CXRs for 1,170 visits in this cohort. CXR are of significant clinical interest as previous studies have shown high rates of ground glass opacity and consolidation. 37 Chest CT may have superior utility for COVID-19

investigation, but are not being widely performed at our institutions as part of risk stratification or prognostic evaluation. 38, 39 CXR reports were classified based on containing bilateral, unilateral, or no opacities or consolidations. We found high inter-rater agreement in this coding, but CXR were not consistently identified by our variable selection models. A ma-jority of patients were coded as having bilateral consolidations, limiting the specificity of the findings. Further studies using natural language processing of radiology reports or direct analysis of CXR with tools like convolutional neural networks will provide more evidence regarding utility of these studies in COVID-19 prognostication. 40 Furthermore, we do not consider other applications of CXR including the identification of other pulmonary findings like diagnosis of bacterial pneumonia.

The Elixhauser comorbidity index, qSOFA, and CURB-65 baseline models provided the opportunity to test wellknown risk stratification and prognostication tools with a COVID-19 cohort. These tools were selected, in part, for their familiarity within the medical community, and because each has been proposed as having potential utility within the COVID-19 epidemic. We note the relatively limited predictive performance of these metrics, while simultaneously recognizing limitations in electronic health records and that none were designed to address the clinical question addressed here. We observed both a high rate of missing mental status documentation and a significant proportion of the population without documented medical histories. In particular, we hypothesize that the CURB-65 pneumonia severity score may still have utility in determining patient disposition with respect to discharge or hospitalization.

Future studies will be required to expand on this work in a number of ways. First, external validation is needed, as is comparison to physician judgement. Second, future studies may evaluate prospective robustness and utility of this scoring metric. Third, we expect related models to be extended to patient admission decisions as well as continuous hospital monitoring. [41] [42] [43] Finally, we anticipate potential applications in stratifying patients for therapeutic interventions. Early proof-of-concept studies for the viral RNA polymerase inhibitor Remdesivir include patients with severe COVID-19 as defined by pulse oximetry of ≤ 94% on ambient air or with any oxygen requirement. 44, 45 Given 13 ongoing drug scarcity, improved pragmatic, prognostic tools like the qCSI may offer a route to expanded inclusion criteria for ongoing trials or for early identification of patients who might potentially benefit from therapeutics.

Taken together, these data show that the qCSI provides easily accessed risk-stratification relevant to ED providers. The points position on the x-axis shows the impact that feature has on the model's prediction for a given patient. Color corresponds to relative variable value. 

Calibration of qCSI and CSI on the independent validation dataset. (a) Each patient in the validation cohort was assigned a score by qCSI, and the percentage who had a critical respiratory illness outcome were plotted with a line plot. Patients were then grouped into risk bins by qCSI intervals (0-3, 4-6, 7-9, 10-12) and the percentage of patients in each group with the outcome is indicated in the bar plot. (b) Each patient in the validation cohort was assigned a CSI, a percent risk from 0 to 100% using gradient boosting and isotonic regression. The percent of patients with CSI scores of 0-33%, 33-66%, and 66-100% who experienced critical respiratory illness at 24 hours are shown.

Novel Coronavirus (2019-nCoV) situation reports

COVID-19) Cases in

Cohort of 4404 Persons Under Investigation for COVID-19 in a NY Hospital and Predictors of ICU Care and Ventilation

Patient factors associated with SARS-CoV-2 in an admitted emergency department population

Rapid response teams: a systematic review and metaanalysis. Archives of internal medicine

Impact of COVID-19 pandemic on severity of illness and resources required during intensive care in the greater New York City area. medRxiv

Identifying infected emergency department patients admitted to the hospital ward at risk of clinical deterioration and intensive care unit transfer

Survival of critically ill patients hospitalized in and out of intensive care

Severe Covid-19

Characteristics of and important lessons from the coronavirus disease 2019 (COVID-19) outbreak in China: summary of a report of 72 314 cases from the Chinese Center for Disease Control and Prevention

US emergency department performance on wait time and length of visit. Annals of emergency medicine

Triage of scarce critical care resources in COVID-19: an implementation guide for regional allocation An expert panel report of the Task Force for Mass Critical Care and the

A modification of the Elixhauser comorbidity measures into a point system for hospital death using administrative data

The third international consensus definitions for sepsis and septic shock (Sepsis-3)

Critically ill SARS-CoV-2-infected patients are not stratified as sepsis by the qSOFA

Defining community acquired pneumonia severity on presentation to hospital: an international derivation and validation study

Performance of the CURB-65 score in predicting critical care interventions in patients admitted with community-acquired pneumonia. Annals of emergency medicine

Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration

Health care and precision medicine research: analysis of a scalable data science platform

First DataBank Enhanced therapeutic classification system (ETC)

Comorbidity measures for use with administrative data

for Healthcare Research A, Quality. HCUP Tools and Software. Healthcare Cost and Utilization Project (HCUP). Agency for Healthcare Research and Quality

Feature Selection Based on the Shapley Value

An introduction to variable and feature selection

From local explanations to global understanding with explainable AI for trees

Xgboost: A scalable tree boosting system

Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures

The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets

An Introduction to the Bootstrap. No. 57 in Monographs on Statistics and Applied Probability

A unified approach to interpreting model predictions

Explainable machine-learning predictions for the prevention of hypoxaemia during surgery

Prediction of gestational diabetes based on nationwide electronic health records

Predicting good probabilities with supervised learning

A prospective validation of the HEART score for chest pain patients at the emergency department

Liver injury in COVID-19: management and challenges

Characteristics of Liver Tests in COVID-19 Patients

Frequency and distribution of chest radiographic findings in COVID-19 positive patients

CT imaging features of 2019 novel coronavirus (2019-nCoV)

Clinically Applicable AI System for Accurate Diagnosis, Quantitative Measurements and Prognosis of COVID-19 Pneumonia Using Computed Tomography

A targeted real-time early warning score (TREWScore) for septic shock

A simple real-time model for predicting acute kidney injury in hospitalized patients in the US: A descriptive modeling study

A clinically applicable approach to continuous prediction of future acute kidney injury

Compassionate use of remdesivir for patients with severe Covid-19

Remdesivir in adults with severe COVID-19: a randomised, double-blind, placebo-controlled, multicentre trial. The Lancet

How do bootstrap and permutation tests work? Annals of Statistics

Performance Characteristics for CSI, qCSI, and comparison models on independent validation. Point estimates for model performance are provided at Youden's index. † The CSI AU-ROC is was statistically greater than qSOFA and Elixhauser after testing with Welch's t-test

Accuracy Sensitivity Specificity AU-PRC Brier score F1 Average Precision

Quick COVID-19 Severity Index (qCSI)

COVID-19 Severity Index (CSI)