key: cord-1015252-a35q5uvx authors: Shi, Yiyun; Pandita, Aakriti; Hardesty, Anna; McCarthy, Meghan; Aridi, Jad; Weiss, Zoe F.; Beckwith, Curt G.; Farmakiotis, Dimitrios title: Validation of pneumonia prognostic scores in a statewide cohort of hospitalised patients with COVID‐19 date: 2020-12-31 journal: Int J Clin Pract DOI: 10.1111/ijcp.13926 sha: 2431a1201f3adb391145455924da3de93d665533 doc_id: 1015252 cord_uid: a35q5uvx OBJECTIVE: We aimed to externally validate the predictive performance of two recently developed COVID‐19‐specific prognostic tools, the COVID‐GRAM and CALL scores, and prior prognostic scores for community‐acquired pneumonia (CURB‐65), viral pneumonia (MuBLSTA) and H1N1 influenza pneumonia (Influenza risk score) in a contemporary US cohort. METHODS: We included 257 hospitalised patients with laboratory‐confirmed COVID‐19 pneumonia from three teaching hospitals in Rhode Island. We extracted data from within the first 24 hours of admission. Variables were excluded if values were missing in >20% of cases, otherwise, missing values were imputed. One hundred and fifteen patients with complete data after imputation were used for the primary analysis. Sensitivity analysis was performed after the exclusion of one variable (LDH) in the complete dataset (n = 257). Primary and secondary outcomes were in‐hospital mortality and critical illness (mechanical ventilation or death), respectively. RESULTS: Only the areas under the receiver‐operating characteristic curves (RO‐AUC) of COVID‐GRAM (RO‐AUC = 0.775, 95% CI 0.525‐0.915) for in‐hospital death, and CURB65 for in‐hospital death (RO‐AUC = 0.842, 95% CI 0.674‐0.932) or critical illness (RO‐AUC = 0.766, 95% CI 0.584‐0.884) were significantly better than random. Sensitivity analysis yielded similar trends. Calibration plots showed better agreement between the estimated and observed probability of in‐hospital death for CURB65, compared with COVID‐GRAM. The negative predictive value (NPV) of CURB65 ≥2 was 97.2% for in‐hospital death and 88.1% for critical illness. CONCLUSIONS: The COVID‐GRAM score demonstrated acceptable predictive performance for in‐hospital death. The CURB65 score had better prognostic utility for in‐hospital death and critical illness. The high NPV of CURB65 values ≥2 may be useful in triaging and allocation of resources. pneumonia (MuLBSTA 6 ) or H1N1-influenza pneumonia (influenza risk score 7 ) in a contemporary cohort in the United States (US). We retrospectively studied 257 adult patients admitted for COVID-19, diagnosed by PCR in a nasopharyngeal sample, at three teaching hospitals (Newport Hospital, Newport; The Miriam Hospital, Rhode Island Hospital, Providence) in RI, USA, between 3/1/2020 and 5/18/2020. The study was approved by the Lifespan Institutional Review Board, with a waiver of informed consent. The first 97 patients were enrolled consecutively from 3/1/2020 to 4/3/2020. During the surge in April and May, 160 additional patients were randomly enrolled because of the limitations in our abstraction capacity. To ensure that we had a relatively representative sample, we compared weekly case fatality rates between the study sample and all COVID-19 patients admitted to the three hospitals during the study period with the Wilcoxon test. The COVID-GRAM, 3 In sensitivity analysis, we validated prognostic scores without LDH (CURB65, MuLBSTA, influenza risk score) in patients with available LDH levels (n = 115), and scores with LDH (COVID-GRAM, CALL), after the removal of LDH values from these models, in the entire cohort (n = 257). We assessed the fitness (agreement between estimated and observed probability) of scores with statistically significant RO-AUC (lowest 95% CI >0.5) for in-hospital death, by means of calibration plots with intercept adjustment. Last, we calculated sensitivity, specificity, positive (PPV) and negative (NPV) predictive value of the CURB65 score, given its good performance and simplicity, for in-hospital death and progression to critical COVID-19, using a cut-off value of 2, similar to CAP. 5 Data were analysed by R software (version 3.6.3, R Foundation). There were no significant differences in weekly in-hospital casefatality rates between the study sample (n = 257) and the whole patient population (n = 817) of patients with COVID-19 admitted to the three hospitals during the study period (Wilcoxon P = .412). The only parameter with >20% missing values in the first 24 hours was lactate dehydrogenase (LDH), in 142 patients (55.3%), who were excluded from the initial comparisons of scores with LDH as one of the parameters. Direct bilirubin levels were missing in 45 (17.5%), neutrophil or lymphocyte counts in 4 (1.6%), and blood urea nitrogen in 3 (1.5%) patients. These values were imputed. Mortality was associated with advanced age, presence of certain comorbidities (hypertension, diabetes), admission from a nursing home, hypoxia or tachypnea on admission, thrombotic events during hospitalisation, higher LDH, BUN, bilirubin, white blood cell count, neutrophil-to-lymphocyte ratio, lower albumin and tCO 2 on admission. The comparisons between patients who developed critical illness vs those who did not follow a similar pattern, except that patients who developed critical illness had a higher percentage of unconsciousness, imaging abnormalities, ferritin and aspartate transaminase (AST) levels and incidence of other viral coinfections. There was no difference in the percentage of hypertension or the level of direct bilirubin (Table S1 ). For in-hospital death, only the RO-AUC of the COVID-GRAM and CURB65 scores were significantly better than random (Table 1, Figure 1 ). For critical illness, only the RO-AUC of the CURB65 score was significantly better than random (Table 1, Figure S1 ). Validation of models without LDH in patients with available LDH levels There has been a worldwide effort to develop COVID-19-specific prognostic tools. The variables used in such models are often similar to previously validated pneumonia prediction tools. The CURB65 model, likely the easiest score to calculate, has been widely used to compare the predictive value of new scoring systems. While its performance is usually inferior compared with novel scores in the derivative populations, its RO-AUC overall has been reproducible between 0.7 and 0.9 in the studies of community acquired, 9 viral 6 or COVID-19 2,3,10,11 pneumonia, in agreement with our results. The 30day mortality rates of patients with COVID-19 and CURB65 values ≥2 were 30.5% 10 and 33.3% 11 in two other COVID-19 cohorts, similar to the 29.9% in-hospital mortality rate in our study. Additionally, CURB65 ≥2 had an NPV of >97% for inpatient death and >88% for critical illness. Therefore, the majority of such inpatients can be safely managed outside of the ICU, which could potentially save valuable resources. Our findings may raise the question of which features of COVID-19 pneumonia are unique enough to warrant specific predictive tools. A higher age cutoff may be needed, compared with other viral pneumonia, as demonstrated by epidemiological studies 12 and better performance of scores with higher age cut-off in our study (Table 1, Table S1 ). Also, patients with COVID-19 experience more endothelial injury and thromboembolic events when compared with patients with influenza pneumonia, 13 and some demonstrate a hyperinflammatory phenotype with rapid progression. 14 Markers reflecting potential COVID-19-specific sequelae, such as inflammation or hypercoagulability, may enhance the predictive value of scores. Our study showed that even scores with satisfactory performance in predicting mortality performed poorly in predicting critical illness, which is in agreement with another recent report from Italy, validating CALL score. 15 This may be the reflection of different thresholds of ICU transfer or mechanical ventilation between different countries and time periods. Our report has limitations, mainly the small number of cases and imputation of missing values. Also, we did not have enough data on coagulation and inflammation to modify the above scores, as In summary, in this contemporary US cohort of inpatients with COVID-19, the easily calculated CURB65 score is potentially useful in predicting critical illness and death. Our findings highlight the value of coordinated efforts to validate and enhance existing scores. These efforts will help streamline patient triage, improve the allocation of resources, and aid in appropriate stratification for the design of future clinical trials. DF has received research grants from Astellas and Viracor, and consultation fee from Viracor, outside of the submitted work. All other authors have nothing to disclose. The present work was partially supported by a Brown Physicians Inc Academic Assessment Grant. The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request. Prediction models for diagnosis and prognosis of covid-19 infection: systematic review and critical appraisal Systematic evaluation and external validation of 22 prognostic models among hospitalised adults with COVID-19: an observational cohort study Development and validation of a clinical risk score to predict the occurrence of critical illness in hospitalized patients with COVID-19 Prediction for progression risk in patients with COVID-19 pneumonia: the CALL score Defining community acquired pneumonia severity on presentation to hospital: an international derivation and validation study Clinical features predicting mortality risk in patients with viral pneumonia: the MuLBSTA score Score to identify the severity of adult patients with influenza A (H1N1) 2009 virus infection at hospital admission Small-sample degrees of freedom with multiple imputation Validation of a predictive rule for the management of community-acquired pneumonia Performance of pneumonia severity index and CURB-65 in predicting 30-day mortality in patients with COVID-19 Applicability of the CURB-65 pneumonia severity score for outpatient treatment of COVID-19 Comparing SARS-CoV-2 with SARS-CoV and influenza pandemics Pulmonary vascular endothelialitis, thrombosis, and angiogenesis in covid-19 COVID-19 illness in native and immunosuppressed states: a clinical-therapeutic staging proposal The CALL score for predicting outcomes in patients with COVID-19