key: cord-1028285-j7nluky5
authors: Sharp, Adam L.; Huang, Brian Z.; Broder, Benjamin; Smith, Matthew; Yuen, George; Subject, Christopher; Nau, Claudia; Creekmur, Beth; Tartof, Sara; Gould, Michael K.
title: Identifying patients with symptoms suspicious for COVID-19 at elevated risk of adverse events: The COVAS score
date: 2020-11-05
journal: Am J Emerg Med
DOI: 10.1016/j.ajem.2020.10.068
sha: 60277e75f7fbd23f53f259627154b9fe75941ef0
doc_id: 1028285
cord_uid: j7nluky5

OBJECTIVE: Develop and validate a risk score using variables available during an Emergency Department (ED) encounter to predict adverse events among patients with suspected COVID-19. METHODS: A retrospective cohort study of adult visits for suspected COVID-19 between March 1 – April 30, 2020 at 15 EDs in Southern California. The primary outcomes were death or respiratory decompensation within 7-days. We used least absolute shrinkage and selection operator (LASSO) models and logistic regression to derive a risk score. We report metrics for derivation and validation cohorts, and subgroups with pneumonia or COVID-19 diagnoses. RESULTS: 26,600 ED encounters were included and 1079 experienced an adverse event. Five categories (comorbidities, obesity/BMI ≥ 40, vital signs, age and sex) were included in the final score. The area under the curve (AUC) in the derivation cohort was 0.891 (95% CI, 0.880–0.901); similar performance was observed in the validation cohort (AUC = 0.895, 95% CI, 0.874–0.916). Sensitivity ranging from 100% (Score 0) to 41.7% (Score of ≥15) and specificity from 13.9% (score 0) to 96.8% (score ≥ 15). In the subgroups with pneumonia (n = 3252) the AUCs were 0.780 (derivation, 95% CI 0.759–0.801) and 0.832 (validation, 95% CI 0.794–0.870), while for COVID-19 diagnoses (n = 2059) the AUCs were 0.867 (95% CI 0.843–0.892) and 0.837 (95% CI 0.774–0.899) respectively. CONCLUSION: Physicians evaluating ED patients with pneumonia, COVID-19, or symptoms suspicious for COVID-19 can apply the COVAS score to assist with decisions to hospitalize or discharge patients during the SARS CoV-2 pandemic.

COVID-19 is a pandemic with worldwide impact which has depleted healthcare resources in many areas. [1, 2] Due to constraints in the availability of testing [3] and the rapid spread of the disease, data are limited about which patients are at highest risk of clinical decompensation.

Presenting symptoms of fever, cough, dyspnea and fatigue have been consistently reported from initial reports from China, Italy, and now the United States. [4] [5] [6] [7] [8] [9] [10] Initial reports describe risk factors and outcomes for admitted patients, but lack information about outcomes for patients outside of the hospital. Frontline physicians who make key decisions about disposition would benefit from rules or models to help them decide who requires hospitalization and who can be safely discharged to home. This is particularly challenging for clinicians that do not have SARS-CoV-2 testing results available at the time hospitalization decisions are made. Historically, scores for bacterial pneumonia, such as the pneumonia severity index (PSI) or CURB-65 have shown to improve the quality and efficiency of healthcare by objectively informing physicians about patient risks of death within 30-days. [11] [12] [13] [14] [15] Given the unique clinical features of COVID- 19 , there is a need for a novel risk score using data commonly available at the time of the emergency department encounter, in order to inform clinical decisions regarding the disposition of patients with symptoms suspicious for COVID-19.

Substantial surges of infected patients have saturated limited resources (hospital beds, ventilators, etc.) [2] in certain areas of the United States. As a result, this pandemic has changed manifest from infection with the virus, [4-6, 8, 10] therefore any patient with a COVID-19 diagnosis, pneumonia, or the infectious symptoms attributed to COVID-19 (eTable 1) were included in the derivation and validation of the risk score. The score was then tested among subgroups of patients with an initial ED ICD-10 COVID-19 or pneumonia diagnosis, as well as among patients who had a positive SARS-CoV-2 PCR test within 7 days of the encounter.

Our primary outcome was a composite measure of death or respiratory decompensation defined as receipt of mechanical ventilation, non-invasive ventilation, high-flow oxygen or oxygen delivered via face mask, within 7-days (ascertained from mortality files, inpatient flowsheet records and CPT/ICD-10 codes; eTable 2) of ED presentation. Secondary outcomes included positive test results for SARS CoV-2 within 7-days of encounter, hospitalizations (inpatient or observation status), and return ED visits or hospitalizations after discharge from the index ED visit.

To develop a parsimonious prediction model using information available at the point of care in the ED, we collected data about demographic characteristics, vital signs and comorbid conditions. For comorbidities, we collected ICD-10 codes from the Elixhauser Index (diabetes, hypertension, cancer, etc.). [25, 26] We used clinical judgement to identify other candidate variables available at the time of triage for ED visits. [8-10, 16, 22] We did not include laboratory test results, because most patients with suspected COVID-19 did not have such testing, and the receipt of testing likely signals that the patient had been identified as seriously ill by the treating physician.

J o u r n a l P r e -p r o o f

We included 42 variables in a least absolute shrinkage selection operator (LASSO) regression model to minimize collinearity and to avoid over-fitting. [27, 28] The LASSO model with the smallest predicted residual sum of squares was selected based on 10-fold cross validation. We next included all variables from the optimal LASSO regression in a logistic regression model. [22] Those variables which remained significant were included in the final risk score. To simplify the clinical score among common variables, we combined the Elixhauser diabetes and hypertension categories (complicated and uncomplicated) for analysis. We used standard National Institutes of Health (NIH) body mass index (BMI) categories [<18.5 (underweight), 18.5-24 (normal), 25-29 (overweight), 30-34 (obese class I), 35-39 (obese class II), and ≥ 40 (obese class III, or extreme obesity). [29] Vital sign thresholds were informed based on histograms stratified by deciles, but primarily were informed based on current clinical recommendations and the clinical judgement among co-authors. Age was categorized by decade, but given small sample sizes among extreme older ages, those ages above the highest significant age group were clustered into the highest significant category. We performed 10-fold cross validation to obtain stable parameter estimates for the variables in the logistic regression.

Standardized scores were created for each variable in the logistic regression model by dividing each variable coefficient by the smallest coefficient. For ease of use, variables were clustered into 5 categories including comorbidities, obesity/BMI, vital signs, age and sex (COVAS).

To assess discrimination, the area under the curve (AUC) was calculated, and a priori we determined that a final score with an AUC ≥ 0.80 would have meaningful accuracy to guide disposition decisions among suspected COVID-19 patients in the ED. [11, 14, 30, 31] We calculated sensitivity, specificity, positive predictive values (PPV), negative predictive values (NPV) at each threshold value for the COVAS score from ≥1 to ≥15. Positive likelihood ratios J o u r n a l P r e -p r o o f (PLR) and negative likelihood ratios (NLR) were also computed for each possible score. To assess model calibration, we evaluated plots of the observed and predicted values and calculated the Brier score. The final score was validated in a separate data set representing 20% of the study sample. We report the same performance metrics in the validation cohort as reported in the derivation sample.

Descriptive tables report the percentage of patients with a 7-day adverse event at each risk score.

We also performed subgroup analyses and assessed risk score performance among patients with a pneumonia diagnosis, or among those with a COVID-19 confirmed diagnosis at the index ED encounter. Last, we describe the number of patients hospitalized and discharged stratified by the primary outcome.

All analyses were performed using SAS (version 9.3; SAS Institute, Inc., Cary, NC). All tests of statistical significance were 2-sided with α=0.05. This study was approved by the [Institution name blinded for review] Institutional Review Board.

The study sample included 26,600 ED patient encounters (21,280 derivation and 5,320 validation cohorts) (eFigure 1). In the derivation cohort, the mean age (SD) was 50.7 (19.6), 9,053 (42.5%) were men, and 1,811 (8.5%) were severely obese (BMI ≥ 40) ( Table 1) LASSO regression identified 20 variables that were associated with the primary outcome. These variables were included in a logistic regression model and 15 variables from five categories, were included in the final comorbidity, obesity, vital signs, age and sex (COVAS) score.

Electrolyte disorders, cardiac arrhythmias, other neurological disorders, weight loss, congestive heart failure, coagulopathy and diabetes were the seven Elixhauser comorbidities included.

Severe obesity (BMI ≥ 40), respiratory rate (20-24 and ≥ 25) , oxygen saturation (93-94% and ≤ 92%), systolic blood pressure (≤ 105), fever (temperature ≥ 100.4 F), heart rate (≥ 110), age (50-59 and ≥ 60) and male sex were also included in the score (Table 2, Figure 1 ). The resulting COVAS score included points ranging from one to seven for each variable, with and an overall possible score from 0-34.

The AUC in the derivation cohort was 0.891 (95% CI, 0.880-0.901) ( Figure 2 ) with sensitivity ranging from 100% (Score 0) to 41.7% (Score ≥ 15) and specificity from 13.9% (score 0) to 96.8% (score ≥ 15) (eTable 5). A COVAS threshold of ≥ 5 had 95.1% sensitivity, 56.0% specificity, while a score of ≥ 11 had 67.8% sensitivity with 89.5% specificity. The Brier score in the derivation cohort was 0.032 demonstrating good calibration (eFigure 2). Similar performance of the COVAS score was observed in the validation cohort (AUC = 0.895, 95% CI, 0.874-0.916; Brier score=0.031) (eFigures 3 and 4, eTable 6).

In the subgroups of patients who were diagnosed with pneumonia, the COVAS score had an Rates of 7-day adverse events among patients in the derivation cohort were calculated for each COVAS score and ranged from 0% (COVAS score 0) to 35.7% (COVAS score ≥ 15) ( Table 3) .

Patients with a COVAS score ≤ 5 had ≤ 1.5% risk of an adverse event, which may represent a "low-risk" group, and patients with a score ≥ 12 have a ≥ 15% risk representing a "high-risk" cohort. Similar rates were observed for the validation cohort (eTable 8).

In this study we derived and validated a risk score (COVAS) to accurately predict death or respiratory decompensation within 7-days among a sample of ED patients with pneumonia or symptoms suspicious for COVID-19. The COVAS score uses variables (comorbidities, This study addresses a call for risk scores to guide clinical decisions in the care of patients during the COVID-19 pandemic, using methods that avoid concerns about the quality of previous reports. [23] A strength of our study is the use of a heterogeneous sample which is representative of the types of patients that frontline emergency, urgent care and primary care physicians are J o u r n a l P r e -p r o o f currently evaluating, and the 7-day timeline is more relevant than longer periods for acute care decisions. Our study also avoids the current challenges related to the availability of SARS CoV-2 testing, and with the uncertain accuracy of different testing strategies, [32] especially at a time when it is challenging to distinguish between COVID-19 and other infections. Targeting an ED patient population adds to the previous reports from hospitalized patient cohorts. [8, 10, 22] A limitation of this study and the derived COVAS score is the omission of diagnostic laboratory tests which have been associated with COVID-19 severity among hospitalized patients. We acknowledge that including laboratory tests may improve discrimination, but implementation of this approach would require that all patients receive lab tests which may not be necessary nor readily available in a pandemic. Also, the COVAS score may have different results in different patient populations or clinical settings; therefore, future research to validate this score will elucidate its generalizability. However, given the high AUC in our overall sample and subgroups, even a modest reduction in discrimination is likely to be useful. Additionally, the COVAS score does not include social risks that may predict adverse outcomes among at-risk populations, such as those with housing instability or food insecurity. [33] Our study population uses only the initial ED patient encounter for risk assessment, therefore, patients with a return visit may have slightly different risks for an adverse event. Our patient population is also diverse racially and ethnically (45% Hispanic and 13% black) which is a strength of our study, but the COVAS score may have different results in more homogenous populations. Lastly, calculating comorbidities based on ICD codes in the preceding 12 months may pose challenges to physicians calculating the score without electronic assistance. Therefore, we recommend using decision support within an electronic health record to facilitate the accurate calculation of the COVAS score.

The COVAS score discriminates as well, or better, than many scores used in routine clinical practice to inform acute care decisions. Commonly used clinical risk scores like the pneumonia severity index (AUC 0.81), CRB65 (AUC 0.79) and CURB-65 (AUC 0.80), which are routinely used in clinical care [11] [12] [13] to predict 30-day mortality [15] , are less accurate than the COVAS score for their specific indications. Of note, a more complicated score with many more variables, including a number of lab test results was not as accurate for predicting adverse outcomes among Chinese patients hospitalized with COVID-19. [22] Since the derivation of the COVAS score, all 15 EDs included in this study implemented the score into routine practice using an automated decision support calculation within the EPIC based electronic health record. From the time of our reported study, the COVAS score was further validated among ED patients who were tested for COVID-19 during a time in Southern California when the incidence and prevalence increased among the study EDs. This additional validation among our study EDs using the same study population in between May and June 2020 resulted in an AUC of 0.822 (95% CI 0.811-0.833), and with much increased capacity to test it performed well among those tested via SARS CoV-2 PCR (n=18,379, AUC 0.792, 95% CI 0.780-0.805) and was best among those whose COVID-19 test was positive (n=2,091, AUC 0.841, 95% CI 0.812-0.869).

Though the COVAS score requires future research to validate its performance among different patient populations in different health care settings, we believe it provides useful information to frontline physicians to assist with risk stratification and disposition decisions. A challenge for any risk score is choosing the appropriate threshold (sensitivity and specificity) to guide recommendations for patient care, because both the frequency and consequences of adverse events need to be considered. Using information about adverse events in conjunction with J o u r n a l P r e -p r o o f COVAS score sensitivity/specificity tables, physicians and systems can choose different thresholds for categorizing patients into low, moderate and high-risk groups. This may inform disposition decisions, in the same way other risk scores have been applied in the ED.

[34] One potential application of the COVAS score may be to objectively identify a moderate risk group of patients who may safely avoid hospitalization, but who may benefit from home O2 monitoring and oxygen supplementation.

In summation, among patients visiting an ED for confirmed or suspicious COVID-19 symptoms, the COVAS score, using information available at the time of patient presentation, may help frontline physicians to identify patients who will experience a serious adverse event within 7days. Future research is needed to further validate this score in other patient populations and clinical settings. List of symptoms and associated ICD-10 codes eTable 2.

CPT and ICD-10 codes for respiratory events eFigure 1. Flow diagram of the study cohort eTable 3.

Primary and secondary outcomes among the study cohort eTable 4.

Patient characteristics of validation cohort eTable 5.

Predictive accuracy of the COVAS score in the derivation cohort eFigure 2. Decile calibration plot of observed and predicted values for derivation cohort eFigure 3. The COVAS score performance for the validation sample eFigure 4. Decile calibration plot of observed and predicted values for validation cohort eTable 6.

Predictive accuracy of risk score in the validation cohort eTable 7.

Predictive accuracy of risk score among pneumonia and COVID-19 subgroups in the derivation cohort eTable 8.

Adverse 

Fair Allocation of Scarce Medical Resources in the Time of Covid-19

mortality and health-care resource availability

Potential preanalytical and analytical vulnerabilities in the laboratory diagnosis of coronavirus disease 2019 (COVID-19)

Clinical features of patients infected with 2019 novel coronavirus in Wuhan

Clinical course and risk factors for mortality of adult inpatients with COVID-19 in Wuhan

Hospitalized Patients With 2019 Novel Coronavirus-Infected Pneumonia in Wuhan

Case-Fatality Rate and Characteristics of Patients Dying in Relation to COVID-19 in Italy

Characteristics and Outcomes of 21 Critically Ill Patients With COVID-19 in Washington State

Presenting Characteristics, Comorbidities, and Outcomes Among 5700 Patients Hospitalized

With COVID-19 in the New York City Area

Characteristics of Hospitalized Adults With COVID-19 in an Integrated Health Care System in California

CURB-65 Performance Among Admitted and Discharged Emergency Department Patients With Community-acquired Pneumonia

pneumonia severity assessment adapted for electronic decision support

Safety and efficacy of CURB65-guided antibiotic therapy in community-acquired pneumonia

Severity assessment tools for predicting mortality in hospitalised patients with communityacquired pneumonia. Systematic review and meta-analysis

Increasing outpatient treatment of mild communityacquired pneumonia: systematic review and meta-analysis

Knowledge, attitude and practice regarding COVID-19 among health care workers in Henan

COVID-19: protecting health-care workers

Drawing on Israel's Experience Organizing Volunteers to Operationalize Drive-Through Coronavirus Testing Centers

Drive-Through Screening Center for COVID-19: a Safe and Efficient Screening System against Massive Community Outbreak

Innovative screening tests for COVID-19 in South Korea

Laboratory abnormalities in patients with COVID-2019 infection

Development and Validation of a Clinical Risk Score to Predict the Occurrence of Critical Illness in Hospitalized Patients With COVID-19

Prediction models for diagnosis and prognosis of covid-19 infection: systematic review and critical appraisal

Community Prevalence of SARS-CoV-2 Among Patients With Influenzalike Illnesses Presenting to a Los Angeles Medical Center

Risk adjustment performance of Charlson and Elixhauser comorbidities in ICD-9 and ICD-10 administrative databases

Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data

Standardization and the Group Lasso Penalty

The lasso method for variable selection in the Cox model

NHLBI Obesity Education Initiative Expert Panel on the Identification E, and Treatment of Obesity in Adults. Clinical Guidelines on the Identification, Evaluation, and Treatment of Overweight and Obesity in Adults: The Evidence Report. National Heart Lung, and Blood Institute

Chest pain in the emergency room: a multicenter validation of the HEART Score

The HEART score for the assessment of patients with chest pain in the emergency department: a multinational validation study

The Laboratory Diagnosis of COVID-19 Infection: Current Issues and Challenges

Understanding High-Utilizing

Author Contributions: ALS and MKG conceived the study and obtained research funding.BZH collected and analyzed the data. CN and ST assisted with statistical advice. All included authors assisted with the study design and interpretation of results. ALS drafted the manuscript, and all authors contributed substantially to its revision. ALS takes responsibility of the manuscript as a whole.J o u r n a l P r e -p r o o f J o u r n a l P r e -p r o o f