key: cord-0744679-0xo3yio2
authors: Alencar, Julio; Marina Gómez Gómez, Luz; Cortez, Andre Lazzeri; Possolo de Souza, Heraldo; Levin, Anna Sara; Salomão, Matias Chiarastelli
title: Performance of NEWS, qSOFA, and SIRS Scores for Assessing Mortality, Early Bacterial Infection, and Admission to ICU in COVID-19 Patients in the Emergency Department
date: 2022-03-02
journal: Front Med (Lausanne)
DOI: 10.3389/fmed.2022.779516
sha: 3e0b7a9e50399199b998160a34466f168bfabdb6
doc_id: 744679
cord_uid: 0xo3yio2

SARS-CoV-2 infection has a wide spectrum of presentations, from asymptomatic to pneumonia and sepsis. Risk scores have been used as triggers for protocols that combine several interventions for early management of sepsis. This study tested the accuracy of the score SIRS, qSOFA, and NEWS in predicting outcomes, including mortality and bacterial infection, in patients admitted to the emergency department (ED) during the COVID-19 pandemic. We described 2,473 cases of COVID-19 admitted to the ED of the largest referral hospital for severe COVID-19 in Brazil during the pandemic. SIRS, qSOFA and NEWS scores showed a poor performance as prognostic scores. However, NEWS score had a high sensitivity to predict in-hospital death (0.851), early bacterial infection (0.851), and ICU admission (0.868), suggesting that it may be a good screening tool for severe cases of COVID-19, despite its low specificity.

INTRODUCTION SARS-CoV-2 infection has a wide spectrum of presentations, from asymptomatic to severe cases of viral pneumonia and Acute Respiratory Distress Syndrome (1) . Considering the pathophysiology and the clinical manifestations, some COVID-19 patients meet the definition of sepsis, described as an unregulated inflammatory host response to infection that results in organ failure and risk of death (2, 3) .

This concept of sepsis is recent and was updated after a better understanding of pathophysiological events (4) . In a consensus definition from 1991, sepsis was defined as a systemic inflammatory response (SIRS-Systemic Inflammatory Response Syndrome) caused by infection (5, 6) . The diagnosis of sepsis was made in patients with suspected or confirmed infection and two of four criteria: abnormalities in body temperature, tachypnea, tachycardia and leukocytosis (6) .

More recently, a new consensus, Sepsis-3, defines sepsis as organ dysfunction, represented as at least 2 points in the Sequential Organ Failure Assessment (SOFA) score in patients with suspected or confirmed infection (3) . In the Emergency Department (ED), the use of a sepsis-related organ failure prediction tool (qSOFA) can help identify patients at high risk of death (3) . Moreover, authors have compared the accuracy of scores based on physical examination for diagnosing sepsis in patients admitted to the ED with suspected or confirmed infection, and the NEWS (National Early Warning Score) score has been shown superior to SIRS and qSOFA (7) .

These three tools have been used as triggers for protocols that combine several interventions for early management of sepsis, including the use of antibiotics (8) . Although there is still controversy about how quickly antibiotics should be administered to septic patients in general (9) , COVID-19 is a viral disease without indication for antibiotic treatment (10) , and there is concern that the use of antibiotics may exacerbate antimicrobial resistance without a clinical benefit (11) .

Thus, we designed a study to test the accuracy of the scores SIRS, qSOFA, and NEWS in predicting outcomes, including mortality and bacterial infection, in patients admitted to the ED during the COVID-19 pandemic.

We conducted a retrospective single center cohort study from March to August 2020 at the ED in Hospital das Clínicas, in São Paulo, Brazil. This is an academic tertiary-care hospital affiliated to São Paulo University with 2,200 beds, comprising five institutes and two auxiliary hospitals. In March 2020, the main institute was converted to a COVID-19-only facility, dedicating 900 beds to the care of infected patients. Admissions to the COVID-19 Institute were centrally managed by the Regulatory Central of the State of São Paulo, and severely ill patients are preferably referred to the hospital We included all consecutive adult patients (≥ 18 years) with confirmed COVID-19, defined as at least one positive result using reverse transcriptase-polymerase chain reaction (Rt-PCR) obtained from nasopharyngeal swabs or bronchial secretions (12) .

We excluded patients for whom we could not calculate scores due missing data. Patient data were collected through electronic medical records, and a database was built using REDCap software (13) .

We applied risk assessment scores according to patients' admission variables. The positive qSOFA cutoff was 2 or greater (3), NEWS score was classified into low risk (1-3 points) and high risk (four or more points) of sepsis (7) , and the positive cutoff for SIRS was 2 or greater (5) .

Besides the SIRS, qSOFA and NEWS variables, we also collected data on demographics (age, sex), clinical history (previous diagnoses and medications, time of symptoms on admission, physical examination, supplemental oxygen), laboratory tests routinely collected on admission (complete blood count, D-dimer, C-reactive protein, urea, creatinine, fibrinogen, lactate), variables of SAPS3, treatment (antibiotics, anticoagulants and corticosteroids), and outcomes (length of hospital stay, dialyses, invasive mechanical ventilation and inhospital mortality). We considered with severe COVID-19, patients who had SpO2 < 90% on room air, clinical signs of pneumonia, or a respiratory rate >30 breaths/min (10) .

The primary outcome was in-hospital mortality within 30 days after admission. Secondary outcomes were admission to intensive care unit (ICU) within 7 days from admission, and early bacterial infection confirmed by bacterial growth in culture.

We defined as early bacterial infection any positive culture of blood, urine or tracheal secretions in the first 7 days of hospitalization. We considered contaminants the coagulasenegative Staphylococci, Corynebacterium species, Bacillus spp. other than Bacillus anthracis, Cutibacterium acnes, Micrococcus spp., viridans group streptococci, and Clostridium perfringens (14) if isolated in only one culture of the patient. The contaminants were excluded.

All patients received standard care, according to the institutional protocol. In the emergency department, this included oxygen supplementation, dexamethasone and antibiotics.

The study protocol was approved by the Local Ethics Committee (number: 3.990.817; CAAE: 30417520.0.0000.0068), which waived the need for written informed consent. We adhered to Transparent Reporting of a Multivariable Prediction for Individual Prognosis or Diagnosis (TRIPOD) guidelines (15) .

Mean, standard deviation (SD), median, and interquartile range (IQR) were used for descriptive statistics according to variable distribution.

Model predictive performance was assessed with the area under the receiver operating characteristics curve (AUROC). Clinical utility was analyzed using sensitivity, specificity, positive predictive values (PPV), negative predictive values (NPV), positive likelihood ratio, negative likelihood ratio, and precision recall curves. Confidence intervals (95%) were calculated after 1,000 bootstrap re-samples (16) (17) (18) (19) .

SIRS, qSOFA and NEWS's variables were submitted to bivariate analysis and factors with statistical significance (p < 0.05) were submitted to logistic regression using multivariate analysis by calculating the Lassos lambda coefficient for the outcomes of in-hospital death, ICU admission, and early bacterial infection.

A Bonferroni correction was used to account for multiple comparisons across the pre-specified outcomes and subgroup analyses.

All statistical analyses were performed using the software R version 3.6.2.

A total of 3,021 patients diagnosed with COVID-19 in the Emergency Department were included in the study, of which 2,473 patients had enough data to calculate the scores. To analyze the predictive power of the three scores, data from these 2,473 individuals were used (Figure 1) .

The median age of patients was 61.6 years, 57% were male, and the median length of hospital stay was 14 days. The median SAPS3 was 65, and the median time between onset of symptoms of COVID-19 and hospitalization was 8 days. A total of 1,904 patients (77%) required ICU admission. In-hospital mortality was 39% (968 patients) (Tables 1, 2). Cultures collected within the first 7 days of hospitalization were available for 1,190 patients, and 684 (62%) of these patients had an infectious agent isolated. The most commonly isolated agents were Staphylococcus aureus (112 isolates), Candida albicans (109 isolates), Pseudomonas aeruginosa (78 isolates), and Acinetobacter baumannii (69 isolates) ( At admission, 1,364 (55%) had positive SIRS, 820 (33%) had positive qSOFA, and 2005 (81%) had high risk NEWS. In-hospital mortality frequency based on these cutoffs were: 629 (46%) for SIRS; 265 (32%) for qSOFA, and 859 (43%) for NEWS. The frequency of patients with early bacterial infection based on the cut-offs were: 423 (62%) for SIRS; 211 (66%) for qSOFA; and 582 (61%) for NEWS ( Table 4 ).

The AUROC for each score to predict mortality was: 0.58 for SIRS, 0.55 for qSOFA, and 0.56 for NEWS. After corrections, only AUROC values for SIRS and qSOFA were considered statistically different (p = 0.003).

We found higher sensitivity for NEWS 0.89 (CI 95% 0.87-0.91) and its NPV was 0.77 (CI 95% 0.73-0.80). However, NEWS had a lower specificity, 0.24 (CI 95% 0.22-0.26) and lower PPV 0.43 (CI 95% 0.42-0.44) (Table 5, Figure 2 ).

There was no difference between the AUROC of the three scores to predict bacterial infection, with poor performance for the three. The NEWS score presented the best sensitivity [0.85 (CI 95% 0.82-0.88)], and qSOFA the best specificity [0.75 (CI 95% 0.71-0.79)] (Table 5, Figure 3 ).

There was also no difference between the AUROC of the three scores. The NEWS score demonstrated the best sensitivity [0.87, CI 95% (0.85; 0.88)], and SIRS [0.62, CI 95% (0.58; 0.66)] the best specificity (Figure 4 ).

The factors associated with in-hospital death were: use of steroids, cancer, male sex, and immunosuppression. Protective factors were: use of ACEi, rheumatologic disease, and hematologic disease ( Table 6) .

The factors associated with ICU admission were: dialysis, supplemental oxygen therapy, use of steroids, anticoagulation, cardiovascular disease, and immunosuppression ( Table 6) .

All scores show low performance on precision-recall. They only presented a high recall value, but with small precision values. According to precision-recall, the score with the best performance is the qSOFA, which has the best specificity ( Table 4 ). The scores also show a low performance to predict positive culture of patients with COVID-19.

High precision values only are present with low recall (Figures 5, 6 ). 

In this study we described 2.473 cases of COVID-19 admitted to the emergency department of a tertiary hospital during the pandemic, in order to evaluate the performance of SIRS, qSOFA and NEWS scores to predict in-hospital mortality, early bacterial infection, and ICU admission. Our findings suggest a poor performance of the 3 prognostic scores. However, they indicate a possible use of the NEWS as a screening tool for severe cases of COVID-19, given its high sensitivity to predict in-hospital death, early bacterial infection and ICU admission, despite its low specificity. To our knowledge, this is the largest study to assess the performance of SIRS, qSOFA and NEWS scores in patients with COVID-19. Other authors have also evaluated prognostic scores to predict unfavorable outcomes for patients with COVID-19, but few have performed this assessment in the emergency department. Prognostic scores are tools that, in this context, help to make better-informed decisions (16, 17) . In favor of the NEWS score, we must consider that this tool is already widely validated for the care of patients with sepsis. And, although not ideal, its high sensitivity allows NEWS to be used as a screening tool for cases that may progress badly during hospitalization. We also evaluated the performance of these tools to predict early bacterial infection, with similar results and NEWS also presented higher sensitivity than SIRS and qSOFA.

Our results are in agreement with the literature. The first study which systematically evaluated the use of NEWS2 for severe COVID-19 outcomes was carried out in five hospitals in the United Kingdom, one hospital in Norway, and two hospitals in Wuhan, China. Their results demonstrated a poor-to-moderate discrimination for 14-day ICU and death (AUC between 0.63 and 0.77 according to center) (20) . Higher NEWS' cutoffs probably are better to predict COVID-19 outcomes. At Emergency Department, NEWS-2 score ≥ 6 at admission predicted severe disease with 80.0% sensitivity and 84.3% specificity (AUC 0.822, 95% CI 0.690-0.953), and was higher than qSOFA score ≥ 2 (AUC 0.624, 95% CI 0.446-0.810, p < 0.05) (21) . Although this study was conducted in an emergency department of a single center, this hospital was the main state referral for severe COVID-19. São Paulo has a population over 44 million, and 600 of the 6,000 critical COVID-19 care beds were located in this hospital. Because of this, our sample represents the selection of the most severe cases of the State of São Paulo, one of the world's epicenters of the pandemic at that time. This is evident when evaluating the median SAPS 3 value of 68 for patients admitted to the emergency department, which would have an expected mortality of 66.8% for patients seen in Latin America. We highlight that tools presented lower AUROCs than those found in some studies (12, 13, 16, 22, 23) , mainly due to the lower specificity and PPV values. This may have happened because of the high severity of the cases. In a scenario with a higher prevalence of milder cases, there would be a better chance of detecting survivors, resulting in higher specificity and PPV values.

The incidence of early bacterial infection was high, 59% among those who collected cultures, and 26% among the 2,403 patients studied. This result is much higher than that found in other studies, 3-8% (24, 25) . It would be expected that these infections had occurred later, but the median time of COVID-19 symptoms on admission was 8 days. This finding may be one of the factors related to the greater severity of our patients.

There were no factors strongly associated with early bacterial infection, but antibiotic use was associated with a reduced risk. This finding may be explained by the use of antibiotics resulting in negative cultures. Despite the high incidence of early bacterial infection, it is important to note that the use of antibiotics was not associated with lower risk of admission to the ICU or death. It was not possible to analyze the risk of developing infection by resistant bacteria in our study, but the indiscriminate use of antibiotics has been shown to be associated with the emergence of resistance, and the prescription of these drugs should be done cautiously and rationally.

Among the factors associated with in-hospital death, we found the use of steroids to be the most important factor. This may represent a bias as steroids are prescribed for severe COVID-19 as well as for comorbidities such as cancer and immunodeficiency. Paradoxically, rheumatologic disease and hematologic disease were not associated with death. The latter, it was not even a factor associated with admission to the ICU. These patients were prioritized for hospital care, which may have positively influenced the outcome, despite their potentially higher risk (26, 27) . The use of ACEi was a protective factor against death in our study, as demonstrated by other authors (28, 29) .

The most important factors associated with admission to the ICU admission were factors associated with the need for intensive support, such as dialysis, or the severity of COVID-19 (supplemental oxygen therapy, use of steroids and anticoagulation). The presence of cardiovascular disease and immunodeficiency were also factors associated with admission to the ICU. Factors not associated with hospitalization in ICU were: cancer, dementia, hematologic disease and asthma. Although not expected, patients with asthma had lower risk of hospitalization in ICU, as demonstrated in other studies (30) (31) (32) .

This study has limitations. Data for this study were collected prospectively, but their analysis was performed later, and it was not possible to obtain retrospectively some data that were not collected initially. For instance, it was not possible to collect data on Glasgow Coma Scale for all patients, as this information was sometimes described as mental status alert, somnolent, and unconsciousness in the electronic medical record. We considered any positive culture as bacterial infection. It was not possible to evaluate the clinical features of the patients, so patients that were only colonized may have been considered as infected in our definition. We could not evaluate the antimicrobial resistance profiles in our study, so we could not analyze the impact of antibiotic use. This study was performed in a single-center, which is a limitation. However, this center was the reference hospital for severe cases of COVID-19 in the State of São Paulo, so we feel that it was broadly representative of the state which was hit hard by the pandemic. Our cases reflected the selection of the most severe cases in the state, actually representing a wider population than the study design would suggest, especially among critically ill patients in the emergency department.

In conclusion, for patients with severe COVID-19 admitted to the emergency department, SIRS or qSOFA did not perform well in predicting in-hospital mortality, early bacterial infection, or admission to the ICU. However, high sensitivity in predicting these three outcomes suggests that the NEWS score can be useful as a screening tool.

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

The studies involving human participants were reviewed and approved by Hospital of Clinics, Faculty of Medicine, University of São Paulo, Brazil. Written informed consent for participation was not required for this study in accordance with the national legislation and the institutional requirements.

JA and MS conceptualized the project, analyzed the results, wrote the first draft, and wrote the final manuscript. LM performed the statistical analysis. AC collected data. HP and AL conceptualized the project and reviewed the manuscript. All authors contributed to the article and approved the submitted version.

A novel coronavirus from patients with pneumonia in China

Pathological inflammation in patients with COVID-19: a key role for monocytes and macrophages

The Third International Consensus definitions for sepsis and septic shock (Sepsis-3)

Developing a new definition and assessing new clinical criteria for septic shock: for the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3)

Definitions for sepsis and organ failure and guidelines for the use of innovative therapies in sepsis

International sepsis definitions conference

Comparison of SIRS, qSOFA, and NEWS for the early identification of sepsis in the Emergency Department

Association of a care bundle for early sepsis management with mortality among patients with hospital-onset or community-onset sepsis

The impact of timing of antibiotics on outcomes in severe sepsis and septic shock

World Health Organization. Clinical Management of COVID-19

COVID-19 pneumonia and the appropriate use of antibiotics

Detection of 2019 novel coronavirus (2019-nCoV) by real-time RT-PCR

The REDCap consortium: building an international community of software platform partners

Practical guidance for clinical microbiology laboratories: a comprehensive update on the problem of blood culture contamination and a discussion of methods for addressing the problem

Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement

Interval estimation for a binomial proportion

Confidence intervals for predictive values with an emphasis to case-control studies

Area under the precision-recall curve: point estimates and confidence intervals

The relationship between precision-recall and ROC curves

Evaluation and improvement of the National Early Warning Score (NEWS2) for COVID-19: a multi-hospital study

National Early Warning Score 2 (NEWS2) on admission predicts severe disease and in-hospital mortality from Covid-19 -a prospective cohort study

The performance of the National Early Warning Score and National Early Warning Score 2 in hospitalised patients infected by the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2)

Prognostic accuracy of the SIRS, qSOFA, and NEWS for early detection of clinical deterioration in SARS-CoV-2 infected patients

Empiric antibacterial therapy and community-onset bacterial coinfection in patients hospitalized with Coronavirus Disease 2019 (COVID-19): a multi-hospital cohort study

Antibiotic prescribing in patients with COVID-19: rapid review and meta-analysis

Risk factors and outcome of COVID-19 in patients with hematological malignancies

The outcome of COVID-19 in patients with hematological malignancy

Association between renin-angiotensin-aldosterone system inhibitors and clinical outcomes in patients with COVID-19: a systematic review and meta-analysis

Correction: mortality and other outcomes of patients with coronavirus disease pneumonia admitted to the emergency department: a prospective observational Brazilian study

Allergic disorders and susceptibility to and severity of COVID-19: a nationwide cohort study

Effect of asthma and asthma medication on the prognosis of patients with COVID-19

SARS-CoV-2 pneumonia in hospitalized asthmatic patients did not induce severe exacerbation

Fabiane Yumi Ogihara Kawano fabiane.kawano@hc.fm.usp.br

Izabel Cristina Rios Izabel Cristina Rios izabel

Izabel Marcilio Izabel Marcilio izamarcilio@gmail

Rodrigo Antonio Brandão Neto rodrigo.neto@hc.fm.usp.br

Jacon suze.jacon@hc.fm.usp.br

Leila Letaif Leila Harima leila

Maria Cristina Peres Braido Francisco maria.braido@hc.fm.usp.br

Ésper Georges Kallas esper

Barros-Filho tarcisio

Thaís Guimarães Thaís Guimarães thais

FUNDING LM acknowledges the partial support of FAPESP (Fundação de Amparo à Pesquisa do Estado de São Paulo), grant 2019/23078-1. HP acknowledges the partial support of FAPESP, grants 2016/14566-4 and 2020/04738-8.

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Name for publication e-mail ORCID-ID