key: cord-0910195-iepo07ph authors: Wong, Andrew; Cao, Jie; Lyons, Patrick G.; Dutta, Sayon; Major, Vincent J.; Ötleş, Erkin; Singh, Karandeep title: Quantification of Sepsis Model Alerts in 24 US Hospitals Before and During the COVID-19 Pandemic date: 2021-11-19 journal: JAMA Netw Open DOI: 10.1001/jamanetworkopen.2021.35286 sha: 642b8c75ac3ae00c8ca52d9601b43bdb0ebcf5f9 doc_id: 910195 cord_uid: iepo07ph This descriptive study evaluates the association between nursing reports of sepsis overalerting and alert volume by quantifying the number of alerts generated by the Epic Sepsis Model at 24 US hospitals before and during the COVID-19 pandemic. In a recent single-center evaluation, the ESM was found to generate alerts in 18% of hospitalized patients at the University of Michigan (during a "silent" evaluation using an alerting threshold of 6) when examining data from 2018 to 2019 while missing 67% of sepsis cases (i.e., a sensitivity of 33%), where sepsis was defined by a composite of the Centers for Disease Control and Prevention surveillance criteria and (2) the Centers for Medicare and Medicaid Services SEP-1 Criteria. This same threshold of 6 was selected in this study for three reasons. First, it is within the range (5-8) recommended by the model developer. 1 Second, multiple preprints, conference proceedings, and publications evaluating the ESM have relied on thresholds of 5 or 6. [1] [2] [3] [4] And third, it is the threshold that has been in clinical use to generate alerts at the University of Michigan. 1 As compared to an ESM threshold of 5, a threshold of 6 would be expected to generate fewer alerts and thus is a more conservative choice. At each hospital included in this analysis, the ESM runs at intervals between 15-20 minutes on patients in the emergency department and inpatient settings, including both intensive care unit (ICU) and non-ICU settings. Using an alerting threshold of 6, we calculated the number of patients in whom the ESM would have generated alerts based on prospectively calculated ESM scores. Alerts were generated on three inpatient units at the University of Michigan during this time period, though ESM scores were calculated for all adult emergency department and inpatient settings. Although no actual alerts were generated at New York University Langone Health, Mass General Brigham, and BJC HealthCare during this time, the prospective calculation of ESM scores at these institutions allowed us to calculate the number of alerts that would have been generated. Assuming that alerts would be muted and thus could only be generated at most once per day for a given patient even if they exceeded the alerting threshold multiple times, we first calculated how many alerts would have been calculated per each calendar week (Sunday to Saturday) for patients, with a maximum of 7 alerts per week possible for each patient. We then calculated the mean number of alerts per day by dividing the total number of weekly alerts by 7. To calculate the proportion of patients generating an alert, we divided the number of patients generating an alert by the number of patients scored by the ESM. The Figure depicts the total number of patients evaluated by the ESM across the 24 hospitals in our study. While this is labeled "hospital census," this number is actually larger than the actual hospital census because it includes all patients for whom the ESM was calculated (and thus were alert-eligible), including patients evaluated in the emergency department who were discharged. External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients 1235: Validating the Epic Sepsis Inpatient Predictive Analytic Tool as a Sepsis Alert System Accuracy of the epic sepsis prediction model in a regional health system How Useful Is the Epic Sepsis Prediction Model for Predicting Sepsis?