key: cord-0904244-gmumunqz
authors: Cahan, Amos; Gottesman, Tamar; Katz, Michal Tzuchman; Masad, Roee; Azulay, Gal; Dicker, Dror; Zeidman, Aliza; Berkov, Evgeny; Tadmor, Boaz; Lev, Shaul
title: Development and validation of a knowledge-driven risk calculator for critical illness in COVID-19 patients
date: 2020-09-23
journal: Am J Emerg Med
DOI: 10.1016/j.ajem.2020.09.051
sha: 0a723741400d26e4db1fe09dc4698247433013a0
doc_id: 904244
cord_uid: gmumunqz

Facing the novel coronavirus disease (COVID-19) pandemic, evidence to inform decision-making at all care levels is essential. Based on the results of a study by Petrilli et al., we have developed a calculator using patient data at admission to predict critical illness (intensive care, mechanical ventilation, hospice care, or death). We report a retrospective validation of the calculator on 145 consecutive patients admitted with COVID-19 to a single hospital in Israel. Despite considerable differences between the original and validation study populations, of 18 patients with critical illness, 17 were correctly identified (sensitivity: 94.4%, 95% CI, 72.7%–99.9%; specificity: 81.9%, 95% CI, 74.1%–88.2%). Of 127 patients with non-critical illness, 104 were correctly identified. Our results indicate that published knowledge can be reliably applied to assess patient risk, potentially reducing the cognitive burden on physicians, and helping policymakers better prepare for future needs.

Facing the rapidly spreading novel coronavirus disease (COVID-19), evidence to inform decision-making at both the clinical and policy-making level is highly needed [1] . In an impressive work, Petrilli and coworkers [2] have recently reported a multivariable analysis of data collected on 2,729 hospitalized patients with COVID-19 at an academic health system in New York City (NY) population), to predict critical illness (defined as a composite of care in the intensive care unit, use of mechanical ventilation, discharge to hospice, or death). Based on this analysis, we have developed a computed calculator for risk stratification of hospitalized COVID-19 patients. Since rates of severe disease and mortality vary widely by country [3] , we aimed to validate the risk calculator on a population of COVID-19 patients in Israel (IL) population) .

We used the odds ratios (OR's) obtained by Petrilli Thousands of papers have been published to date on COVID-19. For clinicians, keeping current on medical literature is challenging in normal times but even more demanding during the COVID-19 pandemic, when their abilities are stretched to the limit. Moreover, given the limited capacity of humans (including physicians) to apply probabilistic reasoning [5] , integration of published evidence probably remains mostly at the intuitive level in the minds of clinicians. This state of affairs calls for equipping clinicians with reliable tools to properly evaluate the abundant empirical knowledge, and properly weigh it against their own patients' data. Many current EHR systems document the information required by the calculator in a machine-readable format, allowing for Our study shows that risk assessment could to be done automatically (without active physician involvement),. Automatic extraction of information and risk calculation performed in the background as patients present to the hospital can reduce reducing the time and effort required from physicians for risk assessment, . It can also assure that policy makers are provided a complete view of predicted disease burden for continuous monitoring.

Public health measures, such as quarantine and shelter-in-place, are guided by the capacity of the healthcare system, with ICU beds and ventilator availability being the "rate limiting factor".

As severe disease often develops during the second week of illness, there is a reported 12 day average lag between illness onset and ICU admission or the development of acute respiratory J o u r n a l P r e -p r o o f Journal Pre-proof distress syndrome (ARDS) [6] . In an effort to avoid overwhelming the healthcare system capacity, further lifting of restrictions on social interactions is thus delayed until the effects of policy changes can be measured. The turnaround time for policy-makers to get feedback on policy changes is therefore around two weeks. Our results provide reason to believe that future critical illness can be reliably predicted at the time of admission.Such predictions may be used in triage to make sure that high risk patients remain where medical care is rapidly available, or to select patients for (investigational) interventions. At the institutional and healthcare system level, predicting the future burden of critical illness could improve allocation of resources (e.g., personnel and supplies) to preempt shortages. At the State and National levels, such predictions could shorten the feedback turnaround time, allowing policy makers to effectively flatten the epidemic curve and avoid breakdown of medical care, while minimizing restrictions on the workforce to curb the financial crisis.

The proposed risk calculator is limited in the sense that I Our work has several limitations. It is based on a single study, which, albeit large in scale, includes patients from a single metropolitan area, presenting during a relatively short period of time, in which the local healthcare system was heavily burdened by scores of severely ill patients. The validation population was relatively small and taken from a single hospital. The paucity of critically ill patients in our cohort may contribute to the high area under the ROC curve. As the threshold score for predicting critical illness was empirically determined from the validation population, results may reflect over-fitting.

Finally, our dataset lacked information on some laboratory tests which were included in the original analysis, however this could have weakened our results. Further research will be needed to validate its performance our findings in other patient populations. Another limitation is the lack of information on some laboratory tests which were included in the original analysis, however this could have weakened our results.

Journal Pre-proof CONCLUSION We believe that computer-aided risk assessment is a means to put research-derived knowledge to work in the clinical setting. As new research is published, other calculators could be developed. Based on local circumstances, a calculator using predictors from the most relevant study could be used. Moreover, several calculators could be applied in parallel, and calculator voting used to derive potentially more robust predictions. Integration with EHR systems can facilitate automatic data extraction, which could make the use of such tools more user friendly and practical.

J o u r n a l P r e -p r o o f 

Defining the Epidemiology of Covid-19 -Studies Needed

Factors associated with hospital admission and critical illness among 5279 people with coronavirus disease

World Health Organization COVID-19 situation reports

R: A language and environment for statistical computing. R Foundation for Statistical Computing

An evaluation of clinicians' subjective prior probability estimates

Clinical course and risk factors for mortality of adult inpatients with COVID-19 in Wuhan, China: a retrospective cohort study -The Lancet

Writing-Original draft

Data Curation, Validation; Dror Dicker: Validation, Writing -Review & Editing, Supervision; Aliza Zeidman: Validation, Writing -Review & Editing, Supervision; Evgeny Berkov: Validation, Writing -Review & Editing, Supervision

Shaul Lev: Validation, Writing -Review & Editing

All other authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work