key: cord-0829949-pi7dbcg2
authors: Pollock, Benjamin D.; Carter, Rickey E.; Dowdy, Sean C.; Dunlay, Shannon M.; Habermann, Elizabeth B.; Kor, Daryl J.; Limper, Andrew H.; Liu, Hongfang; Franco, Pablo Moreno; Neville, Matthew R.; Noe, Katherine H.; Poe, John D.; Sampathkumar, Priya; Storlie, Curtis B.; Ting, Henry H.; Shah, Nilay D.
title: Deployment of an Interdisciplinary Predictive Analytics Task Force to Inform Hospital Operational Decision-Making During the COVID-19 Pandemic
date: 2020-12-30
journal: Mayo Clin Proc
DOI: 10.1016/j.mayocp.2020.12.019
sha: c0b1b9b63b5923f240e1aa0b3c6e0791ffb49077
doc_id: 829949
cord_uid: pi7dbcg2

In March 2020, our institution developed an interdisciplinary predictive analytics task force to provide COVID-19 hospital census forecasting to help clinical leaders understand the potential impacts on hospital operations. As the situation unfolded into a pandemic, our task force provided predictive insights through a structured set of visualizations and key messages that have helped the practice to anticipate and react to changing operational needs and opportunities. The framework shared here for the deployment of a COVID-19 predictive analytics taskforce could be adapted for effective implementation at other institutions to provide evidence-based messaging for operational decision-making. For hospitals without such a structure, immediate consideration may be warranted in light of the devastating COVID-19 ‘third-wave’ which has arrived for winter 2020-2021.

Prior to the onset of the coronavirus disease-2019 (COVID- 19) pandemic in the United States (US), the American Hospital Association (AHA) hosted a webinar in which a 'best guess epidemiology' scenario suggested that COVID-19 cases could double every 7-10 days and lead to 5 million COVID-19 hospitalizations across the US, while warning that hospitals should prepare for a healthcare burden ten times worse than a severe flu season. 1 On the date of that webinar (Feb. 26, 2020), there had been approximately 200 confirmed COVID-19 cases in the US. By the following week, the US confirmed case count had climbed over 1,000, indicating an alarming potential for a quicker case doubling time (CDT) than the predicted 7-10 days. 2 This explosive early trajectory sparked a national discussion that the pandemic could soon overwhelm US health system capacities and pose a threat to the health and safety of both COVID-19 and non-COVID-19 hospitalized patients. At that time, this exact scenario was unfolding simultaneously across northern Italy, with overwhelmed hospitals reportedly housing patients on mattresses on the floor while struggling to provide basic services such as palliative care and child delivery. 3 Healthcare institutions recognized an immediate need for evidence-based guidance to inform hospital operational decision-making and ensure the provision of high-quality COVID-19 and non-COVID-19 patient care throughout the unknown course of the impending pandemic.

Here, we share the approach initiated at Mayo Clinic in early March to rapidly assemble and deploy an interdisciplinary, agile task force of academic, clinical, and hospital administrative experts to provide actionable prediction of the COVID-19 impacts on hospital census and operations.

Following Mayo Clinic's rich history of being a physician-lead matrix organization, the Executive Dean of Practice funded the identification of a COVID-19 Data Governor group consisting of the Chief Value Officer, Division Chair, Health Care Policy Research, and Chair, Quality, Experience, and Affordability. The first task of the three Data Governors was to develop an interdisciplinary predictive analytics task force. The interdisciplinary COVID-19 predictive analytics task force was conceived as a partnership between the physician, scientific, and administrative leadership of Mayo Clinic entities including Quality, Experience, and Affordability; Table 1) . During the first two months (March and April 2020) the task force meeting schedule consisted of daily 1-hour Zoom video conferences (Mon-Fri.) with ad-hoc weekend calls as necessary. An assigned project manager coordinated task force meeting agendas and key messages.

The first priority undertaken by the task force was to comprehensively identify and assess all external COVID-19 prediction models to gain an understanding of their methodologies and outputs. We evaluated approximately 15 publicly available models including those from the Institute for Health Metrics and Evaluation (IHME), 5 Qventus, 6 University of Basel, 7 Covid ActNow, 8 Youyang Gu, 9 and Cornell. 10 There were several similarities in the methodologies of these external models, with most relying on a generalized Susceptible, Exposed, Infected, Recovered (SEIR) framework. [11] [12] [13] However, there was significant variation across models regarding their inputs, assumptions, outputs, and applicability to the Mayo Clinic practice. For example, many models predicted COVID-19 cases at the state or county level, but were unable to account for local hospital-level inputs such as length of stay, total available medical and ICU beds, or intra-institutional patient transfer between regional hospitals.

As the task force continued to test external models, a solid conceptual understanding of the key variables and inputs needed for predictive COVID-19 modeling specific to our hospital system was formed within the group. This was accomplished using rapid cycles of change social distancing. Ultimately, the handful of external models which provided flexibility for endusers to input community-and hospital-specific parameters utilized small studies reliant on early Chinese data 14, 15 to set default parameter values. The generalizability of these values to other hospitals was often unknown and it was not possible to refresh these values with 'real-time' hospital-and community-specific data. Furthermore, for these tools to be operational for the practice, we needed to have high data availability, automation in the data curation and tabulation processes, and succinct visuals to convey the message at a time when both hospital staff and leadership were under extreme pressures. As such, the task force recognized a need for the development of an internal data framework to compile state-, county-, and hospital-level testing and case data so that we could update critical model assumptions on a daily basis with data relevant to each of our unique hospital regions.

Having recognized a need for real-time, region-specific COVID-19 data for each of our hospitals, the data scientists on our taskforce began compiling and automating an internal data framework which would be updated daily with state-and county-level case data using application programming interfaces (APIs) from aggregated national COVID-19 data sources such as those provided by USAFACTS 16 and the NY Times. 17 From this data, digital scientists on the task force created red-yellow-green status visualizations of test positivity rates and case doubling times at the state, county, and health referral region (HRR) for each of our hospitals. This dashboard provided the task force with one central location containing high-level, nearreal-time summary visualizations from which to quickly surveil key COVID-19 parameters in each of our hospital regions.

In parallel with the CDT and test positivity dashboards described above, several task create a daily dashboard displaying internal hospital-specific metrics such as total tests ordered, total positive tests and test positivity rate, current hospital and ICU COVID+ census, and cumulative COVID-19 admissions and inpatient fatalities. This COVID-19 leadership dashboard utilized data captured by the electronic health record (EHR) and was updated nightly; it is conceptually similar to COVID-19 dashboards recently described by other institutions. 18, 19 Curation of a centralized, EHR-based 'real-time' COVID-19 data mart

As we began to internally disseminate the dashboards described above, many clinicians, administrative leaders, and researchers across Mayo Clinic began to express a need to access COVID-19 data for various clinical, operational, and research initiatives, resulting in a significant amount of task force time spent responding to queries and sharing knowledge about the current COVID-19 data framework that the task force had developed. To facilitate, centralize, and operationalize COVID-19 analytics, a distinct COVID-19 data mart was built by Mayo Clinic's Department of Data and Analytics (DDA) to house formatted tables containing Mayo Clinic's COVID-19 lab testing, hospital and ICU census data, and critical supply information (e.g., mechanical ventilator status). The data mart build was agile and customizable based on task force feedback, with separate tables built for polymerase chain reaction (PCR) testing, serology testing, and inpatient COVID-19 data. Key variables included test date/times, test results (including all positive, negative, indeterminate, and repeat tests), demographics (age, race, ethnicity, county/state of home address), bed type (ICU versus medical/surgical), status of mechanical ventilator use, and admit/discharge dates. The resulting COVID-19 data mart went 'live' during the first month and continues to undergo refinement with additional tables and more 'real-time' data from the EHR. This data mart exists within our Unified Data Platform (UDP) and contains easily queryable (using Structured Querying Language) COVID-19 data. In addition to

providing key information to inform predictive modeling efforts, the data mart has shown J o u r n a l P r e -p r o o f extended utility and is being utilized by many of the ongoing surveillance and research efforts at our institution.

The COVID-19 models we evaluated. For each of our hospitals, if the model predicted >30% chance of COVID-19 ICU census being greater than 50% of ICU bed capacity in the next 4 weeks, that hospital was given a 'red' alert. If the model predicted <10% of COVID-19 ICU census being greater than 50% of ICU bed capacity in the next 4 weeks, that model was given a 'green' status, while intermediate risks (10-30%) were given a 'yellow' warning. A pre-formatted 20-slide Powerpoint including alert levels, site-specific predictions, historical model accuracy summaries, and 3-5 brief key messages was emailed twice weekly to practice leadership at each of our hospitals.

While the primary mission of the task force was to provide predictions of hospital and 

Shortly after the pandemic began, personal protective equipment (PPE) became a central focus as a key consumable that was essential for the safety of patients and health care workers. The task force commissioned a sub-team to model the PPE resources available to the institution. A two-pronged approach was taken for modeling PPE. First, the historical records maintained by institutional supply chain management were added to the COVID-19data marts so that the data would be consumable by the applications. These data elements allowed for J o u r n a l P r e -p r o o f historical consumption rates to be tabulated for various face masks, gloves, eye protection, gowns, and other critical PPE stock. Next, detailed observational studies were initiated to empirically measure consumption rates for each PPE category per day, per patient, and per staff member. Finally, the estimated consumption rates were linked with model estimates to provide a prediction of how long current supplies would last based on projected volumes and inventory. As the pandemic continued, changes to PPE guidelines were factored into our modeling to update projected consumption.

Effectively messaging the predictions of supply on hand to the practice required pilot testing a variety of visuals. There was initial interest in having highly detailed counts, by site, of inventory, consumption and time until critical shortages. However, since all Mayo Clinic sites work from a single inventory, a single dashboard was used to ensure an equitable distribution of PPE among all sites in the enterprise. Like the hospital and ICU COVID-19 census predictions, a simple red-yellow-green dashboard was created to convey status regarding the amount of supplies available (Figure 2) . As the pandemic progressed, detailed accounting of the supplies on hand, predictions made, and errors in predictions were tabulated. The internal development team utilized this information to refine the prediction models and add the necessary automation to the system to ensure accurate predictions were available to leadership.

With flattened pandemic curves in our DMC regions becoming evident from our modeling in April, (consistently <10% of hospital beds being used for COVID-19 patients), our hospital and procedural practices cautiously prepared to increase provision of non-COVID-19 care. However, whether patients presenting for non-COVID-19 care would be at risk of contracting COVID-19 in our facilities was of concern. Therefore, leadership widened the task force's analytic scope to include monitoring patient safety related to COVID-19 by characterizing J o u r n a l P r e -p r o o f risk and rate of healthcare associated infection in patients hospitalized for non-COVID indications at the three Mayo Clinic DMCs.

As our task force evolved from its formation in March 2020 through several peaks and plateaus at various times across our different hospital regions as of December 2020, the consistency of our modeling efforts led to adoption of our summary reports by hospital leadership. The task force's predictive COVID-19 census output was specifically cited by hospital leadership as a key piece of evidence which directly influenced their pre-emptive decision processes for changing visitation policies, planning for elective surgery reductions, projecting ICU, ED, and respiratory therapy staffing needs, coordinating staffing for expansion of testing and hospital bed capacities, and projecting the number of patients that could be enrolled in clinical trials.

The task force's consistency also allowed a 'steady-state' transition in its workload in several ways. First, the Data Governors were able to pass forward the operational management of the task force to administrative and clinical leadership of the Kern Center. Likewise, with key metric dashboards formatted and automated to update daily, task force members were able to converse more frequently over email during plateau/low spread months and reduce meeting frequency to 'as necessary' when red-yellow-green alert status changed meaningfully in any of our hospital regions. As localized COVID-19 spikes occurred, such as those at our Arizona and Florida hospitals in July and our Midwest hospitals in November, the task force was able to rely on our surveillance dashboards and predictive modeling framework to identify increasing alert status and ramp-up meeting frequency and modeling updates until hospital COVID-19 census returned to a lower, non-alert status. The task force has documented its experience in a general framework ( Table 2 ) and now maintains a baseline 'steady-state' of sentinel COVID-19 activity J o u r n a l P r e -p r o o f with minimum effort needed on an ongoing basis for model or metric development, but with the ability to rapidly and effectively re-engage with the practice using our standardized modeling and communication toolset as new COVID-19 peaks occur.

During the weeks prior to the COVID-19 pandemic in the US, our institution rapidly developed an interdisciplinary predictive analytics task force to provide COVID-19 hospital census forecasting to help clinical leaders understand the potential short-and long-term impacts of the pandemic on hospital operations. Throughout the continually evolving pandemic, our task force has provided frequent and consistent summary messaging and insights that have helped the practice to anticipate and react to changing operational needs and opportunities, thus protecting our ability to care for all patients. Practice leadership has relied on the task force's predictive output as an important data-driven tool in their arsenal for hospital census and quality management. We believe the framework shared here for the deployment of a COVID-19 predictive analytics taskforce could be adapted for effective implementation at other institutions to provide evidence-based messaging for operational decision-making. For hospitals without such a structure, immediate consideration may be warranted with COVID-19 models warning of a sustained 'third-wave' through the winter of 2020. ) Create a concise mission and actionable deliverable(s) for a task force to accomplish. Example: Predict ICU COVID-19 census in 1-, 2-, and/or 4 weeks 2.) Recruit task force members from a diverse set of subject-matter experts, pulling from operational/administrative, academic, and analytic resources, with representation of clinical leaders who can rapidly disseminate key findings to the practice. 3.) Undertake rigorous and rapid assessment of available tools to determine applicability of potential external solutions. 4.) Identify and aggregate key data sources -both internal and external. 5.) Utilize the EHR and lab data as close to 'real-time' as possible. 

What Healthcare Leaders Need to Know: Preparing for COVID-19. Coronavirus Update Webinar. American Hospital Association

The COVID tracking project. The Atlantic Monthly Group

At the Epicenter of the Covid-19 Pandemic and Humanitarian Crises in Italy: Changing Perspectives on Preparation and Mitigation. NEJM Catalyst Innovations in Care Delivery

Forecasting COVID-19 impact on hospital bed-days, ICU-days, ventilator-days and deaths by US state in the next 4 months

Localized COVID-19 Model and Scenario Planner

COVID-19 Scenarios: an interactive tool to explore the spread and associated morbidity and mortality of SARS-CoV-2

Jonathan Kreiss-Tomkins, Anna Blech, and many others

COVID-19 Projections Using Machine Learning

Cornell COVID Caseload Calculator C5V

Analysis of a fractional SEIR model with treatment

Estimation of the Transmission Risk of the 2019-nCoV and Its Implication for Public Health Interventions

A fractional order SEIR model with vertical transmission

Early Transmission Dynamics in Wuhan, China, of Novel Coronavirus-Infected Pneumonia

Incubation Period and Other Epidemiological Characteristics of 2019 Novel Coronavirus Infections with Right Truncation: A Statistical Analysis of Publicly Available Case Data

US Coronavirus Cases and Deaths. USAFACTS. 2020

Covid-19) Data in the United States. NY Times

Rapid response to COVID-19: health informatics support for outbreak management in an academic health system

Responding to COVID-19: The UW Medicine Information Technology Services Experience

Bayesian nonparametrics for stochastic epidemic models

Unacast 2020. Unacast Social Distancing Dataset

Total teamwork--the Mayo Clinic