key: cord-336732-9zp5agja authors: Williams, R. D.; Markus, A. F.; Yang, C.; Duarte Salles, T.; Falconer, T.; Jonnagaddala, J.; Kim, C.; Rho, Y.; Williams, A.; An, M. H.; Aragon, M.; Areia, C.; Burn, E.; Choi, Y.; Drakos, I.; Fernandes Abrahao, M.; Fernandez-Bertolin, S.; Hripcsak, G.; Kaas-Hansen, B.; Kandukuri, P.; Kostka, K.; Liaw, S.-T.; Machnicki, G.; Morales, D.; Nyberg, F.; Park, R. W.; Prats-Uribe, A.; Pratt, N.; Rivera, M.; Seinen, T.; Shoaibi, A.; Spotnitz, M. E.; Steyerberg, E. W.; Suchard, M. A.; You, S. C.; Zhang, L.; Zhou, L.; Ryan, P. B.; PRIETO-ALHAMBRA, D.; Reps, J. M.; Rijnbeek, P. R. title: Seek COVER: Development and validation of a personalized risk calculator for COVID-19 outcomes in an international network date: 2020-05-27 journal: nan DOI: 10.1101/2020.05.26.20112649 sha: doc_id: 336732 cord_uid: 9zp5agja Abstract Importance COVID-19 is causing high mortality worldwide. Developing models to quantify the risk of poor outcomes in infected patients could help develop strategies to shield the most vulnerable during de-confinement. Objective To develop and externally validate COVID-19 Estimated Risk (COVER) scores that quantify a patient's risk of hospital admission (COVER-H), requiring intensive services (COVER-I), or fatality (COVER-F) in the 30-days following COVID-19 diagnosis. Design Multinational, distributed network cohorts. Setting We analyzed a federated network of electronic medical records and administrative claims data from 13 data sources and 6 countries, mapped to a common data model. Participants Model development used a patient population consisting of >2 million patients with a general practice (GP), emergency room (ER), or outpatient (OP) visit with diagnosed influenza or flu-like symptoms any time prior to 2020. The model was validated on patients with a GP, ER, or OP visit in 2020 with a confirmed or suspected COVID-19 diagnosis across four databases from South Korea, Spain and the United States. Outcomes Age, sex, historical conditions, and drug use prior to index date were considered as candidate predictors. Outcomes included i) hospitalization with pneumonia, ii) hospitalization with pneumonia requiring intensive services or death, and iii) death in the 30 days after index date. Results Overall, 43,061 COVID-19 patients were included for model validation, after initial model development and validation using 6,869,127 patients with influenza or flu-like symptoms. We identified 7 predictors (history of cancer, chronic obstructive pulmonary disease, diabetes, heart disease, hypertension, hyperlipidemia, and kidney disease) which combined with age and sex could discriminate which patients would experience any of our three outcomes. The models achieved high performance in influenza. When transported to COVID-19 cohorts, the AUC ranges were, COVER-H: 0.73-0.81, COVER-I: 0.73-0.91, and COVER-F: 0.82-0.90. Calibration was overall acceptable, with overestimated risk in the most elderly and highest risk strata. Conclusions and relevance A 9-predictor model performs well for COVID-19 patients for predicting hospitalization, intensive services and death. The models could aid in providing reassurance for low risk patients and shield high risk patients from COVID-19 during de-confinement to reduce the virus' impact on morbidity and mortality. Importance COVID-19 is causing high mortality worldwide. Developing models to quantify the risk of poor outcomes in infected patients could help develop strategies to shield the most vulnerable during de-confinement. To develop and externally validate COVID-19 Estimated Risk (COVER) scores that quantify a patient's risk of hospital admission (COVER-H), requiring intensive services (COVER-I), or fatality (COVER-F) in the 30-days following COVID-19 diagnosis. Multinational, distributed network cohorts. We analyzed a federated network of electronic medical records and administrative claims data from 13 data sources and 6 countries, mapped to a common data model. Model development used a patient population consisting of >2 million patients with a general practice (GP), emergency room (ER), or outpatient (OP) visit with diagnosed influenza or flu-like symptoms any time prior to 2020. The model was validated on patients with a GP, ER, or OP visit in 2020 with a confirmed or suspected COVID-19 diagnosis across four databases from South Korea, Spain and the United States. Outcomes Age, sex, historical conditions, and drug use prior to index date were considered as candidate predictors. Outcomes included i) hospitalization with pneumonia, ii) hospitalization with pneumonia requiring intensive services or death, and iii) death in the 30 days after index date. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 27, 2020. . https://doi.org/10.1101/2020.05.26.20112649 doi: medRxiv preprint Overall, 43,061 COVID-19 patients were included for model validation, after initial model development and validation using 6,869,127 patients with influenza or flu-like symptoms. We identified 7 predictors (history of cancer, chronic obstructive pulmonary disease, diabetes, heart disease, hypertension, hyperlipidemia, and kidney disease) which combined with age and sex could discriminate which patients would experience any of our three outcomes. The models achieved high performance in influenza. When transported to COVID-19 cohorts, the AUC ranges were, COVER-H: 0.73-0.81, COVER-I: 0.73-0.91, and COVER-F: 0.82-0.90. Calibration was overall acceptable, with overestimated risk in the most elderly and highest risk strata. Conclusions and relevance A 9-predictor model performs well for COVID-19 patients for predicting hospitalization, intensive services and death. The models could aid in providing reassurance for low risk patients and shield high risk patients from COVID-19 during de-confinement to reduce the virus' impact on morbidity and mortality. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The growing number of infections due to the Corona Virus Disease 2019 (COVID-19) has resulted in unprecedented pressure on healthcare systems worldwide, and a large number of casualties at a global scale. Although the majority of people have uncomplicated or mild illness (81%), some will develop severe disease leading to hospitalization and oxygen support (15%) or fatality (4%) 1, 2 . The most common diagnosis in severe COVID-19 patients is pneumonia, other known complications include acute respiratory distress syndrome (ARDS), sepsis, or acute kidney injury (AKI) 1 . While there is currently no known cure or vaccine, the current approach to management of COVID-19 includes monitoring and controlling symptoms. In response to the global pandemic, many countries have implemented measures aimed to reduce the average number of people a person with COVID-19 will infect 3-6 . The purpose of this was to ideally prevent the spread of the virus, or at least to slow the spread, a process known as flattening the curve 7 . However, strategies such as social distancing have impacted economies globally and economic worries are causing countries to consider lifting measures earlier than epidemiologists recommend 8 . There are worries that this may cause a second wave of infections, as seen historically in other pandemics 9 . Multiple governments are starting to release de-confinement strategies, for example the state of New York 10 , British 11 , and Dutch 12 governments have detailed plans to ease restrictions. However, they only concern populationlevel effects of likely disease spread and contain no information on how an individual's risk impacts their likely morbidity and mortality if they were to contract the virus. Research has shown that COVID-19 does not impact all ages and sexes equally 13 and as such a more personalised risk assessment can aid in improving outcomes. In a recent BMJ editorial 14 , the authors conclude that the COVID-19 response "is about protecting lives and communities most obviously at risk in our unequal society". Quantifying a patient's risk of having severe or critical illness when infected with COVID-19, could be used to help countries plan strategies to shield the most vulnerable patient populations. This is essential during the planning of deconfinement strategies. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted May 27, 2020. . https://doi.org/10.1101/2020.05.26.20112649 doi: medRxiv preprint The WHO Risk Communication Guidance distinguishes two categories of patients at high risk of severe disease: those older than 60 years and those with "underlying medical conditions" which is non-specific 15 . Using general criteria to assess the risk of poor outcomes is a crude risk discrimination mechanism as entire patient groupings are treated homogeneously ignoring individual differences. Prediction models can quantify a patient's individual risk and data-driven methods could identify risk factors that have been previously overlooked. The number of studies developing prediction models for COVID-19 is still limited and of insufficient quality, as suggested in a recent systematic review 16 . Previously published COVID-19 prediction models have been criticised for being i) poorly reported, ii) developed using small data samples, and iii) lacking external validation. In this paper we aim to develop COVID-19 Estimated Risk (COVER) scores to quantify a patient's risk of hospital admission (COVER-H), requiring intensive services (COVER-I), or fatality (COVER-F) due to COVID-19 using the Observational Health Data Sciences and Informatics (OHDSI) Patient-Level Prediction framework 17 . The research collaboration known as OHDSI has developed standards and tools that allow patient-level prediction models to be developed and externally validated rapidly following accepted best practices 18 . This allows us to overcome the previously identified shortcomings of previous COVID-19 prediction papers by reporting according to open science standards and implementing widespread external validation. To overcome the shortcoming of using small data for development, we made use of the abundant data from patients with influenza or flu-like symptoms to develop the models and then we tested whether the models transport to COVID-19 patients. Given the symptomatic similarities between the two diseases we hypothesized that the developed models will be able to transport between the two problem settings. We performed a retrospective cohort study to develop COVID-19 prediction models for severe and critical illness. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted May 27, 2020. . https://doi.org/10.1101/2020.05.26.20112649 doi: medRxiv preprint This study used observational healthcare databases from six different countries. All datasets used in this paper were mapped into the Observational Medical Outcomes Partnership Common Data Model (OMOP-CDM) 19 . The OMOP-CDM was developed for researchers to have diverse datasets in a consistent structure and vocabulary. This enables analysis code and software to be shared among researchers which facilitates external validation of the prediction models. All databases obtained IRB approval or used deidentified data that was considered exempt from IRB approval. Informed consent was not necessary at any site. The OMOP-CDM datasets used in this paper are listed in Table 1 . For validation in COVID-19 we used a cohort of patients presenting at an initial healthcare provider interaction in a general practice (GP), emergency room (ER), or outpatient (OP) visit with COVID-19 disease. COVID-19 disease was identified by a diagnosis code for COVID-19 or a positive test for the SARS-COV-2 virus that was recorded after January 1st 2020. We required patients to be aged 18 or over, have at least 365 days of observation time prior to the index date and no diagnosis of influenza, flu-like symptoms, or pneumonia in the preceding 60 days. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted May 27, 2020. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 27, 2020. . https://doi.org/10.1101/2020.05.26.20112649 doi: medRxiv preprint For model development, we identified patients over 18 with a GP, ER, or OP visit with influenza or flu-like symptoms (e.g. fever and either cough, shortness of breath, myalgia, malaise, or fatigue), at least 365 days of prior observation, and no symptoms in the preceding 60 days. We investigated three outcomes of COVID-19: 1) hospitalization with pneumonia from index up to 30 days after index, 2) hospitalization with pneumonia that required intensive services or death after hospitalization with pneumonia from index up to 30 days after index, and 3) death from index up to 30 days after index. The full details of the participant cohorts and outcomes used for development and validation can be found in the study package. When using a data-driven approach to model development, generally the resulting models contain a large number of predictors. We developed a data-driven model using age in groups (18-19, 20-25, 26-30, …, 95+), sex and binary variables indicating the presence or absence of recorded conditions and drugs any time prior to index. In total, we derived 31,917 candidate predictors indicating the presence of the 31,917 unique conditions/drugs recorded prior to the index date (GP, ER, or OP visit) for each patient. This may optimise performance, but a large number of predictors can be a barrier to clinical implementation. The utility of models for COVID-19 requires that they can be widely implemented across worldwide healthcare settings. Therefore, in addition to a data-driven model, we investigated two models that include fewer candidate predictors. The age/sex model used age groups and sex as candidate predictors. The COVER models included 7 candidate predictors, in addition to age groups and sex, that corresponded to the following conditions existing any time prior to the index date (GP, ER, or OP visit): cancer, chronic obstructive pulmonary disease, diabetes, heart disease, hypertension, hyperlipidemia and kidney disease (chronic and acute). Full details on how these 7 predictors were created can be found in Appendix A. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 27, 2020. . https://doi.org/10.1101/2020.05.26.20112649 doi: medRxiv preprint The models were developed using the Optum © De-Identified Clinformatics® Data Mart Database. We identified 7,344,117 valid visits with influenza or flu-like symptoms, of which 4,431,867 were for patients aged 18 or older, 2,977,969 of these had >= 365 days observation prior to the visit, and 2,082,277 of these had no prior influenza/symptoms/pneumonia in the 60 days prior to index. We selected a stratified sample of 150,000 patients from the total population to efficiently develop models to address the current pandemic, while preserving the outcome rate. Model development followed a previously validated and published framework for the creation and validation of patient-level prediction 17 . We used a person 'train-test split' method to perform internal validation. In each development cohort, a random split sample (`training sample') containing 75% of patients was used to develop the prediction models and the remaining 25% of patients (`test sample') was used to validate the models. We trained models using LASSO regularised logistic regression, using a 3-fold cross validation technique in the influenza training sample to learn the optimal regularization hyperparameter through an adaptive search 20 . We used R (version 3.6.3) and the OHDSI Patient-Level Prediction package (version 3.0.16) for all statistical analyses 17 . To evaluate the performance, we calculate the overall discrimination of the model using the area under the receiver operating characteristic curve (AUC), the area under the precision recall-curve (AUPRC), and the model calibration. The AUC indicates the probability that for two randomly selected patients, the patient who gets the outcome will be assigned a higher risk. The AUPRC shows the trade-off between identifying all patients who get the outcome (recall) versus incorrectly identifying patients without outcome (precision) across different risk thresholds. The model calibration is presented in a plot to examine agreement between predicted and observed risks across deciles of predicted risk. Calibration assessment is then performed visually rather than using a statistic or numeric value as this provides a better impression of the direction and scale of miscalibration 21 . Summary statistics are reported from the test samples. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 27, 2020. . We performed two types of external validation. A classical external validation in which we applied the models to identical settings across diverse patient populations with influenza or flulike symptoms prior to 2020 not used to develop the model, and a specific COVID-19 validation for databases containing COVID-19 data. To do this we assessed patients with confirmed COVID-19 in 2020. We examined the external validation using AUC, AUPRC and model calibration in the same way as internally. The complete results are available as an interactive app at: http://evidence.ohdsi.org/Covid19CoverPrediction. Table 2 Table 2 for the full outcome rates across the databases included in this study. A total of 43,061 participants with COVID-19 disease were further included for external validation. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 27, 2020. . https://doi.org/10.1101/2020.05.26.20112649 doi: medRxiv preprint In the databases used for external validation, the patient numbers ranged from 395 (TRDW) to 3,146,743 (CCAE). The datasets had varied outcome rates ranging from 0.06-12.47 for hospital admission, 0.01-4.91 for intensive services, and 0.01-12.27 for death. Characteristics at baseline differed substantially between databases as can be seen in Table 2 , with MDCR (a database representing retirees) containing a relatively old population of patients and a high number of comorbidities, and IPCI (a database representing general practice) showing a relatively low condition occurrence. The data-driven models for hospitalization, intensive services, and death contained 521, 349, and 205 predictors respectively. The COVER-H, COVER-I, and COVER-F scores are presented in Figure 1 . These models are also accessible online: http://evidence.ohdsi.org/Covid19CoverPrediction. Figure 1 also provides a risk converter, which allows for easy conversion between the risk score and predicted risk of the outcomes. Furthermore, we provide a plot of the probability distribution for the three models from patients in ClinFormatics to demonstrate the expected regions the probabilities fall into. To calculate the COVER scores using Figure 1 , a clinician needs to identify which predictors the patient has. The points for each of those predictors are then added to arrive at the total score. For example, if a 63-year-old female patient has diabetes and heart disease, then her risk score for hospital admission (COVER-H) is 43 (female sex) + 4 (heart disease) + 3 (diabetes) + 15 (age) = 65. The risk scores for intensive services (COVER-I) and fatality (COVER-F) are 51 and 47, respectively. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 27, 2020. Model performance The internal validation performance for each model is presented in Table 4 . The external validation of the COVER scores on the COVID-19 patients is shown in Table 5 . Full validation results can be seen in Appendix B. Receiver operating characteristic and calibration plots are included in Appendix C. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 27, 2020. . https://doi.org/10.1101/2020.05.26.20112649 doi: medRxiv preprint We developed and externally validated models using large datasets of influenza patients to quantify a patient's risk of developing severe or critical illness due to COVID-19. In the development data, the 9-predictor COVID-19 Estimated Risk (COVER) scores were a good tradeoff between model complexity and performance, as the AUCs were generally close to the large data-driven models. The COVER scores achieved an AUC of 0.84 when predicting which patients will be hospitalized or require intensive services and an AUC of 0.9 when predicting which poor in CUIMC, often underestimating risk, but this may be due to CUIMC containing mostly hospitalized COVID-19 patients, so the CUIMC cohort are sicker. Given the calibration was good for the vast majority of patients, recalibration was not deemed necessary for transporting the model to COVID-19 patients. We also performed sensitivity analyses using more sensitive COVID-19 definitions which also included patients with symptoms, or symptoms and influenza. The results did not show much deviation from the specific definition (see Appendix B). These results showed that training in large historical influenza data was an effective strategy to develop models for COVID-19 patients. We also validated the age/sex and data-driven models on the COVID-19 patients and the age/sex models already appear to do well. This shows that age and sex are strong predictors of disease severity in COVID-19. Our results show that quantifying a symptomatic patient's risk based on a small selection of comorbidities as well as age/sex gives improved model performance. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 27, 2020. We identified one other model that addressed a similar problem setting. The COVID-19 Vulnerability Index built from a 5% sample of Medicare claims data from 2015-2016 using a proxy for COVID-19. The model predicts hospitalization due to pneumonia (except when caused by tuberculosis), influenza, acute bronchitis, or other specified upper respiratory infections 24 . The model achieved an AUC of 0.73, but has not been validated on a COVID-19 cohort. Several other models have been proposed to predict severity of COVID-19 25-27 , but these only consider patients already hospitalized. Limitations of the study included being unable to develop a model on COVID-19 patient data due to the scarcity of databases that contain this information in sufficient numbers, however we were able to validate the models developed and as such are confident the performance is transportable. In CUIMC, HIRA and SIDIAP COVID-19 data we reached the threshold for reliable external validation 28,29 . The results of TRDW are promising, but might not be reliable due to the low number of outcomes. As larger COVID-19 databases become available, training a model using this data may highlight predictors of severity amongst uncommon influenza presentations, for example younger and healthier patients experiencing severe or critical illness. Further limitations include misclassification of predictors, for example if disease is incorrectly recorded in a patient's history, as well as in the cohorts through incorrect recording of influenza or COVID-19. We were unable to validate the COVER-H model in CUIMC as it mostly contained ER or hospitalized COVID-19 patients and the COVER-I model in SIDIAP due to a lack of information on intensive services in the database. The results show we were able to develop models that use a patient's socio-demographics and medical history to predict their risk of becoming severely or critically ill when infected with COVID-19. To our knowledge, this is the first study that has been able to extensively externally validate prediction models on COVID-19 patients internationally. The strong performance in COVID-19 patients of the COVER scores can be used to identify patients who should be shielded . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 27, 2020. . https://doi.org/10.1101/2020.05.26.20112649 doi: medRxiv preprint from COVID-19. This can have multiple benefits; i) it can help reassure low risk people who may be psychologically impacted by the stress of the virus, and ii) it can help identify which people would be at increased risk of severe or critical outcomes and as such should continue to be shielded during the first stages of de-confinement. In this paper we developed and validated models that can predict which patients presenting with COVID-19 are at high risk of experiencing severe or critical illness. These models can be used to identify vulnerable patient populations that require shielding as they have the worst COVID-19 prognosis. This evidence can be particularly impactful as governments start to lift measures and should be used to aid strategic planning to help us protect the most vulnerable. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 27, 2020. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 27, 2020. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 27, 2020. . https://doi.org/10.1101/2020.05.26.20112649 doi: medRxiv preprint Appendix A: COVER score derivation The go from the data-driven model to the model with reduced variables we implemented this process for each of the scores. 1 . A clinician inspected the data-driven model to identify variables that had a high standardized mean difference between those who with and without outcome. For example, often there are multiple predictors which are related and correlated selected by the model, for example a model might select as associated a condition occurrence in different time periods predating the index date. This could be simplified to a predictor saying only "Patient had condition X in history", rather than multiple predictors specifying within which the condition occurred, or multiple codes that are probably related to a specific condition. We identified general categories from these such as 'heart disease' and 'diabetes'. 5. This gave us the simple score-based model. The phenotypes for each COVER predictor are available in Supplement A. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 27, 2020. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted May 27, 2020. . https://doi.org/10.1101/2020.05.26.20112649 doi: medRxiv preprint . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted May 27, 2020. . https://doi.org/10.1101/2020.05.26.20112649 doi: medRxiv preprint TRDW COVER-I CUIMC HIRA . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted May 27, 2020. . https://doi.org/10.1101/2020.05.26.20112649 doi: medRxiv preprint COVER-F CUIMC HIRA SIDIAP . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted May 27, 2020. . https://doi.org/10.1101/2020.05.26.20112649 doi: medRxiv preprint Clinical management of severe acute respiratory infection (SARI) when COVID-19 disease is suspected: interim guidance Hospitalization and 30-day fatality in 121,263 COVID-19 outpatient cases. medRxiv How will country-based mitigation measures influence the course of the COVID-19 epidemic? The Lancet Coronavirus action plan: a guide to what you can expect across the UKCoronavirus: action plan. Department of Health Interrupting transmission of COVID-19: lessons from containment efforts in Singapore Response to COVID-19 in Taiwan: Big Data Analytics, New Technology, and Proactive Testing Nonpharmaceutical Interventions Implemented by US Cities During the 1918-1919 Influenza Pandemic FORWARD: A guide to reopening New York & building back better Our plan to rebuild: The UK Government's COVID-19 recovery strategy. UK Government Factsheet versoepelen maatregelen corona An international characterisation of patients hospitalised with COVID-19 and a comparison with those previously hospitalised with influenza. medRxiv The UK's public health response to covid-19 World Health Organization. Coronavirus disease 2019 (COVID-19) Situation report -51 2020 Prediction models for diagnosis and prognosis of covid-19 infection: systematic review and critical appraisal Design and implementation The authors would like to thank the OHDSI community for their contributions to the tools used for this analysis.The authors appreciate healthcare professionals dedicated to treating COVID-19 patients in Korea, and the Ministry of Health and Welfare and the Health Insurance Review & Assessment All authors have filled and provided an ICMJE form with any potential competing interests.