key: cord-0993235-4hz8p8y9 authors: Gerlovin, Hanna; Posner, Daniel C; Ho, Yuk-Lam; Rentsch, Christopher T; Tate, Janet P; King Jr, Joseph T; Kurgansky, Katherine E; Danciu, Ioana; Costa, Lauren; Linares, Franciel A; Goethert, Ian D; Jacobson, Daniel A; Freiberg, Matthew S; Begoli, Edmon; Muralidhar, Sumitra; Ramoni, Rachel B; Tourassi, Georgia; Gaziano, J Michael; Justice, Amy C; Gagnon, David R; Cho, Kelly title: Pharmacoepidemiology, Machine Learning and COVID-19: An intent-to-treat analysis of hydroxychloroquine, with or without azithromycin, and COVID-19 outcomes amongst hospitalized US Veterans date: 2021-06-24 journal: Am J Epidemiol DOI: 10.1093/aje/kwab183 sha: 0b7122e138af40c4bdc203936638a5908d98f15c doc_id: 993235 cord_uid: 4hz8p8y9 Hydroxychloroquine (HCQ) was proposed as an early therapy for coronavirus disease 2019 (COVID-19) after in vitro studies indicated possible benefit. Previous in vivo observational studies have presented conflicting results, though recent randomized clinical trials have reported no benefit from HCQ amongst hospitalized COVID-19 patients. We examined the effects of HCQ alone, and in combination with azithromycin, in a hospitalized COVID-19 positive, United States (US) Veteran population using a propensity score adjusted survival analysis with imputation of missing data. From March 1, 2020 through April 30, 2020, 64,055 US Veterans were tested for COVID-19 based on Veteran Affairs Healthcare Administration electronic health record data. Of the 7,193 positive cases, 2,809 were hospitalized, and 657 individuals were prescribed HCQ within the first 48-hours of hospitalization for the treatment of COVID-19. There was no apparent benefit associated with HCQ receipt, alone or in combination with azithromycin, and an increased risk of intubation when used in combination with azithromycin [Hazard Ratio (95% Confidence Interval): 1.55 (1.07, 2.24)]. In conclusion, we assessed the effectiveness of HCQ with or without azithromycin in treating patients hospitalized with COVID-19 using a national sample of the US Veteran population. Using rigorous study design and analytic methods to reduce confounding and bias, we found no evidence of a survival benefit from the administration of HCQ. In the swell of the coronavirus disease 2019 (COVID-19) pandemic, the world rushed to find therapeutic and prophylactic treatments, and hydroxychloroquine (HCQ) became an early front-runner (1, 2) . HCQ is a common anti-malarial/-rheumatologic drug with immunosuppressive functions. Early in vitro studies suggested HCQ might be repurposed to treat infections with a strong immune component (1, 3, 4) , such as COVID- 19 . This was appealing considering its low cost and widespread availability. The United States (US) Food and Drug Administration issued an emergency use authorization for HCQ on March 28, 2020(5) prior to the completion of a randomized controlled trial, only to revoke it less than 3 months later, following concerns about HCQ associated adverse events reported by observational studies (6, 7) . Around the same time as the US Food and Drug Administration's retraction, several randomized controlled trials, ORCHID, RECOVERY and SOLIDARITY discontinued their HCQ arms due to interim analyses showing no benefit in reducing COVID-19 inpatient mortality (8) (9) (10) . These trials recently made their results public (11) (12) (13) . While randomized controlled trials are a gold standard for evaluating the effectiveness of a drug (14) , none of those investigating HCQ treatment explored the combination with azithromycin in their study design. Azithromycin has also been given Results from observational studies of HCQ in treating COVID-19 have been inconsistent, and subject to bias (15) (16) (17) (18) (19) . Early studies claiming a benefit were from small samples with limited data and little control of potential confounders. Timing of treatment during hospitalization was often poorly defined, no studies appeared to control for secular trends in the timing of treatment, and several studies used data from HCQ use prior to the US Food and Drug Administration's initial emergency use authorization (20) . Particularly, the study design and analytic techniques may not have been able to account for the various sources of potential and residual confounding (21) (22) (23) . In a recent meta-analysis of HCQ and mortality in patients hospitalized with COVID-19(16), 25 of the 29 studies used observational data, and 10 of these peerreviewed and pre-print publications used some form of propensity adjustment. One main goal of propensity analysis is to balance confounding factors in order to emulate a randomized controlled trial setting (24) . Recent studies on propensity scoring have found that machine learning methods can achieve better balance than traditional regression methods in observational studies (25) (26) (27) (28) (29) . Gradient boosted modeling using decision trees allows for interactions among the variables used in propensity score calculation and makes no assumptions about the shape of the relationship between the confounder and treatment received (25) . In this paper we apply careful study design and statistical analytic approaches, considered as potential confounders of treatment and both primary outcomes were chronic medications, concurrent inpatient treatments (for COVID-19 or HCQ contraindications), chronic conditions (based on diagnostic codes and including a frailty score(33)), and acute laboratory results and vital signs (those related to acute illness). All potential confounders were included in the propensity model. Complete descriptions of diagnostic and medication codes can be found in Web Tables 1-2 and Web Appendix 1. All analyses were performed using R software (R Foundation for Statistical Computing, Vienna, Austria)(34) and publicly available packages. Missing data -Missing covariate information was imputed using the multiple imputation from chained equations "mice" package in R (35, 36) . Ten imputed data sets were generated, analyzed separately, and the final results were subsequently combined using Rubin's rules to determine final effect sizes and confidence intervals (37) . Propensity score calculation -Propensity scores for each treatment were estimated from a Gradient Boosting Machine (GBM) (38) , an ensemble of models that take baseline measures and characteristics as inputs and outputs the patient's predicted probability (or propensity score) for receiving each treatment (Both, HCQ alone, azithromycin alone, or neither). We employed decision trees as base learners for GBM, using the "gbm" and "WeightIt" R packages to fit our models (39, 40) . The hyperparameters were set as: interaction depth of 4, maximum of 5000 trees, and shrinkage of 0.1. We optimized the maximum of standardized mean differences between potential confounders across the treatment arms. For each patient, the propensity score was converted to a stabilized inverse probability of treatment weight. We evaluated the propensity scores using the "cobalt" package in R to look at the distributions of average standardized mean differences between each pair of treatments (41) . The relative influence (42) was calculated as the normalized amount of change in the balance metric for each variable when it was used to split a node. Outcome models -The stabilized inverse probability of treatment weights from the propensity modeling steps were included as subject-level weights in Cox proportional hazards multivariable models for estimating treatment effects on mortality and intubation using the "survival" package(43) in R. An alpha level of 0.05 was used. We assessed design assumptions and data restrictions with a series of sensitivity analyses to address questions regarding timing, analytic design, and methods. To consider whether timing of treatment initiation made a difference in survival, we considered a shorter 24-hour exposure window, with corresponding adjustments in exclusions and outcomes. We explored the effect of the secular prescribing trend(s) by limiting analyses to time windows framed by regulatory guidelines and patterns of use within the VA. The final set of sensitivity analyses focused on the statistical and machine learning methods and assumptions. We additionally considered a set of doubly-robust models, where select confounders were included in both the propensity and outcome models (44, 45) . Complete details about cohort restrictions and sensitivity analyses performed can be found in Web Table 3 and Web Appendix 2. There were few measurable changes in the effect estimates and confidence intervals of the two comparisons (Both vs. neither; HCQ alone vs. neither) for many of the sensitivity analyses. Figure 5 summarizes the average treatment effect HR (95% CI) for those initiating any combination of HCQ compared to neither treatment in the 48 hours following admission. Complete results, including event counts and number exposed, from all sensitivity analyses can be found in Web Tables 6-7 . Censoring at change in treatment (adding either azithromycin or HCQ after 48 hours post-hospitalization) produced substantially different results for mortality (HCQ vs. Figure 5A ). This corresponded to 75 fewer "cases", mostly from the neither group. A similar pattern of inflated hazard ratios and fewer cases can be seen for the intubation outcome ( Figure 5B ). (57), thus is not preferred over the intention-to-treat method used. In fact, we observed this bias in the shifted HRs and confidence intervals that made HCQ (both with and without azithromycin) appear harmful compared to neither treatment. After 48 hours from index date, approximately 25% of the combination treatment patients were in the intensive care unit, compared to 5% in the neither group, 19% in the azithromycin alone group, and 13% for those on HCQ alone. We did not look at this particular outcome or adjust for it as a confounder in the propensity models. However, in a sensitivity analysis removing these individuals, the HRs for both mortality and intubation of the combined treatment group, relative to neither treatment, shifted completely to the null, indicating that HCQ may have been seen as a "rescue" therapy in intensive care unit patients. Of note, even with this restriction, there is no evidence of benefit. Despite our array of sensitivity analyses, we acknowledge that there is still a possibility of some unmeasured and residual confounding that we were unable to account for. However, the GBM approach allowed us to control for many variables, and any remaining unmeasured confounders would likely require strong associations with both the treatment assignment and outcomes, to explain away the null relationship observed in the data. Squares with solid lines and diamonds with dashed lines represent the hazard ratios and 95% confidence intervals corresponding to the average treatment effects of hydroxychloroquine (HCQ) and HCQ with azithromycin (Both) compared to neither treatment, respectively. Sensitivity analyses -Differences from Primary Analysis: In vitro testing of combined hydroxychloroquine and azithromycin on SARS-CoV-2 shows synergistic effect Race to find COVID-19 treatments accelerates A pharmacological perspective of chloroquine in SARS-CoV-2 infection: An old drug for the fight against a new coronavirus Pharmacokinetic Basis of the Hydroxychloroquine Response in COVID-19: Implications for Therapy and Prevention Letter of Authorization -chloroquine phosphate and hydroxychloroquine sulfate Letter revoking EUA for chloroquine phosphate and hydroxychloroquine sulfate FDA Drug Safety Communication: FDA cautions against use of hydroxychloroquine or chloroquine for COVID-19 outside of the hospital setting or a clinical trial due to risk of heart rhythm problems NIH halts clinical trial of hydroxychloroquine Statement from the Chief Investigators of the Randomised Evaluation of COVid-19 thERapY (RECOVERY) Trial on hydroxychloroquine WHO discontinues hydroxychloroquine and lopinavir/ritonavir treatment arms for COVID-19 Repurposed antiviral drugs for COVID-19 -interim WHO SOLIDARITY trial results Effect of Hydroxychloroquine on Clinical Status at 14 Days in Hospitalized Patients With COVID-19: A Randomized Clinical Trial A comparison of observational studies and randomized, controlled trials COVID-19 coronavirus research has overall low methodological quality thus far: case in point for chloroquine/hydroxychloroquine Effect of hydroxychloroquine with or without azithromycin on the mortality of coronavirus disease 2019 (COVID-19) patients: a systematic review and meta-analysis Hydroxychloroquine in patients with COVID-19: A Systematic Review and meta-analysis Hydroxychloroquine or Chloroquine for Treatment or Prophylaxis of COVID-19: A Living Systematic Review A systematic review on use of aminoquinolines for the therapeutic management of COVID-19: Efficacy, safety and clinical trials Outcomes of Hydroxychloroquine Usage in United States Veterans Hospitalized with COVID-19 A Rush to Judgment? Rapid Reporting and Dissemination of Results and Its Consequences Regarding the Use of Hydroxychloroquine for COVID-19 Considerations for pharmacoepidemiological analyses in the SARS-CoV-2 pandemic Need for Transparency and Reliable Evidence in Emergency Use Authorizations for Coronavirus Disease 2019 (COVID-19) Therapies An Introduction to Propensity Score Methods for Reducing the Effects of Confounding in Observational Studies Improving propensity score weighting using machine learning Propensity score estimation with boosted regression for evaluating causal effects in observational studies The Right Tool for the Job: Choosing Between Covariate-balancing and Generalized Boosted Model Propensity Scores Propensity score methods to control for confounding in observational cohort studies: a statistical primer and application to endoscopy research Propensity Score Estimation: Machine Learning and Classification Methods As Alternatives To Logistic Regression Graphical Depiction of Longitudinal Study Designs in Health Care Databases A Natural Language Processing System for National COVID-19 Surveillance in the US Department of Veterans Affairs Accuracy and completeness of mortality data in the Department of Veterans Affairs The Burden of Frailty Among U.S. Veterans and Its Association With Mortality R: A Language and Environment for Statistical Computing, release 3.6.1. Vienna, Austria: R Foundation for Statistical Computing Recursive partitioning for missing data imputation in the presence of interaction effects Multivariate Imputation by Chained Equations Multiple imputation in health-care databases: an overview and some applications Boosting and Additive Trees Generalized Boosted Regression Models Weighting for Covariate Balance in Observational Studies Covariate Balance Tables and Plots Gradient boosting machines, a tutorial survival: A Package for Survival Analysis in S A tutorial on propensity score estimation for multiple treatments using generalized boosted models Comparison of the ability of doublerobust estimators to correct bias in propensity score matching analysis. A Monte Carlo simulation study Moving towards best practice when using inverse probability of treatment weighting (IPTW) using the propensity score to estimate causal treatment effects in observational studies Association of Treatment With Hydroxychloroquine or Azithromycin With In-Hospital Mortality in Patients With COVID-19 in New York State Utilization of COVID-19 Treatments and Clinical Outcomes among Patients with Cancer: A COVID-19 and Cancer Consortium (CCC19) Cohort Study Hydroxychloroquine with or without azithromycin and in-hospital mortality or discharge in patients hospitalized for COVID-19 infection: a cohort study of 4,642 in-patients in France Treatment with hydroxychloroquine, azithromycin, and combination in patients hospitalized with COVID-19 An observational cohort study of hydroxychloroquine and azithromycin for COVID-19: (Can't Get No) Satisfaction Chronological Bias Hydroxychloroquine and Chloroquine Prescribing Patterns by Provider Specialty Following Initial Reports of Potential Benefit for COVID-19 Treatment -United States Patterns of COVID-19 testing and mortality by race and ethnicity among United States veterans: A nationwide cohort study Observational Study of Hydroxychloroquine in Hospitalized Patients with Covid-19 Estimating effects from randomized trials with discontinuations: the need for intent-to-treat design and G-estimation