key: cord-0296243-nam2fosn
authors: Reynolds, M. W.; Secora, A.; Joules, A.; Albert, L.; Brinkley, E.; Kwon, T.; Mack, C. D.; Toovey, S.; Dreyer, N.
title: Evaluating Real-World COVID-19 Vaccine Effectiveness Using a Test-Negative Case-Control Design
date: 2022-01-06
journal: nan
DOI: 10.1101/2022.01.06.22268726
sha: 839a5c2cce263e0e67c6836462e16b068e3fc517
doc_id: 296243
cord_uid: nam2fosn

It is important to assess the extent to which the real-world effectiveness of marketed vaccines is consistent with that observed in the clinical trials, and to characterize how well vaccines prevent COVID-19 symptoms. We conducted a modified test-negative design (TND) to evaluate the RW effectiveness of three COVID-19 vaccines by leveraging data from an on-going, US community-based registry. Vaccine effectiveness was examined in two ways: considering cases who (1) tested positive for COVID-19 (695 cases, 1,786 controls) and who (2) tested positive with at least one moderate/severe COVID-19 symptom (165 cases, 2,316 controls). Any vaccination (full or partial) was associated with a 95% reduction in the odds of having a positive COVID-19 test [adjusted odds ratio (aOR) = 0.05 (95% confidence interval (CI): 0.04, 0.06)]. Full vaccination was associated with an aOR of 0.03 (95% CI: 0.03, 0.05) while partial vaccination had an aOR of 0.08 (95% CI: 0.06, 0.12). Any vaccination was associated with a 71% reduction in the odds of testing positive and having at least one moderate/severe symptom (aOR=0.29 (95% CI: 0.20, 0.40)). High effectiveness was observed across all three vaccine manufacturers both for prevention of positive COVID-19 test results and prevention of moderate/severe COVID-19 symptoms.

with that observed in the clinical trials, and to characterize how well vaccines prevent COVID-19 23 symptoms. We conducted a modified test-negative design (TND) to evaluate the RW effectiveness of 24 three COVID-19 vaccines by leveraging data from an on-going, US community-based registry. Vaccine 25 effectiveness was examined in two ways: considering cases who (1) tested positive for COVID-19 (695 26 cases, 1,786 controls) and who (2) tested positive with at least one moderate/severe COVID-19 symptom 27 (165 cases, 2,316 controls). Any vaccination (full or partial) was associated with a 95% reduction in the CC-BY-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted January 6, 2022. ; https://doi.org/10.1101/2022.01.06.22268726 doi: medRxiv preprint decreased appetite, vomiting, persistent pain or pressure in the chest, trouble waking up after sleeping, 84 anxiety, feeling disoriented or having trouble thinking, depression, and insomnia or trouble sleeping. 85

We used a modified TND to evaluate the effectiveness of the COVID-19 vaccines. In contrast to a 86 traditional TND that is site-based and prospectively tests all participants at the point of care, this study 87 employs more efficient online recruitment of primarily unpaid volunteers via social media and self-88

reporting of COVID-19 vaccines, COVID-testing, results, COVID symptoms and severity. While this is a 89 diversion from the traditional approach, these participants have more direct and immediate access to the 90 relevant COVID-testing than to testing for other infectious diseases like influenza. To ensure that the 91 study population had the potential for vaccination, we restricted these analyses to participants who 92 reported a COVID-test result between March 1 through September 16, 2021, a time-period during which 93 vaccines were widely available in the United States. 94

We approached our analysis in two ways. Our first analysis (COVID-19 case positivity) defines cases as 95 those participants who were tested for any reason and reported a positive COVID-19 test result, and 96 controls as those participants who were tested for any reason and reported a negative COVID-19 result. 97

When participants reported both positive and negative test results in that time interval, we selected the 98 positive result and classified them as a case as of that date. When they reported multiple positive tests, 99

we randomly selected one of the results as the study test result; similarly, if a participant reported multiple 100 negative results, a random result was selected and deemed the study test result. Random selection was 101 used to ensure that there was no bias implemented by always taking earlier or later tests, knowing that 102 access to vaccination increased over time starting early in 2021. 103 A second analysis incorporates the severity of self-reported COVID-like symptoms. In this analysis, cases 104 were defined as participants who reported a positive COVID-19 test result and also reported at least one 105 moderate or severe COVID-19 symptom within +/-7 days of that test result. Controls included all 106 participants who reported a negative COVID-19 test result as well as those who reported a positive test 107 result but did not report any moderate/severe symptoms within +/-7 days of that test result. Symptoms 108 . CC-BY-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

The copyright holder for this preprint this version posted January 6, 2022. ;

were queried from participants at the time of positive or negative test report; when multiple surveys were 109 available within +/-7 days of the reported COVID-test, the survey with the largest number of 110 moderate/severe symptoms reported was chosen to assure we identified the highest severity of symptoms 111

Participants reports were used to determine each person's vaccination status, which was subsequently 113 days post-2 nd dose, or had received Janssen but were not yet 14 days post-vaccination. Five participants 120 who reported receiving doses from different manufacturers are treated as fully vaccinated using the same 121 time intervals described previously but were excluded from the manufacturer-specific analysis. If 122 participants noted vaccination prior to December 1, 2020 (n=10) or reported vaccination with a COVID-123 19 vaccine other than Janssen, Pfizer, or Moderna, they were excluded from this study (n=6). Also 124 excluded were those who reported the same date for their first and second vaccine doses for Moderna and 125

Pfizer (n=8). Boosters were largely unavailable at the time these data were collected and are not 126 considered in this study. 127

The core analyses aimed to compare the exposure of vaccinated (fully or partially) vs unvaccinated, but 128 also included sensitivity analyses separately for those that were fully vaccinated vs unvaccinated and 129 partially vaccinated vs. unvaccinated. Further, each vaccine manufacture was examined separately for 130 effectiveness using the primary study vaccine exposure definition (fully or partially vaccinated vs 131 unvaccinated). 132 . CC-BY-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

The copyright holder for this preprint this version posted January 6, 2022. Table  140 one 

Of the 2,481 participants who were tested and reported vaccination status, 695 individuals reported a 155 positive COVID-19 test result and 1786 individuals reported a negative test result. Among these 156 . CC-BY-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted January 6, 2022. (Table two) . Most participants (80.2%) reported not having any 158 moderate/severe symptoms; 6.2% having at least one moderate to severe symptom, and 13.6% having two 159 or more such symptoms. Among participants that reported a positive COVID-19 test, 12.5% of cases 160 reported at least one moderate/severe symptom as compared to 3.75% of those reporting a negative test. (Table two) . This resulted in an unadjusted odds 174 ratio (aOR) of 0.05 (95% CI: 0.04, 0.07) and adjusted odds ratio of 0.05 (95% CI: 0.04, 0.06), indicating 175 that being vaccinated was associated with a 95% reduction in the odds of having a positive COVID-19 176

The second analysis focused on evaluating the effectiveness of preventing at least one moderate/severe 178 COVID-19 symptom in COVID-19 positive cases and included 165 cases and 2,316 controls (Table  179 three). Fifty-seven cases reported being fully or partially vaccinated at the time of their positive COVID-180

19 test with at least one moderate/severe symptom, while 1547 control patients reported being fully or 181 . CC-BY-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted January 6, 2022. results at preventing symptomatic cases with an 86% effectiveness (aOR=0.14, 95% CI: 0.07, 0.28), 206 . CC-BY-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted January 6, 2022. ; https://doi.org/10.1101/2022.01.06.22268726 doi: medRxiv preprint followed by Janssen at 65% (aOR=0.35, 95% CI: 0.11, 1.15) and Pfizer at 62% (aOR=0.38, 95% CI: Our study also contributes to the broader understanding of vaccine effectiveness at mitigating moderate to 236 severe COVID-19 disease symptoms. While we observed the strongest effectiveness in the context of 237 preventing a positive test, these COVID-19 vaccines had substantial (~70%) effectiveness in preventing 238 one or more moderate to severe symptoms, noting that the severity of COVID-19 symptoms here requires 239 self-reporting, which may be difficult from a hospital bed or among very sick patients, and had no 240 requirement for or link to healthcare utilization (this study only included 21 participants that reported any 241 hospitalizations during the study period). Our findings show that COVID-19 vaccines protect against 242 moderate-to severe symptoms that did not necessarily require hospitalization or even interactions with the 243 healthcare system. 244

While vaccination status was clearly the most important variable in the multivariate models at explaining 245 the risk of a positive COVID-19 test, there were several other variables that were significant and may 246 provide insight into the bigger picture of COVID-19 risk and vaccine effectiveness. Participants who 247

reported a blood disorder in their baseline medical history were associated with a 3-fold risk of testing 248 positive for COVID-19. This blood disorder variable is broad, with our 66 respondents having been 249

prompted by examples that included blood clots, sickle cell disease, thalassemia, thrombocytopenia, or 250 other blood disease. Our finding in this heterogenous group requires some explanation. While it is known 251 that COVID-19 infection may be pro-thrombotic, and this may potentially exacerbate pro-thrombotic 252 blood disorders, our broad and heterogenous 'blood disorder' group may also have included conditions 253 associated with immunosuppression, or the use of immunosuppressive medications, in turn resulting in 254 decreased viral clearance and more severe disease. This result may highlight another potential at-risk 255 . CC-BY-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted January 6, 2022. ; https://doi.org/10.1101/2022.01.06.22268726 doi: medRxiv preprint population for increased risk of the COVID-19 infection, and merits further investigation to better define 256 the heamatological conditions associated with greater risk.Anaemia has been previously suggested to be 257 an independent risk factor for Covid-19 related mortality, and it would be very likely that our group also 258 included anaemic patients, but their exact contribution to the rather large effect observed is uncertain in 259 our data set. [28, 29] Also, being Caucasian was associated here with a more than a twofold increased risk 260 of testing COVID-19 positive, but no elevated risk of testing positive and being symptomatic; however, 261

we suspect that white race may actually be serving here as a proxy for access to COVID-19 testing rather 262 than having a direct causal relationship. 263

In contrast, participants who reported lung disorders were significantly less likely to test positive for 264 COVID-19 (aOR = 0.55; 95% CI: 0.37, 0.81). It is possible that there may be a common drug/treatment in 265 this group of participants that maybe be COVID-19 protective, but it also makes sense that those at 266 highest risk of severe COVID-19 outcomes may embark on other effective methods at preventing participants were tested for COVID-19 because they were symptomatic or exposed to someone who 294 tested positive, or if it was mandated for some other reason. It is possible that vaccinated individuals are 295 more likely to take a COVID-19 test out of caution, whereas unvaccinated individuals are not concerned 296 about COVID-19, driving their unvaccinated status., which could bias TND studies if testing practices 297 were differentially affected by vaccination status. While the study was conducted in a period not affected 298 by the more transmissible Delta and Omicron variants, we chose a period that reflects the circulating virus 299 during the Phase III clinical trials to assess consistency of results. This approach, design, and data 300 collection could also be implemented and/or re-directed in an efficient and effective manner for new and 301 emerging COVID-19 vaccine and treatment effectiveness questions. 302 303 304 . CC-BY-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted January 6, 2022. 

The authors declare that they have no known competing financial interests or personal relationships that 316 could have appeared to influence the work reported in this paper. 317 . CC-BY-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted January 6, 2022. ; https://doi.org/10.1101/2022.01.06.22268726 doi: medRxiv preprint CC-BY-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted January 6, 2022. ; https://doi.org/10.1101/2022.01.06.22268726 doi: medRxiv preprint 

. CC-BY-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted January 6, 2022. 328 . CC-BY-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted January 6, 2022. . CC-BY-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

The copyright holder for this preprint this version posted January 6, 2022. ; https://doi.org/10.1101/2022.01.06.22268726 doi: medRxiv preprint CC-BY-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

The copyright holder for this preprint this version posted January 6, 2022. ; https://doi.org/10.1101/2022.01.06.22268726 doi: medRxiv preprint CC-BY-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

The copyright holder for this preprint this version posted January 6, 2022. ; https://doi.org/10.1101/2022.01.06.22268726 doi: medRxiv preprint CC-BY-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

The copyright holder for this preprint this version posted January 6, 2022. ; https://doi.org/10.1101/2022.01.06.22268726 doi: medRxiv preprint

Efficacy and Safety of the mRNA-1273 SARS-CoV-2 Vaccine

Safety and Efficacy of the BNT162b2 mRNA Covid-19 Vaccine

Safety and Efficacy of Single-Dose Ad26.COV2.S Vaccine against Covid-19

Postlicensure Evaluation of COVID-19 Vaccines

The Use of Test-negative Controls to Monitor Vaccine Effectiveness: A 369 Systematic Review of Methodology

Covid-19 Vaccine Effectiveness and the Test-Negative 371

Evaluation of COVID-19 vaccine effectiveness: interim guidance

Estimating COVID-19 vaccine effectiveness against severe acute respiratory 375 infections (SARI) hospitalisations associated with laboratory-confirmed SARS-CoV-2: an evaluation 376 using the test-negative design: guidance document: World Health Organization. Regional Office for 377 Europe

Estimating Influenza Vaccine Effectiveness With the Test-Negative Design Using 379 Alternative Control Groups: A Systematic Review and Meta-Analysis

Basic principles of test-negative design in evaluating influenza 382 vaccine effectiveness

A comparison of the test-negative and the traditional case-control study designs for 384 estimation of influenza vaccine effectiveness under nonrandom vaccination

Seasonal Influenza Vaccine Effectiveness in People With Asthma: A National 387 Test-Negative Design Case-Control Study

Effectiveness of the Pfizer-BioNTech and Oxford-AstraZeneca vaccines 389 on covid-19 related symptoms, hospital admissions, and mortality in older adults in England: test negative 390 case-control study

Effectiveness of BNT162b2 and mRNA-1273 covid-19 vaccines against 392 symptomatic SARS-CoV-2 infection and severe covid-19 outcomes in Ontario, Canada: test negative 393 design study

Effectiveness of Covid-19 Vaccines in Ambulatory and Inpatient Care 395 Settings

Interim Estimates of Vaccine Effectiveness of Pfizer-BioNTech and Moderna 397 COVID-19 Vaccines Among Health Care Personnel -33

Effectiveness of COVID-19 Vaccines in Preventing SARS-CoV-2 Infection 400

Among Frontline Workers Before and During B.1.617.2 (Delta) Variant Predominance -Eight

Real-world effectiveness of BNT162b2 mRNA vaccine: a meta-analysis of 403 large observational studies

Effectiveness of Messenger RNA Coronavirus Disease

Vaccines Against Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) Infection in a 406 Cohort of Healthcare Personnel

Effectiveness of the Pfizer-BioNTech COVID-19 Vaccine Among Residents of 408 Two Skilled Nursing Facilities Experiencing COVID-19 Outbreaks -Connecticut

Effectiveness of Pfizer-BioNTech and Moderna Vaccines Against

Among Hospitalized Adults Aged >/=65 Years -United States

Interim Estimates of Vaccine Effectiveness of BNT162b2 and mRNA-414 1273 COVID-19 Vaccines in Preventing SARS-CoV-2 Infection Among Health Care Personnel

Self-reported symptoms from exposure to Covid-19 provide support to clinical 418 diagnosis, triage and prognosis: An exploratory analysis

Identification of a Vulnerable Group for Post-Acute Sequelae of SARS-CoV-2 420 (PASC): People with Autoimmune Diseases Recover More Slowly from COVID-19

How frequent are acute reactions to COVID-19 vaccination and who is at risk? 423 medRxiv

Outpatient Adult and Adolescent Subjects in Clinical Trials of Drugs and Biological Products for 426 COVID-19 Prevention or Treatment. In, 2020. 427 (27) Center for Disease Control and Prevention. COVID-19 Vaccinations in the United States

Protecting vulnerable patients with inherited anaemias from unnecessary death 430 during the COVID-19 pandemic

Mild anemia as a single independent predictor of mortality in patients with 432 COVID-19