key: cord-0702210-6e0lr8to
authors: Kahn, Rebecca; Schrag, Stephanie J; Verani, Jennifer R; Lipsitch, Marc
title: Identifying and alleviating bias due to differential depletion of susceptible people in post-marketing evaluations of COVID-19 vaccines
date: 2022-01-27
journal: Am J Epidemiol
DOI: 10.1093/aje/kwac015
sha: 81b669204dfe4729c5ef24b9c12617238aae5199
doc_id: 702210
cord_uid: 6e0lr8to

Recent studies have provided key information about SARS-CoV-2 vaccines’ efficacy and effectiveness (VE). One important question that remains is whether the protection conferred by vaccines wanes over time. However, estimates over time are subject to bias from differential depletion of susceptibles between vaccinated and unvaccinated groups. Here we examine the extent to which biases occur under different scenarios and assess whether serologic testing has the potential to correct this bias. By identifying non-vaccine antibodies, these tests could identify individuals with prior infection. We find in scenarios with high baseline VE, differential depletion of susceptibles creates minimal bias in VE estimates, suggesting that any observed declines are likely not due to spurious waning alone. However, if baseline VE is lower, the bias for leaky vaccines (that reduce individual probability of infection given contact) is larger and should be corrected by excluding individuals with past infection if the mechanism is known to be leaky. Conducting analyses both unadjusted and adjusted for past infection could give lower and upper bounds for the true VE. Studies of VE should therefore enroll individuals regardless of prior infection history but also collect information, ideally through serologic testing, on this critical variable.

to wear a face covering), the riskiest individuals will be depleted preferentially among the unvaccinated group when the vaccine is effective, leading to the same bias downwards in VE, growing over time and thus seemingly showing waning of VE (1) .

Serologic testing for SARS-CoV-2 antibodies has the potential to help correct the first bias. By identifying non-vaccine antibodies (e.g. N-protein), these tests could be used to identify individuals with prior infection and exclude them from studies of VE over time. Likewise, adjustment for individual-level risk of infection (in practice, for proxies such as occupation or behavior) can help address the second bias.

While each of these issues can in principle affect VE estimates and induce a spurious impression of waning VE, the magnitude of this bias under various assumptions about baseline VE is not clear, nor has it been shown before to our knowledge how adjustments can solve the problems. Here we examine the extent to which these biases occur under different scenarios and assess approaches to alleviate bias under various assumptions.

We first create a network model of 20,000 individuals, similar to models described previously (2, 5) . The probability of connections between individuals in the network is calibrated in combination with the parameter for the probability of infection given contact to result in a reproduction number (R) of 1.25 or 1.50 (see Table 1 for a full list of parameters) (6) . We seed an epidemic of a SARS-CoV-2-like pathogen with ten exposed individuals. Each day, each susceptible individual has a daily probability of infection from their infected connections in the network. A random half of the population is high risk, and the other half is low risk. High risk individuals have a daily probability of infection three times that of low risk individuals. This binary

risk status is a simplified proxy for multiple factors that could affect individuals' risks for infection, such as occupation, demographics, geography, or behavioral patterns (7) (8) (9) .

We assume that half of those who are infected become symptomatic and that people are infectious for seven days. We assume that symptomatic, pre-symptomatic, and asymptomatic infected individuals have the same level of infectiousness. After individuals recover, we assume that complete protection from natural immunity lasts for 90 days (10), after which individuals can be reinfected; we then assume recovered individuals' susceptibility is 95% lower than those without prior infection, resulting in low numbers of reinfection during the study period examined in the simulations (11) . It is unknown exactly how VE differs for recovered individuals, although there is evidence that vaccination further reduces previously infected individuals' risk (12) . For simplicity we assume vaccinated recovered individuals' susceptibility is further decreased by the same amount as for vaccinated susceptible individuals.

We simulate random vaccination (to prevent unmeasured confounding) of 2500 individuals, or 12.5% of the population, on the first day of the simulation. Another 2500 unvaccinated individuals are also randomly selected for potential follow-up over the course of the simulations.

We compare four primary scenarios (Table 2 ). In the first scenario, vaccine efficacy against susceptibility to infection (VE S ) is 0.90, and vaccine efficacy against progression to symptoms (VE P ) is 0.5. These measures combine to give a vaccine efficacy against symptomatic disease (VE SP ), the primary outcome of most SARS-CoV-2 vaccine trials (13) (14) (15) (16) , of 0.95, under the (17) . These values are similar to those that have been observed in the trials (13, 15) and initial observational studies (18) (19) (20) of the mRNA vaccines. In the second scenario, we assume VE S and VE SP are 0.7, similar to the findings from the Janssen vaccine trial (14) .

In the first two scenarios, we assume the vaccine is "leaky", meaning it reduces the probability of infection given contact to an equal degree, but not perfectly, in all vaccinated individuals (3).

However, in the third scenario, to assess the impact of the vaccine mechanism, we model an all-or-nothing vaccine, meaning it protects a certain proportion of vaccinated individuals completely and provides no protection to the rest. In this scenario, VE S and VE SP are both 0.9. In supplementary scenarios, we also examine an all-or-nothing vaccine with lower VE S and VE SP , as well as a leaky vaccine with VE SP = 0.95, similar to Scenario 1, but with lower VE S and higher VE P. (21) .

Finally, in scenario 4, we examine a setting with a leaky vaccine with VE SP = 0.95 in which some of the population has already been infected and recovered before the simulations and vaccination begin. We explore a range from 0-30% of individuals with prior infection under a higher R than in the other scenarios (R=2.0) to prevent herd immunity from prior infections from substantially slowing the epidemics before spurious waning can be observed. In these simulations, 20 individuals are exposed on the first day and 100 individuals are infectious (except in the simulations with 0 individuals previously infected).

We In the first analysis (baseline), we estimate VE SP by calculating the odds ratio (OR), using data from all individuals sampled:

, where D is disease (symptoms and a positive virologic test) and V is vaccine.

In the second analysis, we estimate the OR using logistic regression, controlling for risk (i.e. the binary measure described above for increased or decreased susceptibility to infection).

In the third analysis, we simulate serologic testing for non-vaccine antibodies (i.e. evidence of past infection) and then restrict the analysis to individuals who had not previously been infected.

In the fourth analysis, we both restrict to those without evidence of previous infection and also control for risk. In the primary analyses, we assume perfect sensitivity and specificity of the serologic test for prior infection, but we relax these assumptions in sensitivity analyses. We examine lower sensitivity for cases and controls and lower specificity for cases only, as antibodies detected could reflect either current or prior infection.

As a comparison to the TND, we repeat the same four analyses to estimate VE using a cohort design, where the time of symptomatic infection is known for the 5000 people under follow-up.

We again examine different lengths of follow-up for this study design. We assume no unmeasured confounding: that is, no common causes of vaccination and infection, as would be true with adequate control for confounders. In practice, this study could be done using an

electronic health records database using stratification, matching, or modeling for example to control for confounding factors such as occupation, age, insurance, and other factors affecting both vaccination and the likelihood of infection given vaccination. Because there is no unmeasured confounding and vaccination is random in these simulations, this design is comparable to a randomized controlled trial (RCT) in which all symptomatic cases are identified.

We vary key parameters of interest to examine their impact on the results. First, we vary the reduction in susceptibility conferred by past infection, using a value of 70% (24) reduction compared to the baseline parameter of 95% reduction. Next, we vary the proportion of the population that is high risk, examining a scenario in which only 10% of the population is high risk, with five times higher risk than lower risk individuals. Finally, we vary the proportion of infections that are symptomatic, using a higher value of 0.8 (23) compared to the baseline of 0.5.

Ethics: This activity was reviewed by CDC and was conducted consistent with applicable federal law and CDC policy.

In scenario 1 with high VE S and VE SP , we find that for most time points, all four TND analyses Table 1) In scenario 4, we see that the degree of spurious waning bias increases with the number of cumulative cases since vaccination (Web Figure 3 and Web Table 2 ). This trend occurs because the bias is driven by differential depletion of susceptibles between vaccinated and In the cohort study analysis, which under our assumptions is equivalent to an RCT, we find similar trends to those observed in the TND (Figures 4-6) ; however, the cohort studies that do not exclude those with prior infection are less biased than equivalent TND studies for each scenario. In scenario 1, with the highest VESP, the bias is negligible. The bias is smaller because in cohort studies and RCTs, those with past symptomatic infection are censored at the time of infection, meaning fewer people are incorrectly treated as still at risk in the analysis.

In sensitivity analyses, we relax assumptions of perfect tests for prior infection and examine lower sensitivity for both cases and controls and lower specificity for cases. We find that while lower sensitivity results in slight downward biases of the estimates, which are more pronounced in scenario 2 than scenario 1 (Web Figures 4 and 5) , lower specificity for cases does not induce a bias (Web Figure 6 ). This is because imperfect specificity only reduces the sample size, but we assume it does not do so differentially by vaccination status. A study of breakthrough infections in Israel found antibody levels on day of diagnosis were not greatly impacted by the current infection, suggesting imperfect specificity may not be a large concern (25).

Finally, in analyses that vary the parameters for reduction in susceptibility following infection and the proportion of the population at high risk, we find similar results to the baseline scenario (Web Figures 7-8) . In analyses with a higher proportion symptomatic, we find less bias as expected given that a smaller proportion of cases will go undetected (Web Figure 9 ); we focus here on the cohort/RCT designs in which all symptomatic cases are identified and therefore the proportion symptomatic is a key parameter of interest.

We find that in scenarios with high baseline VE, differential depletion of susceptibles creates Moderna showing consistent efficacy over time (26) and Pfizer's estimates slightly declining (27) . It is challenging to disentangle if this decline is due to lower effectiveness against variants, true waning, spurious waning, or some combination of these factors; given the minimal bias found in our RCT-like analysis of vaccines with high VE (Figure 4 ), our findings suggest the decline is likely not due to spurious waning alone. Similarly, spurious waning is likely not the only cause of the declines in effectiveness observed in Israel, given the high effectiveness estimated when vaccines were first given (19, 20) , the magnitude of the declines and that they occurred following a period of low incidence (28) .

If baseline VE is lower, the bias over time for leaky vaccines is larger and ideally should be corrected if the mechanism is known to be leaky. However, leaky and all-or-nothing mechanisms are two extremes; in reality, vaccines will fail to take in some individuals due to This study has several limitations. First, we make many simplifying assumptions in the model.

For example, we assume all individuals are grouped into one large community and do not examine the potential impact of geographic heterogeneity. Other studies have shown epidemic dynamics due to differences in geography are important to control for in vaccine (29) and serologic (5) studies. We also assume perfect sensitivity and specificity of virologic tests, as implications of these parameters have been explored in detail previously (30, 31) . There are many potential biases in studies of vaccine effectiveness, which are described in detail in World

Health Organization guidance (30); here we focus specifically on spurious waning bias from differential depletion of susceptibles. While we incorporate heterogeneity in risk of acquiring infection, we do not model differences in risk of transmitting infection (e.g. due to host factors).

Second, using serologic tests to identify prior infection is subject to error from imperfect test characteristics and waning of antibodies over time. However, we find only small biases in VE estimates from imperfect sensitivity, and information on past infection can also be obtained through self-report or medical records. Third, as described above, we assume random vaccination and no unmeasured confounding; the strategies discussed here alone do not address most other sources of potential confounding, which are important to account for in analyses, particularly given that vaccine rollout in some cases prioritized those at highest risk to receive vaccines first. Fourth, we simulated epidemics with higher R values than much of the United States experienced during most of the pandemic to uncover scenarios where spurious waning might be of concern (https: Table 3 for number of cases, which refers to the median number of people with COVID-19 included in that day's analysis, and cumulative cases, which refers to the median total number of cases of COVID-19 by that day since vaccination (denominator 5000). Table 3 for number of cases, which refers to the median number of people with COVID-19 included in that day's analysis, and cumulative cases, which refers to the median total number of cases of COVID-19 by that day since vaccination (denominator 5000). Vaccine efficacy against symptomatic disease for scenario 2 (VE SP = 0.7, VE S = 0.7, VE P = 0 for a leaky vaccine) with a cohort/randomized controlled trial design. Columns are days since vaccination, and rows are values of the reproduction number R. Median and IQR of 100 simulations shown. See Table 3 for number of cases, which refers to the median number of people with COVID-19 included in that day's analysis, and cumulative cases, which refers to the median total number of cases of COVID-19 by that day since vaccination (denominator 5000). Table   3 for number of cases, which refers to the median number of people with COVID-19 included in that day's analysis, and cumulative cases, which refers to the median total number of cases of COVID-19 by that day since vaccination (denominator 5000). 

Depletion-of-susceptibles bias in influenza vaccine waning studies: how to ensure robust results

Analyzing Vaccine Trials in Epidemics With Mild and Asymptomatic Infection

Assessment of the Protective Efficacy of Vaccines against Common Diseases Using Case-Control and Cohort Studies

Measurement of Vaccine Direct Effects Under the Test-Negative Design

Potential Biases Arising from Epidemic Dynamics in Observational Seroprotection Studies

Network theory and SARS: predicting outbreak diversity

US-county level variation in intersecting individual, household and community characteristics relevant to COVID-19 and planning an equitable response: a cross-sectional analysis

Community Use of Cloth Masks to Control the Spread of SARS-CoV-2

Reductions in commuting mobility correlate with geographic differences in SARS-CoV-2 prevalence in New York City

Interim Guidance on Ending Isolation and Precautions for Adults with COVID-19

SARS-CoV-2 antibody-positivity protects against reinfection for at least seven months with 95% efficacy

Reduced Risk of Reinfection with SARS-CoV-2 After COVID-19

Efficacy and Safety of the mRNA-1273 SARS-CoV-2 Vaccine

Safety and Efficacy of Single-Dose Ad26.COV2.S Vaccine against Covid-19

Safety and Efficacy of the BNT162b2 mRNA Covid-19 Vaccine

Single-dose administration and the influence of the timing of the booster dose on immunogenicity and efficacy of ChAdOx1 nCoV-19 (AZD1222) vaccine: a pooled analysis of four randomised trials

Design and Analysis of Vaccine Studies

Interim Estimates of Vaccine Effectiveness of BNT162b2 and mRNA-1273 COVID-19 Vaccines in Preventing SARS-CoV-2 Infection Among Health Care Personnel, First Responders, and Other Essential and Frontline Workers -Eight U.S. Locations

BNT162b2 mRNA Covid-19 Vaccine in a Nationwide Mass Vaccination Setting

Impact and effectiveness of mRNA BNT162b2 vaccine against SARS-CoV-2 infections and COVID-19 cases, hospitalisations, and deaths following a nationwide vaccination campaign in Israel: an observational study using national surveillance data

Decreased infectivity following BNT162b2 vaccination: A prospective cohort study in Israel. The Lancet Regional Health -Europe

Theoretical Basis of the Test-Negative Study Design for Assessment of Influenza Vaccine Effectiveness

The case test-negative design for studies of the effectiveness of influenza vaccine. Vaccine [electronic article

Assessment of protection against reinfection with SARS-CoV-2 among 4 million PCR-tested individuals in Denmark in 2020: a population-level observational study

Covid-19 Breakthrough Infections in Vaccinated Health Care Workers

Moderna Reports Second Quarter Fiscal Year 2021 Financial Results and Provides Business Updates

Safety and Efficacy of the BNT162b2 mRNA Covid-19 Vaccine through 6 Months

Waning Immunity after the BNT162b2 Vaccine in Israel

Impact of stochastically generated heterogeneity in hazard rates on individually randomized vaccine efficacy trials

Test sensitivity is secondary to frequency and turnaround time for COVID-19 screening

Quantifying antibody kinetics and RNA detection during early-phase SARS-CoV-2 infection by time since symptom onset