key: cord-0806418-t7v9jgij
authors: Lin, D.; Zeng, D.; Gilbert, P.
title: Evaluating the Long-Term Efficacy of COVID-19 Vaccines
date: 2021-01-21
journal: medRxiv : the preprint server for health sciences
DOI: 10.1101/2021.01.13.21249779
sha: 3cc13cdeb31d6ad89ddbb2767b41cdc2511f4c5e
doc_id: 806418
cord_uid: t7v9jgij

Large-scale deployment of safe and durably effective vaccines can halt the COVID-19 pandemic. However, the high vaccine efficacy reported by ongoing phase 3 placebo-controlled clinical trials is based on a median follow-up time of only about two months and thus does not pertain to long-term efficacy. To evaluate the duration of protection while allowing trial participants timely access to efficacious vaccine, investigators can sequentially cross placebo recipients to the vaccine arm according to priority groups. Here, we show how to estimate potentially time-varying placebo-controlled vaccine efficacy in this type of staggered vaccination of placebo recipients. In addition, we compare the performance of blinded and unblinded crossover designs in estimating long-term vaccine efficacy.

A number of studies have been conducted around the world to evaluate the efficacy and safety of investigational vaccines against novel coronavirus disease-2019 (COVID-19). 4 Although continuing blinded follow-up of the original treatment arms offers the ideal opportunity to evaluate long-term vaccine efficacy and safety, placebo recipients should be offered the vaccine at some point after an EUA. One strategy is the "rolling crossover", which vaccinates placebo recipients around the same time as general population members in the same priority tier. Under this design, placebo participants are vaccinated at different times, with the timing of vaccination depending on enrollment characteristics that define their priority tier.

In this article, we show how to evaluate long-term vaccine efficacy under the rolling crossover design. Because of staggered enrollment, existing statistical methods do not produce valid estimates of time-varying vaccine efficacy even if the placebo group is maintained throughout the study. Our approach properly accounts for staggered enrollment and change of background incidence over time, as well as for the fact that higher-risk placebo volunteers may get vaccinated earlier than lower-risk ones during the crossover period. It provides unbiased estimation of the entire curve of placebo-controlled vaccine efficacy as a function of time elapsed since vaccination, up to the point where most of the placebo recipients have crossed over to the vaccine arm. We investigate the bias and precision in estimating longterm vaccine efficacy under various blinded and unblinded crossover designs and discuss how to perform sensitivity analysis when unblinded follow-up data are used.

Methods Figure 1 provides a schematic illustration of the rolling crossover strategy in the context of the two mRNA vaccine trials. In this scenario, participants were screened and randomly assigned to vaccine or placebo over a 4-month period, and the vaccine was approved in the 5th month on the basis of interim results. Under the proposed design, crossover based on priority groups will occur over the next five months.

The endpoint of interest is time to symptomatic COVID-19 disease. We allow the background risk of disease to vary over the calendar time and to depend on baseline risk factors, such as age, gender, ethnicity, race, occupation, and underlying health conditions. In addition, we allow the relative risk of disease for vaccine versus placebo to depend on the duration elapsed since vaccination. We define the τ -day vaccine efficacy as the percentage reduction in the cumulative incidence of disease over the τ -day period for those who were vaccinated

We conducted a series of simulation studies mimicking the BNT162b2 vaccine trial. We considered 40,000 participants, who entered the trial at a constant rate over a 4-month period and were randomly assigned to vaccine or placebo in a 1:1 ratio. The vaccine received an EUA from FDA at the 5th month, by which time there were about 300 COVID-19 cases in the placebo group. To reflect the increase of COVID-19 cases since last summer and the expected downward trend in the spring due to herd immunity, we let the disease risk increase over the first 7 months and decrease afterward. We set the vaccine efficacy at 5 months to be 95% and the vaccine efficacy at 10 months to be 95%, 50%, or 0%.

We considered the statistically optimal design of keeping all participants on their original treatment assignments until the end of the trial, which was set to be slightly over 10 months.

We refer to this design as Plan A, which serves as a benchmark. We also considered three blinded crossover designs: B. Crossover occurs over months 6-10 according to priority tier groups.

C. 20% of participants follow Plan A, and the rest follow Plan B. D. Crossover occurs for all participants in month 6.

Both B and C are priority-tier dependent rolling crossover designs. The difference is that under Plan B, all placebo recipients cross over to the vaccine arm, whereas under Plan C, 20% of participants choose for altruistic reasons to stay on their original treatment assignments.

For Plan D, all placebo recipients are vaccinated over the course of a month without any priority tiering. With blinded crossover, placebo participants receive the vaccine and vaccine participants receive the placebo at the point of crossover, and all participants are followed until the end of the trial, which is 10.5 months since trial initiation. The designs of these simulation studies are detailed in Supplemental Appendix 2.

The results, based on 10,000 simulated datasets, are summarized in Table 1 and Figure   2 . The proposed method yields virtually unbiased estimates of the vaccine-efficacy curves over the 10-month period for Plans A-C in all three scenarios (95%, 50%, 0%) of long-term 4 All rights reserved. No reuse allowed without permission.

(which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted January 21, 2021. ; https://doi.org/10.1101/2021.01.13.21249779 doi: medRxiv preprint vaccine efficacy; it also yields accurate variance estimates, such that the confidence intervals have proper coverage probabilities. When vaccine efficacy is constant over time, the standard errors for the estimates of vaccine efficacy under Plans B and C are slightly lower than those of Plan A. When vaccine efficacy wanes over time, the standard errors for the estimates of 5-month vaccine efficacy under Plans B and C are also slightly lower than those of Plan A; however, the standard errors for the estimates of 10-month vaccine efficacy under Plans B and C are higher than those of Plan A, with the standard errors being slightly lower under Plan C than under Plan B. Under Plan D, the estimates of 10-month vaccine efficacy are slightly biased, with high standard errors; these results are not surprising, because under this plan, the number of unvaccinated participants diminishes rapidly after month 6.

We also evaluated the performance of the standard Cox regression, 16−17 with vaccine status as a potentially time-dependent covariate. Because it estimates an overall vaccine efficacy among individuals who have been vaccinated for different amounts of time, this method over-estimates long-term vaccine efficacy when vaccine efficacy decreases over time, although the estimation is unbiased when vaccine efficacy is constant over time.

We conducted a second series of simulation studies by considering unblinded crossover, under which participants are notified of their original treatment assignments and placebo recipients are vaccinated soon after. In this series, crossover occurs at the same time as in Plans B-D but is unblinded; we refer to these three plans as B'-D' (Supplementary Appendix 2). Because vaccine recipients may engage in riskier behavior upon unblinding, we discarded the data collected after unblinding for both the vaccine and placebo groups by censoring each participant's time to disease at their time of unblinding visit; the resulting analysis avoids any bias due to behavioral confounding.

The results for the second series of simulation studies, based on 10,000 simulated datasets, are summarized in Table 2 and (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted January 21, 2021. ; https://doi.org/10.1101/2021.01.13.21249779 doi: medRxiv preprint 5-month vaccine efficacy are unbiased; however, the estimates of 10-month vaccine efficacy are biased, especially when vaccine efficacy wanes over time. Naturally, long-term efficacy cannot be estimated as well under unblinded crossover as under blinded crossover because of limited blinded follow-up data, especially under Plan D'. Of note, the bias of the standard Cox regression in estimating waning vaccine efficacy is more severe under unblinded crossover than under blinded crossover. Figure 4 displays the analysis results produced by the proposed methods for one of the trials simulated with 50% 10-month vaccine efficacy. The estimates of vaccine efficacy are close to the truth, and the 95% confidence intervals cover the truth mostly -up to the end of blinded follow-up. In terms of estimating long-term vaccine efficacy, Plan C is nearly as good as Plan A and is slightly better than Plan B, which is better than Plan D; Plans B, C, and D are substantially better than Plans B', C', and D', respectively.

For a preventive COVID-19 vaccine to be administered to millions of people, including healthy individuals, its safety and efficacy must be demonstrated in a clear and compelling manner. Although preliminary results from ongoing phase 3 clinical trials have revealed higher than expected efficacy of COVID-19 vaccines, 4−6 additional follow-up is required to assess long-term efficacy and safety. Indeed, FDA does not consider issuance of an EUA, in and of itself, as grounds for stopping blinded follow-up in an ongoing clinical trial. 15 We recommend the rolling crossover design, which allows placebo recipients to be vaccinated in a timely manner while still making it possible to assess long-term safety and efficacy.

The standard Cox regression seriously over-estimates long-term vaccine efficacy when vaccine efficacy wanes over time, as shown in our simulation studies. We have developed a valid and efficient approach to evaluate the potential waning efficacy of a COVID-19 vaccine. The estimated curve of time-varying vaccine efficacy can be used to determine when a booster vaccination is needed to sustain protection; this information is also an important input parameter in mathematical modeling of the population impact of COVID-19 vaccines.

6 All rights reserved. No reuse allowed without permission.

(which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted January 21, 2021. ; https://doi.org/10.1101/2021.01.13.21249779 doi: medRxiv preprint

To ensure high-quality follow-up data, crossover should ideally be blinded, with placebo participants receiving vaccine and vaccine participants receiving placebo at the time of crossover. Unblinded crossover has practical benefits over blinded crossover: it reduces operational complexity and trial cost. However, unblinding can lead to differential exposure to SARS-CoV-2 between the original vaccinees and the placebo crossovers, which in turn can bias the estimation of vaccine efficacy.

This bias can be avoided by analyzing only the blinded follow-up data. However, discarding the unblinded follow-up data may substantially reduce the precision in estimating long-term vaccine efficacy. We suggest to estimate vaccine efficacy in two ways, one using all follow-up data and one using only blinded follow-up data, and compare the two results.

Another solution is to apply our methods to all follow-up data but perform a sensitivity analysis to assess the robustness of the results to potential unmeasured confounding caused by unblinding of trial participants. In Supplemental Appendix 3, we show how to apply a best-practice general methodology in epidemiological research 18−19 to perform this sensitivity analysis. Specifically, we can assess how strong unmeasured confounding due to unblinding would need to be in order to fully explain away the observed vaccine efficacy. We can also provide a conservative estimate of vaccine efficacy that accounts for unmeasured confounding.

Recently, Follmann et al 20 advocated blinded crossover and continued follow-up of trial participants to assess vaccine durability and potential delayed enhancement of disease. They (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted January 21, 2021. ; https://doi.org/10.1101/2021.01.13.21249779 doi: medRxiv preprint been vaccinated. Our approach requires minimal assumptions and is applicable to both blinded and unblinded crossover plans, with any length of additional follow-up. 9 All rights reserved. No reuse allowed without permission.

(which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted January 21, 2021. 10 All rights reserved. No reuse allowed without permission.

(which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted January 21, 2021. ; https://doi.org/10.1101/2021.01.13.21249779 doi: medRxiv preprint 11 All rights reserved. No reuse allowed without permission.

(which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted January 21, 2021. ; https://doi.org/10.1101/2021.01.13.21249779 doi: medRxiv preprint 12 All rights reserved. No reuse allowed without permission.

(which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted January 21, 2021. ; 13 All rights reserved. No reuse allowed without permission.

(which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. 14 All rights reserved. No reuse allowed without permission.

(which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. 15 All rights reserved. No reuse allowed without permission.

(which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. 16 All rights reserved. No reuse allowed without permission.

(which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted January 21, 2021. ;

Let T and S denote the time to symptomatic COVID-19 and the time to vaccination, respectively; both times are measured in days from the start of the clinical trial. In addition, let X denote baseline risk factors (e.g., age, occupation, comorbidities). We specify that the hazard function of T is related to S and X through a Cox proportional hazards model with a time-dependent regression coefficient One may define the t-day vaccine efficacy (VE) as the proportionate reduction in the cumulative incidence of disease at time t for individuals who have been vaccinated for t days:

which is approximately

where

For simplicity, we define the time-varying vaccine efficacy by equation (2) .

Suppose that the clinical trial enrolls a total of n participants. For i = 1, . . . , n, let R i , T i , S i , C i , and X i denote the entry time, the time to symptomatic COVID-19, the time to vaccination, the time to loss to follow-up, and the baseline risk factors for the ith participant.

The data consist of (

We assume that (R i , S i , C i ) are independent of T i conditional on X i . The likelihood takes the form

We approximate log λ 0 (t) through splines with m basis functions, B 1 (t), . . . , B m (t), such that log λ 0 (t) ≈ m k=1 γ k B k (t). Let θ = (β, γ 1 , . . . , γ m ) T , and Z i (t) = [X i , B 1 (t), . . . , B m (t)] T . We perform the nonparametric maximum likelihood estimation, 1 in which V (·) is treated as a step function jumping at the time points Y i with D i = ∆ i = 1. Thus, we maximize the objective function

where V {t} is the jump size of V (·) at t.

We first maximize the objective function in (3) for fixed θ to yield

Here and in the sequel, S (k) (θ; y) = n j=1 D j I( Y j ≥ y)e θ T Z j (y+S j ) Z j (y +S j ) ⊗k , where a ⊗0 = 1, a ⊗1 = a, and a ⊗2 = aa T for a column vector a. After plugging (4) into (3), we obtain the profile likelihood for θ. Differentiating the profile log-likelihood with respect to θ yields the 18 All rights reserved. No reuse allowed without permission.

(which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted January 21, 2021. ; https://doi.org/10.1101/2021.01.13.21249779 doi: medRxiv preprint estimating function

Denote the resulting estimator of θ by θ. Replacing θ in (4) by θ yields the estimator of V (t)

which is reminiscent of the Breslow estimator 2 for the cumulative baseline hazard function under the standard Cox model.

Using counting-process martingale theory 3−4 and other mathematical arguments, we can

show that V (t) is consistent and asymptotically normal. In addition, the variance of V (t)

can be consistently estimated by

We may define the vaccine efficacy at time t by the proportionate reduction in the hazard rate at time t, i.e., 1 − v(t). We can estimate v(t) through kernel smoothing of V (t). 4

We assumed that 40,000 participants entered the study at a constant rate over four months,

i.e., R ∼ Uniform(0, 4). We created a composite baseline risk score X, which takes values 1, 2, 3, 4, and 5 with equal probability. At study entry, half of the participants were assigned 19 All rights reserved. No reuse allowed without permission.

(which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted January 21, 2021. ; https://doi.org/10.1101/2021.01.13.21249779 doi: medRxiv preprint to vaccine and half to placebo. The statistically optimal design would be to maintain the original vaccine and placebo groups until the end of the study, which was set to be 10.5 months. We refer to this design as Plan A. We also considered three blinded crossover designs, under which placebo participants receive vaccine and vaccine participants receive placebo at the time of crossover and all participants are followed until the end of the study: Plan B. Crossover occurs at month (11 − X + G), where G follows the exponential distribution with mean of 0.5.

Plan C. 20% of participants follow Plan A and 80% follow Plan B.

Plan D. Crossover occurs at month 6 + G, where G follows the exponential distribution with mean of 0.5.

Plan C mimics a scenario where all participants are offered the option of crossover but a small percentage (20%) choose to stay on the original assignments for the sake of science.

In addition, we considered three unblinded crossover designs: Plan B'. Crossover occurs at month (11.5 − X).

Plan C'. 20% of participants follow Plan A and 80% follow Plan B'. Plan D'. Crossover occurs at month 6.5.

Under unblinded crossover, participants are notified of their original assignments at the time of crossover, and placebo participants receive the vaccine soon after. In practice, only placebo participants would cross over, since there is no need to give placebo to vaccine recipients after unblinding. In Plans B'-D', the time of crossover is the time of unblinding rather than the time when placebo participants actually receive the vaccine. Because participants might change their behavior upon discovering their original treatment assignments, we discarded the follow-up data collected after unblinding by censoring each participant's time to disease at their time of unblinding. This strategy avoids bias due to behavioral confounding, at the cost of reduced statistical efficiency.

We generated the event time T from model (1) with γ = 0.2, log λ 0 (t) = −5.93 + 0.1t − 0.3(t − 7) + , and log v(t) = a + bt, t > 0, 20 All rights reserved. No reuse allowed without permission.

(which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted January 21, 2021. ; https://doi.org/10.1101/2021.01.13.21249779 doi: medRxiv preprint where a and b were chosen to achieve the desired VE values at 5 and 10 months. We censored T at the time of unblinding crossover under Plans B'-D'.

For each simulated dataset, we estimated log λ 0 (t) using a piece-wise constant function with 20 pieces placed at the equal quantiles of the observed event times. We then estimated V (t) using the proposed method and estimated V E(t) according to equation (2) . For comparisons, we also fit a Cox model that includes X and time-dependent covariate I(S < t) to estimate VE by 1 minus the estimated hazard ratio of the time-dependent covariate.

We suggest reporting the E-value 5 as a summary measure of the evidence against the null hypothesis H 0 : V E(t) ≤ 0. The E-value is the minimum strength of association on the risk ratio scale, i.e, RR(t) = 1 − V E(t), that an unmeasured confounder would need to have with both vaccination status and disease outcome in order to fully explain away a specific observed vaccine efficacy. Let RR(t) be the estimate of RR(t), and let U L(t) be the upper limit of the 95% confidence interval for RR(t). Then the E-value for RR(t) is given by e(t) = 1 + 1 − RR(t)

, provided that RR(t) < 1. In addition, the E-value for U L(t) is computed as 1 if U L(t) ≥ 1 and as e U L (t) = 1 + 1 − U L(t) U L(t)

otherwise. E-values near one suggest weak support for a causal inference, and greater Evalues provide increasing evidence for causality.

Suppose, for example, that RR(t) = 0.50, with 95% confidence interval (0.08, 0.75).

Then e(t) = 3.4, meaning that the result of RR(t) being less than one could be explained away by an unmeasured confounder associated with both vaccination status and disease by a risk ratio of 3.4-fold each after accounting for the vector X of measured confounders, but not by a weaker unmeasured confounder. In addition, e U L (t) = 2.5, which is the strength of unmeasured confounding at which statistical significance for V E(t) > 0 would be lost.

All rights reserved. No reuse allowed without permission.

(which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted January 21, 2021. ; https://doi.org/10.1101/2021.01.13.21249779 doi: medRxiv preprint

These two E-values judge how confident we can be that V E(t) truly exceeds 0, accounting for potential unmeasured confounding due to unblinding as well as for sampling variability.

We can provide a conservative estimate of V E(t) that accounts for potential unmeasured confounding. 6 Let RR U D (t) be the maximum risk ratio for disease when comparing any two categories of the unmeasured confounder U , within either the vaccinated group or the unvaccinated group, conditional on the vector X of observed covariates, and let RR EU (t) be the maximum risk ratio for any specific level of the unmeasured confounder U when comparing the vaccinated and unvaccinated individuals. Define the bias or bounding factor

Then a conservative (lower bound) estimate of the vaccine efficacy is given by 1− RR(t)B(t), and a conservative confidence interval is obtained by multiplying the lower and upper limits of the confidence interval for RR(t) by B(t).

COVID-19 vaccine trials should seek worthwhile efficacy

Combination prevention for COVID-19

A strategic approach to COVID-19 vaccine R&D

Safety and efficacy of the BNT162b2 mRNA Covid-19 vaccine

Efficacy and safety of the mRNA-1273 SARS-CoV-2 vaccine

Safety and efficacy of the ChA-dOx1 nCoV-19 vaccine (AZD1222) against SARS-CoV-2: an interim analysis of four randomised controlled trials in Brazil, South Africa, and the UK

Discussion of the paper by D. R. Cox

Counting Processes and Survival Analysis

Statistical Models Based on Counting Processes

Sensitivity analysis in observational research: introducing the E-value

Sensitivity analysis without assumptions

The authors thank Yu Gu and Bridget I. Lin for assistance. This work was supported by the National Institutes of Health grants R01 AI029168, R01 GM124104, P01 CA142538, and UM1 AI068635.