key: cord-0948459-utp1p5au
authors: Lisewski, Andreas Martin
title: Interim estimates in null models of COVID-19 vaccine effectiveness
date: 2021-03-18
journal: Int J Infect Dis
DOI: 10.1016/j.ijid.2021.03.050
sha: 1cda2b228feed191647b1ccd785cea83e4c6c48a
doc_id: 948459
cord_uid: utp1p5au

Recently released interim numbers from advanced vaccine candidate clinical trials suggest that a COVID-19 vaccine effectiveness (VE) above 90% is achievable. However, SARS-CoV-2 transmission dynamics is highly heterogeneous and exhibits localized bursts of transmission, which may lead to sharp localized peaks in the number of new cases, often followed by longer periods of low incidence. Here we show that, for interim estimates of VE, this characteristic burstiness in SARS-CoV-2 infection dynamics may introduce a strong positive bias in VE. Specifically, we generate null models of vaccine effectiveness, i.e., random models with burstiness that over longer times converge to exactly zero VE, but that for interim times frequently produce apparent VE near 100%. As an example, by following the relevant clinical trial protocol, we can reproduce recently reported interim outcomes from an ongoing phase 3 clinical trial of a RNA based vaccine candidate. Thus, to avoid potential random biases in VE, it is suggested that interim estimates on COVID-19 vaccine effectiveness should control for the intrinsic inhomogeneity in both SARS-CoV-2 infection dynamics and in reported cases.

effectiveness, i.e., random models with burstiness that over longer times converge to exactly zero VE, but that for interim times frequently produce apparent VE near 100%.

As an example, by following the relevant clinical trial protocol, we can reproduce recently reported interim outcomes from an ongoing phase 3 clinical trial of a RNA based vaccine candidate. Thus, to avoid potential random biases in VE, it is suggested that interim estimates on COVID-19 vaccine effectiveness should control for the intrinsic inhomogeneity in both SARS-CoV-2 infection dynamics and in reported cases.

Keywords: SARS-CoV-2; COVID-19; vaccine; vaccine candidate; vaccine effectiveness

In the international race for vaccines against COVID-19 significant progress has been claimed recently, with Pfizer Inc. (New York, NY, USA) the first with reporting an interim analysis from their current phase III clinical trials of a RNA based vaccine candidate (Pfizer, 2020; Polack et al., 2020) . From this placebo-controlled, randomized, and observer-blind study a vaccine effectiveness (VE) of more than 90% has been reported based on a preliminary number of 94 confirmed cases of symptomatic SARS-CoV-2 infections accrued over 104 days (between July 27th and November 8th, 2020).

While these and other interim numbers (Callaway, 2020) are encouraging in the prospect of potential outcomes from these controlled clinical trials, long-term and thus more realistic estimates of VE will also depend on SARS-CoV-2 transmission features that are intrinsically more difficult to control over short-term.

One such prominent feature is the heterogeneity in intervals between consecutive SARS-CoV-2 infections, which over time leads to highly localized and seemingly random clusters (or, bursts) of recorded cases, followed by longer periods of relative inactivity (Adam et al., 2020) . This heterogeneity is potentially due to underlying superspreading events that are stratified from highly localized (e.g., household) to less localized levels (such as entire communities, see (Liu et al., 2020) ), and that appear to be driven mainly by symptomatic transmission (Kumar et al., 2021) . Statistical evidence for SARS-CoV-2 transmission heterogeneity has been given in the number distribution of secondary cases during superspreading events (Wong and Collins, 2020) , as well as in serial interval distributions (Du et al., 2020) . SARS-CoV-2 serial intervals often follow a log-normal distribution in which the measured mean μ can be smaller (Du et al., 2020) or larger (Nishiura et al., 2020) than the standard deviation σ. This observation is relevant because it allows a classification according to the burstiness parameter B = (σ -μ)/(σ + μ), known for various complex dynamical systems (Goh and Barabási, 2008) , which are characterized by intermittent, heterogeneous time series whenever 0 ≤ B < 1; for the opposite range, -1 < B < 0, their dynamics is fundamentally ordered in time, more homogeneous, and thus also predictable. The log-normal distribution, which theoretically covers the entire range of B between the two extremes -1 and 1, was proposed to be general and found in the statistics of many complex dynamical systems that exhibit burstiness, including the dynamics of viral infections (Goh and Barabási, 2008) .

To test if actual SARS-CoV-2 cases exhibit burstiness, we analyzed the distribution of time intervals in globally recorded SARS-CoV-2 cases from public data repositories that J o u r n a l P r e -p r o o f monitor the current pandemic (Supplementary Material, Fig. S1A and Methods). The analysis did indicate that these intervals follow a log-normal distribution, and that burstiness is present with a mean μ = 1.2 days, standard deviation of σ = 3.9 days, and with positive B = 0.5. A direct consequence of this observation would be that if within a background population SARS-CoV-2 infection dynamics is dominated by bursts (B > 0), then any random subgroup would also exhibit a similar level of heterogeneity. We verified this prediction by generating a series of random group samples from the original set of cases at decreasing sample size, followed by estimating the resulting degree of burstiness B. The data showed that the originally observed high level of burstiness (B = 0.5) was robust and did not decrease after such subsampling (Fig. S1B ). This evidence in turn suggests that burstiness itself could become a significant source of random bias for interim estimates of vaccine effectiveness. Specifically, as the log-normal distribution implies, one can construct an elementary class of null VE models with burstiness, i.e., statistical control models that over time converge to zero VE, but which at shorter interim times frequently produce apparent VE values close to 100%.

To see this effect of burstiness, consider two placebo (no vaccine given) randomized groups with equal number N1 = N2 = N of initially healthy (no COVID-19) individuals, and let a be the yearly SARS-CoV-2 attack rate such that the expected total number of cases after one year becomes Cmax = 2aN. Then use the log-normal distribution to generate a random sequence of new cases for each group such that after one year the accumulated number of cases in each group becomes Cmax/2, thus over time leading to zero VE. Finally, for all number of days t before one year, calculate the interim vaccine J o u r n a l P r e -p r o o f effectiveness, VE(t) = 1 -C2(t)/C1(t), where C1(t) and C2(t) are the accumulated case numbers in group 1 and 2, respectively. Figure S2 gives a representative output of this procedure with input values a = 1.3% and N = 21999, as per Pfizer's published clinical trial protocol (see, Supplementary Material Document S1, page 16). For this model realization, a conservative value of the burstiness parameter was set, B = 0.4, that in the absolute range of B was reduced by one-tenth from the originally observed high degree (B = 0.5). In the resulting model run, between days 48 and 90, the interim VE reaches above 90%, with 94 or more recorded cases that accumulate in several random bursts.

Direct model sampling with the above parameter settings suggests that this vaccine test requirement (VE > 90%, and C1(t) + C2(t) ≥ 94) is already met with a relative frequency of 2% (see Supplementary Material, Methods section), which further increases as burstiness increases (monotonically with B). In contrast, for an equivalent model but with a symmetrically opposite, low degree of burstiness, B = − 0.4, cumulative case numbers increase steadily, large interim fluctuations in vaccine effectiveness do not occur, and overall VE remains close to zero ( Figure S3 ). Thus, as an interim effect, high levels of COVID-19 vaccine effectiveness can be produced by random burstiness alone, i.e. without any immunization background.

These data suggest that advanced COVID-19 vaccine candidate clinical trials should address explicitly the potential inhomogeneity in recorded SARS-CoV-2 cases when releasing vaccine effectiveness data. In addition to the burstiness that characterizes SARS-CoV-2 transmission, the recording of every single positive case during an advanced clinical trial is also a random and variable process that depends on several J o u r n a l P r e -p r o o f external stochastic factors (Fig. S4 ). This process can extend over many days (Fig. S4) , from a trial participant's symptom onset to a standard clinical test result received and documented after a variable turnaround time (Chwe et al., 2020) . It is then, for example, difficult to bring in line these stochastic factors, which would only further amplify any inhomogeneity in recorded case numbers, with the "steadily accumulating cases" in the placebo group of the phase III clinical trial conducted by Pfizer, which remarkably shows no significant heterogeneity during nearly the first 100 days (see, page 57 and figure 13 in Supplementary Material Document S2, and figure 3 in (Polack et al., 2020) . Thus, in contrast to our observations (B > 0), Pfizer's phase III clinical trial data (Polack et al., 2020 ; Document S2) point to the absence of burstiness in accrued COVID-19 cases (B < 0). Consequently, to avoid artificially inflated VE in the more likely situation where case number heterogeneity does occur, cumulative incidence numbers should arguably be released after trial periods long enough to ensure that additional clusters of cases (bursts) do not cause strong fluctuations in these VE estimates. In the context of ongoing phase IV confirmatory trials, such additional control might also help to ensure that high levels of COVID-19 vaccine effectiveness observed over shorter times still persist after extended periods that would ultimately be necessary to stop a pandemic.

The author (AML) declares that there are no conflicts of interest.

The author (AML) declares that he received no specific funding for this work.

The author (AML) declares that no ethical approval was required for this work.

The author declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

J o u r n a l P r e -p r o o f

Clustering and superspreading potential of SARS-CoV-2 infections in Hong Kong

COVID vaccine excitement builds as Moderna reports third positive result

State of the Nation: A 50-State COVID-19 Survey Report #17: COVID-19 Test Result Times. Homel Secur Digit Libr 2020

Serial Interval of COVID-19 among Publicly Reported Confirmed Cases

Burstiness and memory in complex systems

Descriptive epidemiology of SARS-CoV-2 infection in Karnataka state, South India: Transmission dynamics of symptomatic vs. asymptomatic infections

Secondary attack rate and superspreading events for SARS-CoV-2

Pfizer and BioNTech Announce Vaccine Candidate Against COVID-19 Achieved Success in First Interim Analysis from Phase 3 Study

Safety and Efficacy of the BNT162b2 mRNA Covid-19 Vaccine

Evidence that coronavirus superspreading is fat-tailed