key: cord-0796140-2apag5dt authors: Black, Steven B; Law, Barbara; Chen, Robert T.; Dekker, Cornelia L.; Sturkenboom, Miriam; Huang, Wan-Ting; Gurwith, Marc; Poland, Greg title: The Critical Role Background Rates of Possible Adverse Events in the Assessment of COVID-19 Vaccine Safety date: 2021-03-06 journal: Vaccine DOI: 10.1016/j.vaccine.2021.03.016 sha: f9fee385aa7f73b12c4597005759c8867d7197cd doc_id: 796140 cord_uid: 2apag5dt Beginning in December of 2019, a novel coronavirus, SARS-CoV-2, emerged in China and is now a global pandemic with extensive morbidity and mortality. With the emergence of this threat, an unprecedented effort to develop vaccines against this virus began. As vaccines are now being introduced globally, we face the prospect of millions of people being vaccinated with multiple types of vaccines many of which use new vaccine platforms. Since medical events happen without vaccines, it will be important to know at what rate events occur in the background so that when adverse events are identified one has a frame of reference with which to compare the rates of these events so as to make an initial assessment as to whether there is a potential safety concern or not. Background rates vary over time, by geography, by sex, socioeconomic status and by age group. Here we describe two key steps for post-introduction safety evaluation of COVID-19 vaccines: Defining a dynamic list of Adverse Events of Special Interest (AESI) and establishing background rates for these AESI. We use multiple examples to illustrate use of rates and caveats for their use. In addition we discuss tools available from the Brighton Collaboration that facilitate case evaluation and understanding of AESI. Beginning in December of 2019, a novel coronavirus, SARS-CoV-2, emerged in China and is now a global pandemic with extensive morbidity and mortality. With the emergence of this threat, an unprecedented effort to develop vaccines against this virus began. Several manufacturers have now developed and evaluated vaccines in less than one year; some are now being used extensively under emergency or conditional use authorizations in many countries. While unprecedented and based upon thorough evaluation of high quality data, this rapid development and authorization process has meant that vaccines have been introduced for use in the population based on studies with limited population heterogeneity (e.g., no children, pregnant, immunocompromised persons), and limited follow up (~2 months) on recipients in clinical trials. Furthermore, despite the fact that large phase III trials were conducted and completed, the "warp speed" with which the vaccine development process proceeded has augmented vaccine hesitancy concerns expressed by many anti-vaccination groups, the general public, and some health care workers that the process was "rushed" or incomplete. These concerns have arisen despite the fact that large phase III clinical trials were appropriately conducted and completed and that while several administrative and regulatory factors helped to speed up the process, none of the routine safety and efficacy evaluation steps have been bypassed. Indeed, this speed was possible for several reasons: First, because of the emergency, abundant funding was available; second, clinical studies that were usually done consecutively were done in parallel and interntionally; third, enrollment into the trials was very rapid as many volunteers wanted to participate; and finally, regulators conducted rolling reviews of studies minimizing regulatory time delays. Evidence regarding vaccine safety plays a key role in public acceptance of a new vaccine. The most convincing evidence of a causal relationship between a new vaccine and an adverse event derives from randomized controlled trials (RCT) examining whether a statistically significant higher rate of an adverse event (e.g., fever) occurs in the vaccinated group compared to the control group, plus other factors such as biologic plausibility and clustering of onset interval (time elapsed between vaccination and onset of the adverse event). We must recognize that even though these phase III COVID-19 vaccine trials were large (N~30,000-60,000), the follow-up information on individuals in the trials was relatively short prior to authorization (8-12+ weeks) and more importantly, any large phase III trial has limited statistical power to detect rare events that might be occur when millions and even billions of people globally are vaccinated to control a pandemic. For example, the risk of anaphylaxis following the first two m-RNA COVID-19 vaccines appears to be ~2.8-5 /million doses from post-introduction pharmacovigilance -a rate that was too low to be detected in the phase III trials which as currently conducted have an ability to detect events that occur at a frequency of 1:10,000. 1 Once a new vaccine is approved by a National Regulatory Authority [NRA) and recommended for use by the National Immunization Technical Advisory Group, many experts state that it is no longer ethical to withhold a vaccine from an approved target population. 2 Therefore, the ability to infer causality for adverse events from an RCT by comparison with a randomized unvaccinated arm may no longer be feasible. Since humans fall ill due to many causes in the absence of COVID-19 vaccine (including COVID-19 disease), how do we then distinguish if an adverse event is causally related to the COVID-19 vaccine or not? The first way is to see if there is unique mechanistic laboratory-based evidence. For example, a vaccinee is found to have developed a disease caused by the virus strain such as isolation of mumps vaccine strain virus from the CSF of a patient with aseptic meningitis. 3 The second way is if vaccine recipients develop a unique or relatively unique clinical syndrome not otherwise found in this population. 4 The third way is if the adverse event recurs after a second dose of the same vaccination, i.e. a "re-challenge" phenomenon. 5 The fourth way is to collect data in as unbiased a manner as possible from active surveillance observational epidemiologic studies to detect whether the rate of a given event or syndrome in vaccinated individuals exceeds that expected among unvaccinated individuals. In the modern era of rapid dissemination of rumors on social media, a rapid response to a vaccine safety signal is needed to maintain public confidence. Thus, now that large vaccination programs have started, it is critical to provide ongoing information to the public on the safety of the vaccines rapidly and transparently. Already, a concern regarding the risk of anaphylaxis following either of the m-RNA vaccines as well as death in the frail elderly has raised public concern. 6 , 7 Because the first three ways of establishing causal relationship between a vaccine and an adverse event are quite rare, the fourth way, observational epidemiologic studies, is the main method available. In such studies one investigates whether the occurrence of the event changes after vaccination as any factor that is causal should have an impact on disease occurrence. One then compares the risk of an event between vaccinated and unvaccinated individuals or, in some analytic framewords, between exposed and unexposed follow up time in the same individual. 8 A rapid response to a vaccine safety signal is key to maintaining public confidence in mass vaccination programs. Traditionally when a signal is identified, the first step is to calculate whether the occurrence of the event is more frequent than one would expect. In order to do this, one needs background rates of the occurrence of the event to assess what the incidence is in the absence of a vaccine. Such background rates are preferably established before vaccine introduction and therefore a list of adverse events of interest should be established. Accordingly, here we describe two key steps for post-introduction safety evaluation of COVID-19 vaccines:  Defining a dynamic list of Adverse Events of Special Interest (AESI)  Establishing background rates for these AESI Given that COVID-19 is a new disease and SARS-CoV-2 is a newly emerged and still emerging virus, attempts have been made to systematically try and understand what safety events might occur following a variety of SARS-CoV-2 vaccine platforms. As part of its work to support the safety assessment of vaccine development funded by the Coalition for Epidemic Preparedness Importantly, one should recognize that lists of AESIs are dynamic and will be updated as knowledge regarding possible events of concern are identified. Most simply stated, the background rate of any adverse event (e.g., heart attack, seizure, death) is the incidence rate of the event one would observe in a given population in the absence of receipt of the vaccine being tested or any other intervention. This background rate can be assessed in the general population, in a subpopulation such as pregnant women or children, or in populations with pre-specified co-morbidities. The background rate is used to calculate the number of expected cases of an event in a given population and time period in the absence of vaccine or other intervention that can be compared with the number observed following vaccination. Preferably such data are reasonably contemporaneous given that event incidence can change over time even without a vaccine being introduced. In order to understand the rationale for and use of background rates, it is important to understand the difference between temporal association and causality. It is often said that "bad things happen all the time". That is, in the absence of the use of a vaccine, adverse events or "bad things" will occur in a population because most diseases have multiple causes. Under routine conditions without receipt of vaccine, people will develop transverse myelitis, anaphylaxis, or die unexpectedly. In addition, pregnant women will experience spontaneous abortions. The fact that such events occur within a plausible time window following vaccination does not necessarily mean that the vaccine caused the event, i.e. temporality is not causality. However, an observation of temporal association can provide a hypothesis for further evaluation. A given health event might have occurred without the individual having received a vaccine dose, as is frequently observed in the placebo arms of clinical trials. To epidemiologically assess whether the vaccine is associated with an event, one must show that the risk of the event is higher than in a comparable non-vaccinated group. As explained above, many background rates for AESI can be assessed in advance of the introduction of a vaccine and thus can be used to rapidly evaluate and respond to potential safety concerns after vaccine rollout. Background rates can be obtained from the literature or generated de novo from retrospective analysis of electronic healthcare databases. Availability of background rates is important for rapid evaluation of a safety signal that may be identified through passive reporting or through social media. In the face of a dramatic media report of a serious event that followed a new vaccine at the start of a public vaccination campaign, knowing and communicating the expected background rate can provide reassurance to the public. If a cluster of adverse events that are potentially vaccine-associated is observed, one can rapidly calculate the rate of this event in vaccinees and compare this to the "expected number of events" or background rate. If the rate of the event in vaccinees is equal to or less than the background rate, then one can be reassured that the events may have occurred following vaccination by chance. However, one should not use or apply rates without due caution as rates can vary over time, by age, sex, geography and other factors as discussed below. It should also be noted that background rates in a comparison group or in unexposed follow up time are a critical component of active surveillance studies using a cohort or self control case series approach respectively. For events in Table 1 , case definitions and evaluation tools are being prepared by the SPEAC project. These are available on the Brighton Website and discussed further below. For convenience we have categorized illustrative examples of background rates as relatively common, somewhat rare, and extremely rare in Table 2 . Some outcomes are remarkably common; for example, up to 21.2% of all pregnancies end in a spontaneous abortion in women 18-24 years of age in Finland. Other outcomes are very rare such as Kawasaki disease in Europe with only 1.55 cases occurring in a population of 10 million people being observed over a one week time period. While it is easy to search the literature and identify potential sources of background rates for most outcomes, care must be taken to use appropriate comparisons and to be aware of the limitations of the data. In order to reliably evaluate the rate of an AESI or other event and compare it to a background rate, a standard case definition should be used to validate cases. However, it is important to recognize that many published background rates are "crude" rates in that the cases have not been validated. For appropriate comparisons, "like should be compared with like" -that is, validated rates with validated rates and unvalidated rates with unvalidated rates. In addition, due to changes over time and by season for some events, rates should be compared in a similar time frame when possible. This is especially critical for rare events. For a given event, several factors may impact on a background rate including: the year(s) when measured; the age and gender distribution of the population, geographic location, comorbidities, socioeconomic status, medication use, and study methodology. Some specific examples are provided in Table two and discussed below. As can be seen, rates vary by the time period of observation. For example, there was a threefold increase in the rate of acute myocardial infarction (AMI) in Japan between 1979 and 2008. Variation is also seen in the rates of anaphylaxis in both the US and Korea with rates more than doubling in all three studies sited, and primarily among women. Such trends may be due to changes in reporting, but the presence of the same trend in the US and Korean studies for anaphylaxis over time means that the true incidence may also be increasing. Thus, caution must be used in making comparisons for some outcomes using rates from the distant past. Background rates can also vary by age and sex for many outcomes. This is quite dramatic for a neuroinflammatory disease such as multiple sclerosis where the incidence rate is 18.04 cases/100,000 p-y in women 40-49 years of age but only 0.21 cases/100,000 p-y in women 70-74 years of age -a more than 85-fold difference. For the same outcome, incidence also dramatically varies by sex with the incidence in 40-49 year olds being 2.5 time higher in females. Similarly, the rate of Bell's Palsy increases with age and in individuals over 85 years of age is 1.6 times more common in males than females. Background rates can also vary by country. This is unfortunate since rates for many outcomes are only available from selected geographic locations. This is dramatically seen for acute myocardial infarction with the rate in Japan in 2008 being 27 cases/100,000 p-y as compared to 208 cases/100,000 p-y in the US in the same year -a 7.7 fold-difference. Another issue with comparing rates is that the age strata used to report rates in the literature are often not standardized making direct comparisons of different studies problematic. This can be seen, for example, in Table two where where different age strata were used in the reporting of Bell's Palsy rates from the US and Israel. Another factor that must be considered in making comparisons of observed post-vaccination data to background rates is the "healthy vaccinee effect". Since sick people are less likely to be vaccinated, the risk of adverse events may be lower in vaccinees immediately following vaccination. 11 Changes in health care utilization over time may also impact rates. This is highly relevant in the COVID-19 era, where the rates of medical utilization (on which many rates are based) have changed dramatically for many outcomes. Elective procedures have significantly diminished in frequency given surge demands on hospitals but so also have rates of premature births and outcomes related to other infectious diseases. 12 Therefore, for background rates to be most relevant in the evaluation of COVID-19 vaccines, they either need to have been assessed in the post-February 2020 era or possible differences that might have occurred in the most recent time period need to be considered. Thus, historical background rates are very important for rapid assessment of a signal but may have inherent limitations. For a full evaluation of risk, contemporaneous reference cohorts (non-vaccinated or a comparison vaccine) need to be utilized. Several template protocols have been created by the ACCESS project for such evaluations based on electronic health care data or hospital-based data. 13 In order to reliably evaluate individual cases or clusters of cases and compare the rates of outcomes with a background rate, it is critical to classify the cases accurately. The Brighton Collaboration has developed standardized case definitions for several outcomes that have generally been considered the gold standard for vaccine safety review. More recently the SPEAC project has developed additional case definitions for AESI that might be associated with COVID-19 vaccines. These definitions are available on the Brighton website and as individual manuscripts that have been published in Vaccine . In addition, the SPEAC project has developed "companion guides" to assist investigators in case evaluation. In addition to the key components of the case definition, along with algorithms to simplify determination of level of certainty, the companion guide summarizes what is known about the outcome, risk factors, information on reported vaccine association, background rates and ICD-9/10 and MedDRA codes. Once potential cases of an adverse event have been evaluated and classified as to their level of diagnostic certainty, a rate of confirmed events can be calculated and a possible association fully evaluated. An example of this process is the "companion guide" instrument developed for anaphylaxis. evaluation . The "companion guide" includes a description of risk factors for anaphylaxis, a summary of background rates of anaphylaxis from various countries and age groups from the literature, the Brighton Collaboration Anaphylaxis Case Definition, anaphylaxis diagnostic codes in ICD9/10-CM as well as MedDRA, a prototype data abstraction for medical chart review as well as both tabular and pictorial presentations of key case definition criteria for case classification. The goal of this effort is to attempt to assist with and standardize evaluations of SARS-2-CoV-2 AESI. Background rates of events can serve as a useful evaluation tool for a rapid initial response to a vaccine safety concern. Many such rates are available on the Brighton website and from the published literature. These may be complemented by more recent rates that are being generated for the post-February 2020 era. To realize the full potential of such rates, tools are available for standardized case classification and assessment. While comparison of background rates is an important tool, it nonetheless must be realized that background rates are only a single tool and that for possible safety signals, the use of other tools such as rapid cycle analyses 14 and epidemiologic observational studies that employ case validation and classification methods need to be considered. The state of vaccine safety science: systematic reviews of the evidence The Ethics of Continuing Placebo in SARS-CoV-2 Vaccine Trials Allergic Reactions Including Anaphylaxis After Receipt of the First Dose of Pfizer-BioNTech Vaccine-United States Covid-19: Norway investigates 23 deaths in frail elderly patients after vaccination The methodology of self-controlled case series studies Background rates of Adverse Events of Special Interest for monitoring COVID-19 vaccines EUPAS37273 Frequency and impact of confounding by indication and healthy vaccinee bias in observational studies assessing influenza vaccine effectiveness: a systematic review Effect of social distancing due to the COVID-19 pandemic on the incidence of viral respiratory tract infections in children in Finland during early 2020. The Pediatric infectious disease journal Real-time vaccine safety surveillance for the early detection of adverse events Trends in acute myocardial infarction incidence and mortality over 30 years in Japan Population trends in the incidence and outcomes of acute myocardial infarction Importance of background rates of disease in assessment of vaccine safety during mass immunisation with pandemic H1N1 influenza vaccines Trends in severe pediatric emergency conditions in a National Cohort Increasing anaphylaxis hospitalizations in the first 2 decades of life True rise in anaphylaxis incidence: Epidemiologic study based on a national health insurance database Considerations and complications after Bells' palsy Incidence, clinical features and prognosis in Bells palsy Rates of autoimmune diseases in Kaiser Permanente for use in vaccine adverse event safety studies Transverse myelitis. Retrospective analysis of 33 cases, with diff erentiation of casesassociated with multiple sclerosis and parainfectious events Importance of background rates of disease in assessment of vaccine safety during mass immunisation with pandemic H1N1 influenza vaccines Guillain-Barré syndrome. Clinicoepidemiologic features and effect of influenza vaccine A 15 year nationwde epidemiological analysis of Guillain-Barré Syndrome in Taiwan Incidence Rates of Autoimmune Diseases in European Healthcare Databases: A Contribution of the ADVANCE Project Incidence of multiple sclerosis in the United Kingdom Incidence Rates of Autoimmune Diseases in European Healthcare Databases: A Contribution of the ADVANCE Project