key: cord-0273799-bbg6l8m5 authors: Salomon, J. A.; Reinhart, A.; Bilinski, A.; Chua, E. J.; La Motte-Kerr, W.; Rönn, M.; Reitsma, M. B.; Morris, K. A.; LaRocca, S.; Farag, T.; Kreuter, F.; Rosenfeld, R.; Tibshirani, R. title: The U.S. COVID-19 Trends and Impact Survey, 2020-2021: Continuous real-time measurement of COVID-19 symptoms, risks, protective behaviors, testing and vaccination date: 2021-07-26 journal: nan DOI: 10.1101/2021.07.24.21261076 sha: 188fee0ad4f3a93b7bc741451cd6d3c80b9c688f doc_id: 273799 cord_uid: bbg6l8m5 The U.S. COVID-19 Trends and Impact Survey (CTIS) is a large, cross-sectional, Internet-based survey that has operated continuously since April 6, 2020. By inviting a random sample of Facebook active users each day, CTIS collects information about COVID-19 symptoms, risks, mitigating behaviors, mental health, testing, vaccination, and other key priorities. The large scale of the survey -- over 20 million responses in its first year of operation -- allows tracking of trends over short timescales and allows comparisons at fine demographic and geographic detail. The survey has been repeatedly revised to respond to emerging public health priorities. In this paper, we describe the survey methods and content and give examples of CTIS results that illuminate key patterns and trends and help answer high-priority policy questions relevant to the COVID-19 epidemic and response. These results demonstrate how large online surveys can provide continuous, real-time indicators of important outcomes that are not subject to public health reporting delays and backlogs. The CTIS offers high value as a supplement to official reporting data by supplying essential information about behaviors, attitudes toward policy and preventive measures, economic impacts, and other topics not reported in public health surveillance systems. During 2020, the coronavirus disease 2019 (COVID-19) pandemic precipitated the need for new public health surveillance to inform urgent policy decisions. Effective pandemic policy making requires information on a broad array of indicators including local morbidity and mortality, preventive behaviors, healthcare capacity, and economic impacts. Given the critical importance of COVID-19 trends for policy, health departments set up routine public reporting systems for tracking cases, deaths, testing, and hospitalizations (1) . However, supplemental data can both augment official reporting, for example by providing additional indicators of COVID-19 prevalence not subject to reporting delays and backlogs, and supply complementary information about public behavior, attitudes toward policy and preventive measures, mental health, economic impacts, and other items not observed in public health surveillance systems. A number of efforts have used surveys to provide supplemental surveillance data. For example, symptom-tracking smartphone apps invite users to self-report symptoms, in some cases encouraging repeated participation to enable longitudinal tracking (2) (3) (4) . Other surveys have addressed broader impacts of the pandemic, such as economic consequences (5) . In this paper, we present findings from the Delphi Group at Carnegie Mellon University U.S. COVID-19 Trends and Impact Survey (US CTIS), in partnership with Facebook, which has operated continuously since April 6, 2020 and collected over 20 million responses. (An international version of the survey is described in a companion paper in this theme issue.) A random sample of Facebook active users are invited each day to complete a questionnaire comprising survey items on symptoms, COVID testing, social distancing, vaccination, schooling, mental health, and economic security. Results are aggregated and made publicly available, and microdata are available under institutional data use agreement, in both cases with less than three days of lag. These data provide information at a level of geographic and temporal detail that can supply essential inputs into short-term decision-making and longer-term strategic planning. The survey instrument has been updated frequently to incorporate new policy-relevant topics. The largest public health survey ever conducted in the . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted July 26, 2021. ; https://doi.org/10.1101/2021.07.24.21261076 doi: medRxiv preprint United States, CTIS is designed to facilitate detailed demographic and geographic analyses, to track trends over time, and to accommodate rapid response to emerging priorities (6) . In this study, we first compare COVID-19 indicators from CTIS with publicly-reported case, hospitalization and mortality data between April 2020 and April 2021. Despite potential limitations of our Internet-based sample and the voluntary nature of the survey, we demonstrate high correspondence between the two, with CTIS less affected by holiday-related reporting anomalies. Second, we examine patterns and trends in symptoms, risks, mitigating behaviors, testing and vaccination in US states and localities in relation to evolving high-priority policy questions over 12 months of the pandemic. The findings illustrate the value of online surveys for tracking patterns and trends in COVID-related outcomes as an adjunct to official reporting, while also showcasing insights that are only possible through a large-scale survey effort. The U.S. COVID-19 Trends and Impact Survey launched on April 6, 2020 and has run continuously since that time, with an average of more than 350,000 people participating each week over the first year of operation. The survey is implemented by the Delphi Group at Carnegie Mellon University (CMU), with participants recruited via the Facebook platform. Every day, Facebook invites a new sample of active users ages 18 years or older to participate in the survey. Facebook uses stratified random sampling within US states to randomly select a sample of its users to see the survey invitation at the top of their News Feed. Users who click on the invitation are taken to the CMU-administered survey hosted on Qualtrics. To ensure privacy, Facebook does not see any individual survey response during or after the data collection. The survey is . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted July 26, 2021. The survey instrument was deployed in 10 waves from launch through April 5, 2021, with contents of each survey version summarized in Table 1 Full versions of all survey instruments can be found at https://cmudelphi.github.io/delphi-epidata/symptom-survey/coding.html. Analytic weights have been developed to adjust for differences between Facebook users and the United States population, and to adjust for biases related to coverage and non-response (7) . When Facebook links users to the survey, it generates a random unique identifier that is passed to CMU. For users who complete the survey, CMU returns the corresponding identifiers to Facebook, which then calculates analytic weights in two steps: 1. To adjust for non-response bias, Facebook calculates the inverse probability that sampled users complete the survey using their age, gender, and geographical variables, as reported on their Facebook profiles, as well as other characteristics known to correlate with non-response. The inverse probabilities . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted July 26, 2021. ; https://doi.org/10.1101/2021.07.24.21261076 doi: medRxiv preprint are then used to create weights for responses, after which the survey sample reflects the active adult user population on Facebook. 2. To adjust for coverage bias, Facebook post-stratifies the weights created in the first step so that the distribution of age, gender and state or territory of residence in the survey sample reflects that of the general population. The weight value does not identify the survey respondent. The weight value for an individual is scaled to approximate the number of people in the adult population represented by that individual based on age, gender, location, and date. Facebook passes these weights to CMU. CMU cannot use these weights to identify specific Facebook users, and Facebook never receives individual survey responses and cannot link them to specific users. In this study we examined a range of different outcomes measured in the CTIS over the period April 6, 2020 to April 5, 2021. Characteristics of the study sample were compared to data from the American Community Survey 2019 supplemental estimates. We evaluated reported symptoms and symptom patterns in comparison to surveillance data on confirmed COVID-19 hospitalizations from the Department of Health and Human Services (8) and reported COVID-19 cases and mortality aggregated by the Johns Hopkins University Center for Systems Science and Engineering (9). We defined a condition called COVID-like illness or CLI, which comprised reporting a fever of at least 100° F, along with cough, shortness of breath, or difficulty breathing, in line with a working definition of CLI used for syndromic surveillance purposes beginning in early 2020. A second indicator was constructed based on responses to an item on the survey that asks whether respondents know someone personally in their community who is ill with COVID-like symptoms, which we call CLI-in-Community. Some results were stratified by county groupings defined in references to the CDC Social Vulnerability Index (10) , which is a composite measure constructed based on 15 . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted July 26, 2021. De-identified individual participant data are available to academic and nonprofit researchers under a Data Use Agreement that protects the confidentiality of respondents. County-and state-level aggregates of key variables are publicly available in the COVIDcast API, described in detail in a companion paper, and are presented in an interactive online dashboard (https://delphi.cmu.edu/covidcast/survey-results). Demographic breakdowns of key variables over time are available for public download at https://cmu-delphi.github.io/delphi-epidata/symptom-survey/contingencytables.html. The study was approved by the Carnegie Mellon University Institutional Review Board, under protocol STUDY2020_00000162. As of April 5, 2021, a total of 20. 2 States COVID Trends and Impact Survey. Table 2 summarizes characteristics of the survey respondents. Compared to the weighted sample, the unweighted sample had a higher proportion of women (66% vs. 52%) and a slightly higher proportion of respondents between ages 25 and 64 years (72% vs 68%). Household size and prevalence of at least one comorbidity were similar in the unweighted and weighted samples. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted July 26, 2021. ; https://doi.org/10.1101/2021.07.24.21261076 doi: medRxiv preprint Compared to 2019 American Community Survey supplemental estimates, the weighted survey sample slightly overrepresented women, but had a broadly comparable age and geographic distribution. The weighted sample included a larger proportion of respondents with greater than a high school education, and a much smaller proportion with less than a high school education, suggesting the presence of a sampling or response bias correlated with education. This bias was consistent since education was added to the survey instrument. As the weights provided by Facebook do not account for education, the weighting did not correct this bias. First, the survey-based indicators were less susceptible to daily fluctuations and reporting anomalies that appeared in cases and deaths, including abrupt discontinuities around certain holiday periods. Second, trends and patterns in anosmia were similar to patterns in CLI, and the anosmia series provided a closer match than the other two survey indicators to the trends and patterns observed in COVID-19 hospitalizations, mirroring temporal peaks and ordering of levels across regions over . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted July 26, 2021. ; https://doi.org/10.1101/2021.07.24.21261076 doi: medRxiv preprint different waves of the epidemic. Third, CLI-in-community, with the highest levels of prevalence, provided the most temporally stable signals while also expressing broad differences over time and space that were generally similar to those in other indicators. In a companion paper, we performed extensive correlation analyses between reported cases and various auxiliary indicators including the survey-based CLI and CLI-incommunity signals, showing strong correlations between cases and these two survey signals during much of the pandemic. The CTIS includes questions about testing and diagnosis, which since September 8, 2020 have been asked of all respondents. Figure 3 compares weekly CTIS estimates of the proportion of adults reporting that they have ever had a positive test for COVID-19 against cumulative diagnoses from surveillance reports by state in the same week. State surveillance reports were adjusted, using American Community Survey 5-year population estimates and CDC line-level demographic data on confirmed COVID-19 cases, to produce estimated diagnosis rates among the state's population over age 18. As of April 5, 2021, reported diagnoses in the survey ranged from 3.1% in Hawaii to 19% in Idaho, and the correlation between survey reported diagnoses and surveillance reports at state level was 0.83, indicating strong convergent validity. Since Wave 4, the CTIS has included questions about occupation, which can offer valuable insights into exposures among essential workers and also supply signals of where transmission may be concentrated. Using responses from January 2021, we examined the probability of reporting a positive COVID-19 test across different occupation categories, as well as the probability of reporting working outside the home while having symptoms consistent with COVID-19 (Table 3) . Substantial heterogeneity appeared across broad groups of occupation. The large proportion of people reporting never having been tested indicates the limitations of passive surveillance. Combining questions on symptoms, testing and working into a single indicator, we examined the fraction of people who reported both working outside the home and currently having . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted July 26, 2021. atypical symptoms; results ranged from more than 15% for respondents in food preparation and serving related occupations, to 4% of those in arts, design, entertainment, sports, and media. Individual-level data allow for geographically-detailed analysis that can also be disaggregated by demographic features. In Figure 4 , results are stratified in three different ways to illustrate this: by quartiles of the CDC Social Vulnerability Index (SVI), by age and by US Census region. There were minimal differences across counties grouped by SVI in reported mask use, moderately higher contacts among those living in more vulnerable communities, and substantially higher use of public transit in more vulnerable counties. The second row shows age differences, which indicate a pronounced gradient of higher risk mitigation among older respondents, especially with respect to reduced contacts. The third row describes regional patterns that vary across indicators, with higher contacts and lower mask use in the South and Midwest regions compared to the Northeast and West, but greater use of public transit in the Northeast and West compared to South and Midwest. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted July 26, 2021. ; https://doi.org/10.1101/2021.07.24.21261076 doi: medRxiv preprint Since December 19, 2020, the CTIS has included questions on vaccination intent, and since January 6, 2021, the survey has asked about vaccination status. The combination of geographic and demographic resolution in the survey allows a uniquely detailed view on vaccination acceptance and hesitancy across different US population groups. Figure 5A displays results by age group, race/ethnicity, gender, and Census region, pointing to high levels of acceptance among older respondents in all categories, but lower and more variable results at younger ages. (Respondents may identify as non-binary or self-describe their gender, but this group was typically too small to break out and report reliable hesitancy estimates by region.) Figure 5B shows the percentage of respondents indicating that they would probably not or definitely not get vaccinated across US counties, indicating regional patterns but also high variability across counties within a given state. As the vaccination campaign slows across the country, high resolution information on vaccine acceptance can inform policies that aim to increase uptake toward the goal of high levels of population immunity against COVID-19. As SARS-CoV-2 spread throughout the United States during 2020 and into 2021, and policy makers faced decisions that would profoundly impact all sectors of society, the breadth and depth of information needed to support these decisions vastly exceeded the availability of data collected through existing surveillance systems designed to capture reported COVID-19 cases, hospitalizations and deaths. A number of efforts to fill the urgent need for additional information relied on novel data collection and dissemination platforms that leveraged mobile phone technology and new media. In this study, we describe one of these efforts, the COVID Trends and Impact Survey, which is the largest continuous health survey ever conducted in the United States, in operation since April 6, 2020, with more than 20 million responses collected over the first year of operation. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted July 26, 2021. Facebook's non-response weights would be much more difficult to correct. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted July 26, 2021. ; https://doi.org/10.1101/2021.07.24.21261076 doi: medRxiv preprint Additionally, many of the outcome measures related to COVID-19 are based on selfreports, which may diverge from more objective measures due to recall bias, social desirability bias, and other sources of survey bias and measurement error. On the other hand, broad comparisons of indicators such as cumulative COVID-19 diagnoses suggest that measurement of key COVID-19 outcomes are relatively robust to response biases that may be present in the sample. Ultimately, the value of such a large-scale survey is not in accuracy afforded by its sample size, since survey biases persist no matter the size of the survey; smaller surveys more carefully constructed to reduce sampling biases would likely yield more accurate estimates (17). Instead, since these survey biases are unlikely to change rapidly in time or in space, CTIS can accurately track trends in key signals, even if the daily point estimates are systematically biased. This is demonstrated by the strong correlations between survey estimates of CLI-in-community and reported COVID case rates, for example; while CLIin-community is not an unbiased population estimate of COVID case rates, it nonetheless provides useful information about trends in cases. The principal value of CTIS is hence in the detailed spatial and demographic comparisons it makes possible, and in its ability to track changes continuously over time and correlate them with key outcome measures. Although CTIS was initially designed with a relatively limited scope, including a particular focus on syndromic surveillance, its value has ultimately derived in large part from its flexibility as a surveillance platform that can be rapidly adapted to changing information needs. Running a survey of this size has involved many challenges, particularly as it expanded to include key measures of public knowledge, attitudes, and behaviors, and as public health needs evolved continuously during the pandemic. Despite these challenges, however, CTIS has provided both a valuable public information resource during a global health emergency, as well as a potential model for ongoing health surveillance needs. Similar online surveys are likely to play important roles in future epidemics and pandemics by supplementing public reporting systems with information that is difficult to gather any other way. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted July 26, 2021. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted July 26, 2021. ; https://doi.org/10.1101/2021.07.24.21261076 doi: medRxiv preprint . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted July 26, 2021. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted July 26, 2021. ; https://doi.org/10.1101/2021.07.24.21261076 doi: medRxiv preprint Figure 1 . Frequency of reported new or unusual symptoms, pooled over respondents to the COVID Trends and Impact Survey, September 8, 2020 to April 5, 2021. Respondents are grouped by whether they indicated they tested positive in the past 14 days. Dots indicate the ratio of frequency among those who tested positive compared to all others. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted July 26, 2021. ; https://doi.org/10.1101/2021.07.24.21261076 doi: medRxiv preprint Figure 2 . Trends in anosmia, COVID-like illness (CLI), CLI-in-community, confirmed cases, hospitalizations and deaths, by Census region, April 6, 2020 to April 5, 2021. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted July 26, 2021. ; https://doi.org/10.1101/2021.07.24.21261076 doi: medRxiv preprint . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted July 26, 2021. ; https://doi.org/10.1101/2021.07.24.21261076 doi: medRxiv preprint . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted July 26, 2021. ; https://doi.org/10.1101/2021.07.24.21261076 doi: medRxiv preprint . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted July 26, 2021. ; https://doi.org/10.1101/2021.07.24.21261076 doi: medRxiv preprint Population-scale longitudinal mapping of COVID-19 symptoms, behaviour and testing Putting the Public Back in Public Health -Surveying Symptoms of Covid-19 Realtime tracking of self-reported symptoms to predict potential COVID-19 United States Census Bureau. Household Pulse Survey (COVID-19) Partnering with a global platform to inform research and public policy making Weights and Methodology Brief for the COVID-19 Symptom United States Department of Health and Human Services. COVID-19 Reported Patient Impact and Hospital Capacity by State Timeseries An interactive web-based dashboard to track COVID-19 in real time Agency for Toxic Substances and Disease Registry. CDC/ATSDR social vulnerability index 2021 The Impact of State Mask-Wearing Requirements on the Growth of COVID-19 Cases in the United States Mask-wearing and control of SARS-CoV-2 transmission in the USA: a cross-sectional study Better Late Than Never: Trends in COVID-19 Infection Rates, Risk Perceptions, and Behavioral Responses in the USA The relative incidence of COVID-19 in healthcare workers versus non-healthcare workers: evidence from a web-based survey of Facebook users in the United States Household COVID-19 risk and in-person schooling An Operational Deep Learning-driven Framework for Explainable Real-time COVID-19