key: cord-0307932-is1fsqk5 authors: Reinhart, A.; Brooks, L.; Jahja, M.; Rumack, A.; Tang, J.; Al Saeed, W.; Arnold, T.; Basu, A.; Bien, J.; Cabrera, A. A.; Chin, A.; Chua, E. J.; Clark, B.; DeFries, N.; Forlizzi, J.; Gratzl, S.; Green, A.; Haff, G.; Han, R.; Hu, A. J.; Hyun, S.; Joshi, A.; Kim, J.; Kuznetsov, A.; La Motte-Kerr, W.; Lee, Y. J.; Lee, K.; Lipton, Z. C.; Liu, M. X.; Mackey, L.; Mazaitis, K.; McDonald, D. J.; Narasimhan, B.; Oliveira, N. L.; Patil, P.; Perer, A.; Politsch, C. A.; Rajanala, S.; Rucker, D.; Shah, N.; Shankar, V.; Sharpnack, J.; Shemetov, D.; Simon, N.; Srivastava, V.; Tan, S.; Tibshirani, R.; Tuzhilin, title: An Open Repository of Real-Time COVID-19 Indicators date: 2021-07-16 journal: nan DOI: 10.1101/2021.07.12.21259660 sha: 49db57f300b270f16cbcb1891ca39e16981d42b5 doc_id: 307932 cord_uid: is1fsqk5 The COVID-19 pandemic presented enormous data challenges in the United States. Policy makers, epidemiological modelers, and health researchers all require up-to-date data on the pandemic and relevant public behavior, ideally at fine spatial and temporal resolution. The COVIDcast API is our attempt to fill this need: operational since April 2020, it provides open access to both traditional public health surveillance signals (cases, deaths, and hospitalizations) and many auxiliary indicators of COVID- 19 activity, such as signals extracted from de-identified medical claims data, massive online surveys, cell phone mobility data, and internet search trends. These are available at a fine geographic resolution (mostly at the county level) and are updated daily. The COVIDcast API also tracks all revisions to historical data, allowing modelers to account for the frequent revisions and backfill that are common for many public health data sources. All of the data is available in a common format through the API and accompanying R and Python software packages. This paper describes the data sources and signals, and provides examples demonstrating that the auxiliary signals in the COVIDcast API present information relevant to tracking COVID activity, augmenting traditional public health reporting and empowering research and decision-making. 2020. Alongside public data on reported cases and deaths, this database includes several unique data streams, including indicators extracted from de-identified medical claims data, antigen test results from a major testing manufacturer, large-scale public surveys that measure symptoms and public behavior, and indicators based on particular Google search queries. (We use the terms "indicator" and "signal" interchangeably.) We make aggregate signals publicly available, generally at the county level, via the COVIDcast API [19] . We store and provide access to all previous (historical) versions of the signals, a key feature that exposes the effects of data revisions. Lastly, we provide R [20] and Python [21] packages to facilitate interaction with the API, and an online dashboard to visualize the data [22] . In a companion paper, we analyze the utility provided by a core set of the indicators in COVID-19 forecasting and hotspot prediction models. In another companion paper, we elaborate on our research group's (Delphi's) large-scale public surveys, run in partnership with Facebook and available in aggregate form in the COVIDcast API. We receive data daily from healthcare partners, technology companies, and from surveys conducted daily by Delphi in partnership with Facebook. These data sources provide information not available from standard public health reporting or other common sources, such as: Table 1 : Data sources available in Delphi's COVIDcast API [19] , as of date of publication. The first group of data sources are produced by Delphi from data not otherwise available publicly (or only available in limited form); the second group is mirrored from public sources. 2020-05-26 County * * SafeGraph Mobility metrics, such as time away from home or visits to bars and restaurants, based on cell phone mobility data collected by SafeGraph [26, 27] . Google Symptoms Trends in Google search volume for terms related to anosmia and ageusia (loss of smell or taste), which correlate with COVID activity, based on data shared by Google [25] . (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted July 16, 2021. ; Because each data source reports data in different formats, we must convert each source to a common format. In this format, each record represents an observation of one quantity at one time point in one location. Locations are coded consistently using standard identifiers such as FIPS codes; the sample size and standard error for each observation is also reported when applicable. Each signal is reported at the finest geographic resolution its source supports (such as county or state) and also aggregated to metropolitan statistical areas, Health and Human Services regions, and hospital referral regions. National averages are also provided. Crucially, each record is tagged with an issue date referring to when the value was first issued, as described below. This allows tracking of revisions made to individual observations, as each revision is tagged with its own issue date. When appropriate, additional post-processing (often nontrivial) is applied to the data. For example, data on visits to doctors' offices is subject to strong day-of-week effects, and so regression is used to adjust for these effects. Other indicators are available in raw versions and versions smoothed with a 7-day trailing average. All processing is done using open-source code written primarily in Python and R, and available publicly at https: //github.com/cmu-delphi/covidcast-indicators/. Many data sources that are useful for epidemic tracking are subject to revision after their initial publication. For example, aggregated medical claims data may be initially published after several days, but additional claims and corrections may take days to weeks to be discovered, processed, and aggregated. Medical testing data are also often subject to backlogs and reporting delays, and estimates for any particular date are revised over time as errors are found or additional data becomes available. This revision process is generally referred to as backfill. For this reason, the COVIDcast API annotates every observation with two dates: the time value, the date the underlying events (such as tests or doctor's visits) occurred, and the issue date when Delphi aggregated and reported the data for that time value. Importantly, there can be multiple observations for a single time value with different issue dates, for example if data is revised or claims records arrive late. Delphi tracks revisions to all data sources we ingest, including external data sources (such as sources tracking cases and deaths). Many external sources do not keep a public or conveniently accessible record of revisions of their data. For many purposes it is sufficient to use the most recently issued observation at a given time value, and the COVIDcast API returns the most recent issue as its default. However, for some applications it is crucial to know what was known as of a specific date. For example, an epidemic forecasting model will be called upon to make its forecasts based on preliminary data about recent trends, so when it is trained using historical data, it 6 All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted July 16, 2021. ; should be trained using the initial versions of that data, not updates that would have been received later. Moreover, these revision records allow models to be modified to account for noise and bias in early data versions, or to exclude data that is too new to be considered stable, and to "rewind" time and simulate how these revised models would have performed using only the versions of data available as of those times. Research on data revisions in the context of influenza-like illness has shown that backfill can significantly alter forecast performance [7, 29] , and that careful training on preliminary data can reduce this influence [8] . Recent research has shown similar results for COVID-19 forecasts [30] . We also examine this in our companion paper on forecasting, where we observe that training and validating models on finalized data yields overly optimistic estimates of true test-time performance. The data described above is publicly available through the Delphi COVIDcast API [19] . By making HTTP requests specifying the data source, signal, geographic level, and time period desired, users can receive data in JSON or CSV form. For added convenience, we have written covidcast R [20] and Python [21] packages with functions to request data, format it as a data frame, plot and map it, and combine it with data from other sources. The R and Python package software is public and open-source, at https://github.com /cmu-delphi/covidcast/. The API server software is itself also public and open-source, at https://github.com/cmu-delphi/delphi-epidata/. Lastly, most data sources are provided under the Creative Commons Attribution license, and a small number have additional restrictions imposed by the data source; see https://cmu-delphi.github.io/del phi-epidata/api/covidcast_licensing.html. Since July 2020, Delphi has been regularly submitting short-term forecasts of COVID-19 case and death incidence, at the state and county levels, to the COVID-19 Forecast Hub [31] , with "CMU-TimeSeries" as the team-model name. The process of building, training, and deploying our forecasting models leverages much of the infrastructure described in this paper (such as the COVIDcast API's as of feature), and some of our forecasting systems rely on auxiliary indicators (such as survey-based and claims-based COVID-like illness signals, which are described below). The indicators that are available in the COVIDcast API have been used in dashboards produced by COVID Act Now [32] , COVID Exit Strategy [33] , and others; to inform the Delphi, DeepCOVID [34] , and the Institute for Health Metrics and Evaluation (IHME) [35] COVID forecasting models; in various federal and state government reports and analyses; and in a range of news stories. Aside from operational use in decision-making and forecasting, they have also facilitated numerous analyses studying the impacts of COVID-19 on the public, the effectiveness of policy interventions, and factors that influenced the spread of the pandemic [17, 18, [36] [37] [38] . The API currently serves hundreds of thousands of requests to thousands of users every day. In what follows, we present examples of the usefulness of some of the novel signals available in the API. These examples demonstrate that such indicators are meaningfully related to COVID activity, that they provide alternate views on pandemic activity that are not subject to the same reporting glitches and delays as traditional public health surveillance streams, and that they provide information about public behavior and attitudes that are not available from any other source. Code to reproduce all examples (which uses the covidcast R package and fetches data from the API) can be found at https://github.com/cmu-delphi/covidcast-pnas/tree/main/indicators/code/. Many of the indicators in the COVIDcast API are intended to track COVID activity. Five indicators in particular have the closest connections to confirmed cases: • Change Healthcare COVID-like illness (CHNG-CLI): The percentage of outpatient visits that are primarily about COVID-related symptoms, based on de-identified Change Healthcare claims data. • Change Healthcare COVID (CHNG-COVID): The percentage of outpatient visits with confirmed COVID-19, based on the same claims data. 8 All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted July 16, 2021. • COVID-19 Trends and Impact Survey CLI in the community (CTIS-CLI-in-community): The estimated percentage of the population who know someone in their local community who is sick, based on the same surveys. • Quidel test positivity rate (Quidel-TPR): The percentage of positive results among Quidel COVID antigen tests. Figure 1 compares the first three of these signals to COVID cases in the United States (from JHU CSSE, smoothed with a 7-day trailing average) over a year of the pandemic (April 15, 2020 to April 15, 2021), illustrating how they track national trends quite well. Importantly, this same relationship persists across multiple resolutions of the data, down to smaller geographic regions such as states and counties, as shown in the supplement. This will also be illustrated in a more detailed correlation analysis in the next subsection. Besides tracking contemporaneous COVID activity, these and other indicators can be used to improve forecasts of future COVID case trends, as investigated in a companion paper. To quantify the ability of the signals described above to track trends in COVID cases, we use the Spearman (rank) correlation and analyze two key correlation patterns, between each signal and confirmed COVID case rates (cases per 100,000 people): 9 All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted July 16, 2021. ; 1. Geo-wise correlations (i.e., on a specific date, do values of the signal correlate with case rates across locations?): Formally, let X t and Y t be vectors of values of a signal and case rates, over all locations, on date t. The geo-wise correlation at time t is defined as cor(X t , Y t ) (where here and throughout cor(·, ·) denotes Spearman correlation). This examines whether a signal has the capability to help spot locations with high case rates at any given time. 2. Time-wise correlations (i.e., at a specific location, do values of the signal correlate with case rates across time?): Let X ℓ and Y ℓ be vectors of values of a signal and case rates, over all times, at location ℓ. The time-wise correlation at location ℓ is defined as cor(X ℓ , Y ℓ ). This examines whether changes in a signal over time correspond to changes in reported cases at the same location. Figure 2 shows the geo-wise correlations achieved by the five signals and COVID case rates (from JHU CSSE, smoothed using a 7-day trailing average), from April 15, 2020 to April 15, 2021. This calculation is performed over all counties with at least 500 cumulative cases by the end of this period, and at which all indicators are available (956 counties in total). The large positive correlations suggest that these signals could be useful in hotspot detection (identifying counties that have relatively high COVID activity, at a given time). Somewhat surprisingly, the survey-based CLI-in-community signal shows the strongest correlations for much of the time period. This clearly demonstrates the value of a large-scale survey such as CTIS for tracking symptoms and case trends, especially when other data is unavailable. Figure 3 summarizes time-wise correlations from these five signals over the same time period, and for the same set of counties. For each signal, we display the set of correlations that it achieves in histogram form (more precisely, using a kernel density estimate). All signals produce positive correlations in the majority of counties considered (with very little mass in each estimated density being to the left of zero). The largest correlations, in bulk, are achieved by the CHNG-COVID signal; the CTIS-CLI-in-community signal is a close second, and the CHNG-CLI signal is third. There are two noteworthy points: • This is different from what is observed in Figure 2 , where the CTIS-CLI-in-community signal achieves clearly the highest correlations for most of the time period. However, it is worth emphasizing that time-wise and geo-wise correlations are truly measuring different properties of a signal; and the claims signals (CHNG-COVID and CHNG-CLI) seem more appropriate for temporal-rather than spatial-comparisons. We revisit this point in the discussion. • It is still quite impressive (and surprising) that the CTIS-CLI-in-community signal, based on people reporting on the symptoms of others around them, can achieve nearly as strong time-wise correlations to confirmed cases as can a signal that is based on picking up the occurrence of a confirmed case passing through the outpatient system. 10 All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted July 16, 2021. Public health reporting of COVID tests, cases, deaths, and hospitalizations is subject to a number of possible delays and problems. For example, COVID testing data is reported inconsistently by different states using different definitions and inclusion criteria, and differences in reporting processes mean state data often does not match data reported to the federal government [39] . Case and death data is frequently backlogged and corrected, resulting in artificial spikes and drops [40, 41] . As an example, looking back at Figure 1 , we can see clear dips in the confirmed COVID case curve that occur around the Thanksgiving and New Year's holidays. This is artificial, and due to the fact that public health departments usually close over holiday periods, which delays case and death reporting (for this reason, the artificial dips persist at the state-and county-level as well). The CLI signal from the survey, on the other hand, displays no such dips. The claims signals actually display holiday effects going in the other direction: they exhibit spikes around Thanksgiving and New Year's. This is because they measure the fraction of all outpatient visits with a certain condition, and the denominator here (total outpatient visits) drops disproportionately during holiday periods, as people are likely less willing to go to the doctor for more routine issues. Fortunately, in principle, the holiday effects in claims signals should be correctable: they are mainly due to overall changes in medical seeking behavior during holiday, periods, and we can estimate such effects using historical claims data. As a further example, Figure 4 displays data from Bexar County, Texas (which contains San Antonio) during July 2020. On July 16, 2020, San Antonio reported 4,810 backlogged 11 All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted July 16, 2021. ; https://doi.org/10.1101/2021.07.12.21259660 doi: medRxiv preprint cases after reporting problems prevented them from being reported over the past two weeks [42] , resulting in a clearly visible spike in the left-hand panel of the figure (case data from JHU CSSE, smoothed using a 7-day trailing average). Meanwhile, Delphi's COVID Trends and Impact Survey averaged around 350 responses per day in Bexar County over the same time period, and was able to estimate the fraction of the population who know someone in their local community with COVID-Like Illness (CLI). As we can see in the right-hand panel of the figure, this signal was not affected by Bexar County's reporting problems and, as shown in the last subsection, it is (in general) highly correlated with case rates, providing an alternate stream of data about COVID activity unaffected by backlogs. Similar reporting problems have occurred in many jurisdictions across the United States, making it valuable to cross-check against external sources not part of the same reporting systems. The revision tracking feature in the API assists in model-building and evaluation. Figure 5 illustrates how one COVIDcast medical claims signal evolved as it was revised across multiple issue dates, in four different states, between June 1 and August 1, 2020. In each plot, the rightmost ends of the lines correspond to estimates for the last day that data are available for each issue date, which are generally the most tentative estimates, and appear to be significantly biased upward in Arizona in June 2020, and significantly biased downward in New York throughout June and July 2020. Claims-based signals typically undergo heavy backfill as additional claims are processed 12 All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted July 16, 2021. On July 16, 4,810 backlogged cases were reported, though they actually occurred over the preceding two weeks (this shows up as a prolonged spike in the left panel due to the 7-day trailing averaging applied to the case counts). Daily CTIS estimates of CLI-in-community showed more stable underlying trends. and errors are corrected; the median relative error between initial reports and final values is over 20% for such data, and only after roughly 35 days do estimates typically match finalized values within 5%. However, the systematic nature of this backfill, as illustrated in Figure 5 , suggests that statistical models could be fit (potentially separately for each location) to estimate the final values from preliminary reports. On the other hand, official public health reporting of COVID cases and deaths can be subject to revision as death certificates are audited and backlogs cleared, resulting in thousands of cases and deaths being added or removed. This process is much more difficult to predict, and thus claims data and other sources may be a useful stand-in while public health reports are aggregated and corrected. To reiterate a previous point, when training and validating forecast models (on historical data), users will want to use data that was known as of the forecast date, not revised versions that only became available much later. The COVIDcast API makes all historical versions available and easily accessible for this purpose; and this feature plays a prominent role in our own analysis of forecasting and hotspot prediction models appearing in a companion paper. Auxiliary signals (outside of the standard public health reporting streams) can serve as indicators of COVID activity, but they can also illustrate the effect of mitigating actions 13 All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (such as shelter-in-place orders) and can guide resource allocation for fighting the pandemic. For example, medical claims data reflects healthcare-seeking behavior; measures of mobility reflect adherence to public health recommendations; and measures of COVID vaccine acceptance can guide outreach efforts. As an illustration, Figure 6 maps the sharp increase in rates of people staying at home between March 1 and April 15, 2020. Using SafeGraph data on the fraction of mobile devices included in SafeGraph's panel that did not leave the immediate area of their home, it illustrates the sharp drop in travel and work outside the home that occurred in the early stages of the pandemic. It also shows that this drop was much more pronounced in some states than others, enabling analysis of policy impacts and disease spread. Similar maps can be quickly constructed for any signal in the COVIDcast API. The COVIDcast API provides open access to real-time and geographically-detailed indicators of COVID activity in the United States, which supports and enhances standard public health reporting streams in several ways. First, several signals in the API closely track COVID activity (over both time and space); yet they are derived from different data streams (such as surveys, medical insurance claims, and medical devices), and are thus not subject to the same sources of error 14 All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. as public health reporting streams. This can be important both for robustness and situational awareness, allowing decision-makers to diagnose potential anomalies in standard surveillance streams, and for modeling tasks such as forecasting and nowcasting. Our companion paper on forecasting discusses this in more detail. Second, the API features many other signals that are relevant to understanding aspects of the pandemic and its effects on the United States population that are not found in traditional public health streams, such as data on mobility patterns, internet search trends, mask wearing, and vaccine hesitancy, to name just a few. (The latter two signals are derived from the COVID-19 Trends and Impact Survey; our companion paper on this survey gives a more detailed view of its features and capabilities.) These signals have already supported pandemic research and policy-making. Third, the underlying database tracks all revisions made to the data, allowing us to query the API to learn "what was known when," which is critical for understanding the behavior (and potential pitfalls) of real-time surveillance signals. Such revision data is rarely available in standardized format from other sources. Finally, we emphasize that unifying many relevant signals into a single common format, with comprehensive revision tracking, is an important goal in and of itself. The ability to combine public health reporting data, syndromic surveillance data, and digital measures of mobility and behaviors goes beyond providing traditional situational awareness. Convenient and real-time access to this data enables continuous telemetry summarizing how 15 All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted July 16, 2021. ; things are, how they are expected to change, which areas need additional resources to be allocated in response, and how effective public communication is. There are a number of open questions, and challenges that remain. Several signals are subject to biases, such as survey sampling and nonresponse biases, geographic differences in market share for medical claims data, or biases in the population represented in appbased mobility data. Claims data tends also to be subject to biases during major national holidays and other events that change health-seeking behavior. Characterizing these biases will be important for future research and operational systems that use these signals. Several data sources are also subject to extensive revision and backfill, which must be studied and modeled to enable effective real-time use of these sources in forecasting and nowcasting systems. The breadth and unique features of the COVIDcast API will help facilitate this and other related work, which will be vital to advancing pandemic modeling and preparedness. All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted July 16, 2021. ; https://doi.org/10.1101/2021.07.12.21259660 doi: medRxiv preprint • Insurance claims (Change Healthcare and others) Figure 7 : The epidemiological "severity pyramid" represents the progression of cases from the public, through infection, through increasingly severe stages of disease. The annotations here represent the data sources collected by Delphi's COVIDcast Epidata API. All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted July 16, 2021. All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted July 16, 2021. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted July 16, 2021. 26 All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted July 16, 2021. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. 29 All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. 30 All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted July 16, 2021. Here, the HHS data has been limited to values from 2020-08-01 onward due to changes in reporting behavior, and processed into a 7-day trailing average rate per 100,000 resident population using Census Bureau estimates for 2019. HSP-Hosp is the percentage of new hospital admissions with COVID-associated diagnoses, based on claims data from health system partners, smoothed in time and adjusted for systematic day-of-week effects (HSP-Hosp is available from an earlier date when working at finer geographical resolutions). HSP-Hosp has been scaled to have the same maximum value as the HHS data. All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. Figure 17 : Geo-wise correlations with hospitalization rates derived from HHS data, from April 15, 2020 to April 15, 2021, calculated for all times with sufficient available data within this period, over all state-like jurisdictions for which each signal was reported on at least 50 days during this period, limited to state-day combinations for which both signals are available. All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted July 16, 2021. Figure 18 : Time-wise correlations with hospitalization rates derived from HHS data, from April 15, 2020 to April 15, 2021, calculated over all state-like jurisdictions for which each signal was reported on at least 50 days during this period, limited to state-day combinations for which both signals are available. All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted July 16, 2021. ; https://doi.org/10.1101/2021.07.12.21259660 doi: medRxiv preprint An interactive web-based dashboard to track COVID-19 in real time Economic and social consequences of human mobility restrictions under COVID-19 Association between COVID-19 outcomes and mask mandates, adherence, and attitudes It's complicated: characterizing the time-varying relationship between cell phone mobility and COVID-19 spread in the US covidcast: R Client for Delphi's COVIDcast Epidata API COVIDcast Python API client Partnering with a global platform to inform research and public policy making COVID-19 search trends symptoms dataset Social distancing metrics Provisional death counts for coronavirus disease A collaborative multiyear, multimodel assessment of seasonal influenza forecasting in the United States Back2Future: Leveraging backfill dynamics for improving real-time predictions in future The COVID-19 Forecast Hub COVID risk & vaccine tracker Tracking our COVID-19 response DeepCOVID: An operational deep learning-driven framework for explainable real-time COVID-19 forecasting The impact of online misinformation on U.S. COVID-19 vaccinations The Affordable Care Act and the COVID-19 pandemic: A regression discontinuity analysis Locked (down) and loaded (language): Effect of policy and speech on COVID-19 outcomes Federal testing data's last mile Inconsistent reporting practices hampered our ability to analyze COVID-19 data. Here are three common problems we identified What the coronavirus disease 2019 (COVID-19) pandemic has reinforced: The need for accurate data It's frustrating': Bexar County adds 5,000 COVID-19 cases from backlog as Texas disagrees on data