key: cord-312027-5tntdjp9
authors: Charlton, Carmen L.; Kanji, Jamil N.; Johal, Kam; Bailey, Ashley; Plitt, Sabrina S.; MacDonald, Clayton; Kunst, Andrea; Buss, Emily; Burnes, Laura E.; Fonseca, Kevin; Berenger, Byron M.; Schnabl, Kareena; Hu, Jia; Stokes, William; Zelyas, Nathan; Tipples, Graham
title: Evaluation of Six Commercial Mid- to High-Volume Antibody and Six Point-of-Care Lateral Flow Assays for Detection of SARS-CoV-2 Antibodies
date: 2020-09-22
journal: J Clin Microbiol
DOI: 10.1128/jcm.01361-20
sha: 
doc_id: 312027
cord_uid: 5tntdjp9

Coronavirus disease (COVID) serological tests are essential to determine the overall seroprevalence of a population and to facilitate exposure estimates within that population. We performed a head-to-head assessment of enzyme immunoassays (EIAs) and point-of-care lateral flow assays (POCTs) to detect severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) antibodies. Demographics, symptoms, comorbidities, treatment, and mortality of patients whose sera were used were also reviewed. Six EIAs (Abbott, Affinity, Bio-Rad, DiaSorin, Euroimmun, and Roche) and six POCTs (BTNX, Biolidics, Deep Blue, Genrui, Getein BioTech, and Innovita) were evaluated for the detection of SARS-CoV-2 antibodies in known COVID-19-infected individuals. Sensitivity of EIAs ranged from 50 to 100%, with only four assays having overall sensitivities of >95% after 21 days after symptom onset. Notably, cross-reactivity with other respiratory viruses (parainfluenza virus [PIV-4] [n = 5], human metapneumovirus [hMPV] [n = 3], rhinovirus/enterovirus [n = 1], CoV-229E [n = 2], CoV-NL63 [n = 2], and CoV-OC43 [n = 2]) was observed; however, overall specificity of EIAs was good (92 to 100%; all but one assay had specificity above 95%). POCTs were 0 to 100% sensitive >21 days after onset, with specificity ranging from 96 to 100%. However, many POCTs had faint banding and were often difficult to interpret. Serology assays can detect SARS-CoV-2 antibodies as early as 10 days after symptom onset. Serology assays vary in their sensitivity based on the marker (IgA/IgM versus IgG versus total) and by manufacturer; however, overall only 4 EIAs and 4 POCTs had sensitivities of >95% >21 days after symptom onset. Cross-reactivity with other seasonal coronaviruses is of concern. Serology assays should not be used for the diagnosis of acute infection but rather in carefully designed serosurveys to facilitate understanding of seroprevalence in a population and to identify previous exposure to SARS-CoV-2.

S evere acute respiratory syndrome coronavirus 2 (SARS-CoV-2) was first detected as an unknown cause of pneumonia in December 2019 (1) . By 8 January 2020, the Chinese Center for Disease Control and Prevention officially announced a novel coronavirus to be the cause of the outbreak seen in Hubei province (2) . The disease spread quickly; at the time of writing (1 July 2020), there were 10,357,662 confirmed cases and 508,055 deaths associated with SARS-CoV-2 globally, which has affected nearly every country in the world (3) .

The ability to rapidly diagnose disease, isolate infected patients, and employ contact tracing strategies to mitigate spread of the virus is vital to slowing the spread of infection. Public health laboratories and acute diagnostic laboratories globally have rapidly developed and implemented diagnostic tests to identify COVID-19 disease. In the acute phases of illness, molecular detection of the virus is the primary tool for early and accurate diagnosis of disease (4, 5) , as antibody production is usually delayed or absent in the acute phase. Serological assays are now being developed as an epidemiological tool for population-based serosurveys and identification of remote infection (6) . However, the full extent of SARS-CoV-2 infection in large populations has yet to be determined due to limited testing (5, 7) and the presence of asymptomatic infection. Therefore, serosurveys must be well designed to best represent the population of interest. To this end, accurate and high-throughput serology assays that can be integrated into laboratory information systems are key to facilitating these large-scale studies and improving the understanding of the true proportion of the population that has recovered from COVID-19.

The primary objective of this study was to conduct a direct comparison of six highto mid-volume commercial enzyme immunoassays (IgG only or IgG with IgM or IgA or total antibody) and six lateral flow point-of-care assays (IgG and IgM) for detection of SARS-CoV-2 antibodies. All assays were tested against the same panel of serology samples from patients with confirmed COVID-19 and a group of negative controls. High-to mid-volume enzyme immunoassays were also evaluated with a separate panel of convalescent-phase sera to evaluate cross-reactivity to common respiratory viruses and non-SARS-CoV-2 coronaviruses. These performance data, coupled with clinical data from SARS-CoV-2-positive patients, make our evaluation panel particularly robust and significantly add to the current understanding of serology assays for SARS-CoV-2.

Assay evaluation. We evaluated six commercial high-to mid-volume kits: two chemiluminescence immunoassays (a chemiluminescent microparticle immunoassay [CMIA]/chemiluminescence immunoassay [CLIA] [SARS-CoV-2 IgG assay; Abbott Laboratories, Abbott Park, IL, USA] and SARS-CoV-2 S1/S2 IgG [DiaSorin, Stillwater, MN, USA]), three enzyme-linked immunosorbent assays (ELISAs; the EDI novel coronavirus COVID-19 IgM and IgG ELISA [Epitope Diagnostics Inc., supplied by Affinity Diagnostics Corp., Toronto, ON, Canada], a novel coronavirus COVID-19 IgM and IgG assay [DRG International Inc., supplied by Bio-Rad, Hercules, CA, USA], and an anti-SARS-CoV-2 ELISA IgA and IgG assay (Euroimmun, Mississauga, ON, Canada]), and one electrochemiluminescence immunoassay (ECLIA; anti-SARS-CoV-2 [Roche Diagnostics, Indianapolis, IN, USA]) for detection of SARS-CoV-2 antibodies. All assays were assessed for detection of SARS-CoV-2 IgG antibodies. Additionally, the Affinity and Bio-Rad tests were assessed for detection of IgM, and the Euroimmun test was assessed for detection of IgA antibodies. Testing was performed as per manufacturer specifications, and cutoffs were determined as described in the package inserts. All values greater than the published cutoff were considered positive. Importantly, all kits were assessed using the same patient samples from single-use aliquots, which negated analyte degradation due to increased freeze-thaw cycles and allowed direct comparison among a large number of commercially available serology assays. For the purposes of this study, CLIA, ECLIA, and ELISAs are referred to as enzyme immunoassays (EIAs). Where specified, targets of EIAs are listed in Table 1 Precision and reproducibility studies. Patient sera from 4 patients positive for SARS-CoV-2 by reverse transcriptase real-time PCR (rRT-PCR) were pooled and used as the positive control, while patient sera from 4 patients with sera collected prior to 1 November 2019 (from otherwise healthy individuals with specimens sent for immunity screening) were pooled to create the negative control. Reproducibility was assessed by running replicates of 3 in triplicate for all EIAs. Precision was determined with quality control (QC) material provided by the manufacturer (positive and negative controls), which was run singly each day of testing (Abbott and DiaSorin), or in triplicate for each day of testing (Affinity, Bio-Rad, and Euroimmun). As no QC material was provided for the Roche assay, the pooled positive and pooled negative controls were used as positive and negative controls.

Sample collection. Negative samples were retrieved from bio-banked sera stored at the public health laboratory (Alberta Precision Laboratories) in Alberta collected before 1 November 2019. To develop a panel of positive sera from patients with COVID-19, serum samples were collected from hospitalized patients confirmed to be positive for SARS-CoV-2 upon nasopharyngeal swab or endotracheal aspirate testing by rRT-PCR. Samples were collected, spun down (3,000 rpm for 10 min), aliquoted into single-use aliquots, and frozen at Ϫ80°C until the time of testing.

Eleven COVID-19 positive patients had serum collected at multiple time periods; however, only one sample per patient was used per time interval to calculate assay sensitivity. When more than one serum sample from the same individual was within a given time interval, only the most recently collected serum sample was included.

To evaluate cross-reactivity of the EIA serology tests with other respiratory viruses, convalescentphase sera (either retrieved from stored sera or prospectively collected) were used (note that the cross-reactivity panel was not assessed on the POCTs). The sera were from patients who had tested negative for COVID-19 by in-house rRT-PCR but positive for other viruses as follows (with the number of sera used): influenza A virus (n ϭ 5), influenza B virus (n ϭ 5), respiratory syncytial virus (RSVA, n ϭ 6; RSVB, n ϭ 1), rhinovirus/enterovirus (n ϭ 6), human metapneumovirus (hMPV; n ϭ 5), parainfluenza virus (PIV-1 and PIV-4; n ϭ 4), CoV-229E (n ϭ 6), CoV-NL63 (n ϭ 11), CoV-OC43 (n ϭ 7), or CoV-HKU1 (n ϭ 7). One patient was positive for multiple viruses (RSVA and enterovirus/rhinovirus). All non-COVID-19 respiratory virus testing was done using the Luminex respiratory pathogen panel (RPP; NxTAG respiratory pathogen panel; Luminex, Austin, TX, USA).

Chart review. To obtain baseline demographic variables and outcomes for the patients with confirmed SARS-CoV-2 infection, a retrospective chart review of each patient's electronic medical record was performed by two study team members using the provincial electronic medical record (Epic Systems Corporation, Verona, WI, USA). All symptoms listed were at presentation to hospital, and death was attributed to COVID-19 if it occurred within 30 days of symptom onset. Basic summary statistics (proportions, median, and range) were calculated using Microsoft Excel. Determining the date of symptom onset. The date of symptom onset for each case of laboratoryconfirmed COVID-19 was determined via history-taking using a standardized history intake form by a member of the Alberta Health Services Communicable Diseases Team (Public Health). All serum samples were stratified by this date to determine the number of days between collection of serum and time of symptom onset.

Ethics. This study received ethics approval from both the University of Calgary and University of Alberta Health Research Ethics Boards. Certification approval numbers are REB20-0516 and Pro00099818, respectively.

Population demographics. Serum from 28 patients who tested positive for SARS-CoV-2 by rRT-PCR were used in this study. The mean age of patients was 70.1 (range, 34 to 102 years), with a majority being male (57%; Table 2 ). Seven percent of patients were ambulatory, while most were hospitalized (93%), and 35% were admitted to the intensive care unit (ICU). Of those who were hospitalized, 27% required mechanical ventilation and 92% developed COVID-19-associated pneumonia. The most common comorbidities in the cohort were hypertension (64%), dyslipidemia (57%), and hypothyroidism (36%). All dates of symptom onset were reported earlier than the date of diagnostic sample collection (mean, 16 days [range, 2 to 48 days]). The time from the date of symptom onset to the date of hospitalization ranged from 0 to 19 days, with a mean of 5 days after symptom onset. Recent travel was reported in 14% of all cases, with the United States being the most frequent location of travel (Table 2) .

Performance characteristics of EIAs. In total, 46 samples from 28 different patients testing positive for SARS-CoV-2 by rRT-PCR and 50 negative samples from serum samples stored prior to 1 November 2019 were run on each assay. Overall, the positivity rate for each assay increased over time (Table 3) . With the exception of the Affinity assay (100% detection), all assays performed poorly prior to 7 days after symptom onset (range, 40 to 60%; data not shown). However, this improved over time, and all assays had at least 80% sensitivity (range, 80 to 100%) after 21 days ( Table 3 ). The earliest time to detection was seen when assays had a combination of IgG with either IgM or IgA. For example, Affinity IgM and IgG assays had sensitivities of 76% and 62%, respectively, when considered individually at 0 to 14 days after symptom onset; however, when results were combined (with either IgM or IgG being positive in a sample), the overall sensitivity increased to 90% (Table 3) . Notably, sera collected 46 and 48 days after symptom onset still had IgM detectable with both Bio-Rad and Affinity Assays and IgA detectable with the Euroimmun assay.

Only the Affinity IgM assay was able to detect antibodies with Ͼ95% sensitivity before 21 days after symptom onset. After 21 days after symptom onset, four assays (Abbott, Affinity, Bio-Rad, and Euroimmun) achieved Ͼ95% sensitivity; all four had 100% overall sensitivity (Table 3 ). However, due to the relatively small sample size, confidence intervals show a substantial overlap between time periods. All assays, with the exception of the Euroimmun IgA assay had specificities of Ͼ95% for samples collected from patients before 1 November 2019 (Table 3) .

Precision and reproducibility studies. Reproducibility for the EIAs was excellent; all assays showed 100% concordance for all samples. Likewise, precision of the assays was high, with all assays having 100% qualitative agreement for positive and negative controls.

Performance characteristics of POCTs. The same validation panel used for the EIAs was used for all POCTs, with the exception of one specimen (collected on day 13 after symptom onset), as the volume of serum was exhausted following evaluation of the EIAs. As with the EIAs, performance of the POCTs was poor Ͻ7 days after onset, with sensitivities ranging from 40 to 60% (data not shown). The performance of the assays increased over time, and all POCTs had Ͼ75% sensitivity after 14 days after onset and Ͼ90% sensitivity after 21 days (Table 4 ). However, only 4 assays had overall sensitivities that would be acceptable for use in a clinical laboratory (Ͼ95%) after 21 days after (Table 4 ). Of note, IgM was poorly detected by a number of POCTs; Getein detected IgM in only 1 of 42 positive specimens, while Biolidics detected IgM in 10 to 12% of positive samples. The highest sensitivities for IgM detection among the POCTs were consistently reported for BTNX, Deep Blue, and Genrui; these kits had statistically significantly higher overall (i.e., all-time-point calculation) IgM sensitivities than Getein (P Ͻ 0.001 for all), Biolidics (P Յ 0.004 for all), and Innovita (P Յ 0.009 for all) ( Table 4) . Reading the lateral flow assays was often challenging; many kits frequently produced only very faint lines. Because package inserts did not indicate the appropriate density of bands to call a positive result, for this study, any visualization of a band for either IgM or IgG was considered positive. This approach may have overcalled the sensitivity of the lateral flow assays. Equivocal bands were observed in 9% (3 of 32) of positive samples for Biolidics, 9% (3 of 32) for BTNX, 15% (5 of 33) for Deep Blue, 0% (0 of 33) for Genrui, 4% (1 of 26) for Getein GP BioTech, and 37% (11 of 29) for Innovita.

Cross-reactivity studies with EIAs. An additional 62 serum samples were used to assess cross-reactivity with other respiratory viruses on EIAs. Of those, 15 sera were collected prior to the first case of SARS-CoV-2 diagnosis in Alberta, and 47 were collected after the first case of SARS-CoV-2 was detected in Alberta. All samples collected after the first case were confirmed to be from patients who tested negative for SARS-CoV-2 by rRT-PCR on nasopharyngeal swab testing. The time from an RPP-positive result to serum collection ranged from 11 to 135 days (mean 45 days) from the date of the original RPP result.

Overall, all assays performed well, with only a few samples showing cross-reactivity. Most notably, the Bio-Rad IgM assay showed cross-reactivity with hMPV for a serum sample collected 31 days after the RPP and with PIV-4 for a serum sample from 120 days post-RPP, while the Bio-Rad IgG assay showed cross-reactivity with rhinovirus/entero- Affinity IgM assay showed cross-reactivity to PIV-4 (sample collected 120 days post-RPP). Abbott, Roche, and the Affinity IgG assay did not show any cross-reactivity to other respiratory viruses (Table 5) . Overall, PIV-4 and CoV-229E were most cross-reactive across assays, with additional cross-reactivity noted for CoV-NL63 and CoV-OC43 in one assay (Table 5) .

To evaluate the progression of antibody development, 11 patients in our study had serial serum samples collected (Table 6 ). Samples ranged from 5 to 29 days after symptom onset, and patients had between 2 and 6 samples collected over time. Overall, Abbott detected 63.6% (7 of 11 patients) of the earliest sample drawn from a patient, Affinity detected 100%, Bio-Rad detected 72.7%, Euroimmun detected 45.5%, DiaSorin detected 36.4%, and Roche detected 63.6% (Table 6 ). Interestingly, despite four different samples collected from patient 6 (ranging from 18 to 29 days after symptom onset), antibodies were never detected by the Roche assay. Likewise, patient 7 antibodies were not detected by the DiaSorin assay; however, the two samples were collected at days 6 and 8, which is relatively early during the course of infection (Table 6 ). Once a patient was positive by an assay, all sera from subsequent collection days were also positive; and in no instances did a patient go from positive to negative for this time frame. 

We conducted a head-to-head comparison of 12 different serology assays for detection of SARS-CoV-2 antibodies, in 161 different samples from 143 patients. We found that Abbott, Affinity, and Bio-Rad assays had the highest clinical sensitivity and specificity among the EIAs. Despite the relatively small panel size and the wide statistical confidence intervals, we believe that these data are very useful and informative for evaluation, comparison, and validation purposes. Most EIAs and POCTs performed well after 21 days after symptom onset; however, and most importantly, no single assay was sensitive enough to detect antibodies Ͻ7 days after symptom onset. Even assays with IgM or IgA components were unable to detect antibodies reliably before 14 days from time of symptom onset (Tables 3 and 4 ). Furthermore, many of the assays did not perform at a level that is considered acceptable for laboratory diagnostics (Ͼ95% sensitivity and specificity). The use of these assays for diagnostic testing is therefore not recommended.

It is worth noting that these performance characteristics were determined using a cohort of COVID-19-positive patients which represented mostly hospitalized patients with high mortality rates. Preliminary studies have suggested that more severe infection may entail higher seroconversion rates and antibodies may develop earlier than in mild infection (8) . Therefore, our study data may artificially improve performance characteristics of the assays. Further studies should be done for mostly nonhospitalized COVID-19-positive patients to ensure that findings remain consistent across all cohorts.

As the time course for antibody development has not yet been fully determined for SARS-CoV-2, it is important to compare equivalent start points when analyzing and comparing these data. The sensitivity of serology assays decreases significantly when time of symptom onset rather than time from first PCR positive result is used as the start of infection. In our study, time of PCR positivity was 5.3 days after date of symptom onset on average (range, 0 to 19 days). This time difference has the potential to significantly skew the data. If the time of the first positive PCR result rather than the time from symptom onset had been used as the starting point for this study, all EIAs and POCTs would have been shown to perform better earlier in the disease course. However, using a positive PCR result to define the start of the infection would be an inaccurate reflection of the development of antibody in an individual. Therefore, whenever possible, studies should strive to include performance data in relation to date of onset, to allow true comparison of SARS-CoV-2 testing data.

Cross-reactivity was observed with the respiratory viruses PIV-4, hMPV, rhinovirus/ enterovirus, and most importantly CoV-229E, CoV-NL63, and CoV-OC43 in some of the commercial EIAs. To date, only small panels assessing cross-reactivity have been tested, and very little cross-reactivity with coronaviruses other than SARS-CoV-1 has been shown (9). Our larger panel of convalescent-phase sera suggests that cross-reactivity with other respiratory viruses may influence SARS-CoV-2 serology results, particularly following a recent respiratory infection (within 13 to 49 days post-RPP). As the majority of individuals have been exposed to the endemic coronaviruses 229E, NL63, OC43, and HKU1 during their lifetime (10), the cross-reactivity of these viruses in particular should be evaluated. In our hands, two different assays showed cross-reactivity to CoV-229E (Euroimmun IgA and Bio-Rad IgG), and one assay (Euroimmun IgA) showed crossreactivity to two different patient samples positive for CoV-NL63 and an additional two different patient samples positive for CoV-OC43 (Table 5 ). An additional study found a single patient sample to be cross-reactive to coronavirus-OC43 (6) , and although the sample number tested was small (n ϭ 5), these results are consistent with what we observed in our panel. The potential cross-reactivity of the endemic coronaviruses with SARS-CoV-2 is a significant drawback. Cross-reactivity to other coronaviruses should therefore be considered when interpreting serological results, particularly when determining SARS-CoV-2 prevalence of infection.

We assessed the ability of EIAs and POCTs to detect antibodies against SARS-CoV-2 in human sera; however, no correlation with neutralizing antibodies was performed.

While a few neutralizing-antibody studies have been completed and have shown that some antibodies are protective (namely, against the receptor-binding domain [RBD] of the spike domain) (11) , the assays that we evaluated in this study detect total, not specifically neutralizing, antibodies. Additionally, not all assays target the same antigenspecific antibodies, and therefore, the kinetics of detection may not be equivalent between assays (Table 1) . As antigen presentation and trafficking will vary between different epitopes, variability between assays is expected. We therefore recommend caution in the use of these serology results as indicators of immunity, and we recommend that further studies be done to measure appropriate immunity markers. This will be particularly important for the POCTs, as the antigens used are often not described (Table 1) . Additionally, because SARS-CoV-2 is a recently emerged virus, there are few data on the longevity of immunity following infection. Here, we found that both IgM, IgA, and IgG were detected in samples Ͼ45 days after symptom onset, suggesting that antibodies are detectable for extended periods; however, more extensive long-term evaluations on the level and duration of immunity following infection are needed.

While the use of serology assays in clinical testing is currently under debate, some recommendations as to the appropriate use of these assays are beginning to emerge. The American Society for Microbiology (ASM) and the WHO recently published similar recommendations against using serology testing for diagnosis of acute infection (12, 13) . A natural delay is seen from the time a patient is exposed to the virus to the time the patient starts creating antibodies against the virus. Therefore, regardless of the sensitivity of the assay, there will always be a delay between infection and the development of antibodies. Recent studies suggest that most patients seroconvert between 7 and 14 days, with IgM and IgA being detected as early as 3 and 4 days after symptom onset, respectively (9) . However, we found that detection of antibodies earlier than 14 days was unreliable but that the performance of all serology assays improved over time. To this end, both the Centers for Disease Control (14) and the Public Health Agency of Canada (15) have also recommended against using serology assays as an aid in the diagnosis of acute infection. However, serology assays will be helpful in understanding the prevalence of SARS-CoV-2 infection in the population, the timeline of antibody development in different patient populations, and the longevity of the antibody response.

Here, we present results for a comprehensive serology panel consisting of sera from known COVID-19-positive patients and known negatives. We evaluated six different commercial EIA platforms and six POCTs with the same serum panel to give an accurate comparison across all platforms. Based on our results, serology assays should not be used for the diagnosis of acute infections but rather in carefully designed serosurveys to facilitate understanding of seroprevalence in a population and to identify previous exposure to the virus.

A novel coronavirus emerging in China-key questions for impact assessment

Feng Z. 2020. Early transmission dynamics in

Coronavirus disease (COVID-19) situation report 163

The laboratory diagnosis of COVID-19 infection: current issues and challenges

Evaluation of 12 SARS-CoV-2

Diagnostics Committee of the Infectious Diseases Society of America (IDSA). 2020. Infectious Diseases Society of America guidelines on the diagnosis of COVID-19

Evaluation of commercial and automated SARS-CoV-2 IgG and IgA ELISAs using coronavirus disease (COVID-19) patient samples

Developing antibody tests for SARS-CoV-2

SARS-CoV-2 serological analysis of COVID-19 hospitalized patients, paucisymptomatic individuals and blood donors

Serological approaches for COVID-19: epidemiologic perspective on surveillance and control

A serological assay to detect SARS-CoV-2 seroconversion in humans

Rapid generation of neutralizing antibody responses in COVID-19 patients

Report from the American Society for Microbiology COVID-19 international summitt

Laboratory testing strategy recommendations for COVID-19: interim guidance 20

Serology testing for COVID-19 at CDC

Serological tests for use against COVID-19. Government of Canada

We thank the staff of the Public Health Laboratory for their help with specimen logistics. We also thank the following manufacturers for supplying kits for analysis: Abbott, Affinity, Bio-Rad, DiaSorin, Euroimmun, Roche, BTNX, Biolidics, Deep Blue, Genrui, Getein BioTech, and Innovita.