key: cord-0883103-zv36s3q8
authors: Plaga, Alexis; Wei, Ruhan; Olson, Elizabeth; Payto, Drew; Harrington, John; Nwe-Kissig, Pyik That; Strizzi, Michelle; Zilka, Sarah; Ko, Jennifer; Colón-Franco, Jessica M
title: Evaluation of the clinical performance of seven serological assays for SARS-CoV-2 for use in clinical laboratories
date: 2021-04-07
journal: J Appl Lab Med
DOI: 10.1093/jalm/jfab038
sha: dd8d6a01c2541dca93e7b4773f7ebbe6ac2fb9b6
doc_id: 883103
cord_uid: zv36s3q8

BACKGROUND: Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) serological assays have emerged as a response to the global pandemic, warranting studies evaluating their clinical performance. This study investigated seven commercially available SARS-CoV-2 serological assays in samples from non-infected individuals and hospitalized patients. METHODS: SARS-CoV-2 qualitative serological assays by Abbott (IgG), Beckman (IgG), DiaSorin (IgG), EUROIMMUN (IgG and IgA), Roche and Bio-Rad (Total) were evaluated using specimens collected pre-December 2019 (n=393), from nucleic acid amplification testing (NAAT) negative patients (n=40), and from 53 patients with COVID-19 by NAAT collected 3-21 days post-onset of symptoms (POS) (N=83). Negative agreement (NA), positive agreement (PA), and positive and negative predictive values (PPV and NPV) at prevalences of 5% and 10% were calculated. RESULTS: The overall %NA;95% CI in the negative samples were: Roche 99.8%; 99.3-100.2, Beckman 99.8%; 98.7-100.0, Abbott and Bio-Rad 99.3%; 98.0-99.9, DiaSorin 98.4; 97.2-99.6, EUROIMMUN IgG 97.5%; 95.5-98.7, and EUROIMMUN IgA 79.7%; 75.9-83.5), accounting for positive/equivocal results as false positives. The %PA; 95% CI in samples collected 14+days POS (n=24) were: Bio-Rad 83.3%; 68.4-98.2, Abbott and Roche 79.2%; 62.9-95.4, EUROIMMUN IgA 70.8%; 52.6-89.0, Beckman 58.3%; 38.6-78.1, DiaSorin 54.2; 34.2-74.1, and EUROIMMUN IgG 50.0%; 30.0-70.0, accounting for negative/equivocal results as false negatives. NPVs ranged from 97.4-98.9% and 94.7-97.7% for prevalences 5% and 10%, respectively. PPVs ranged from 15.5-94.8% and 27.9-97.4% for prevalences 5% and 10%, respectively. CONCLUSIONS: The Roche and Beckman assays resulted in fewer false positives followed by the Bio-Rad and Abbott assays. While the Bio-Rad assay demonstrated higher antibody detection in COVID-19-positive patients, PA claims cannot be established with a high level of confidence in our sample population.

COVID-19-positive patients, PA claims cannot be established with a high level of confidence in our sample population.

This study evaluated and compared seven commercially available serological assays in three wellcategorized cohorts. The data could provide useful insights into the proper utilization of the assays in clinical settings.

Serological assays for the Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2), including IgM, IgA, IgG, or total antibodies, are widely available worldwide in a variety of testing formats, making them appealing for clinical and epidemiological use during the ongoing coronavirus disease 2019 (COVID- 19) pandemic. However, serological testing has been controversial and there has been confusion about interpretation and applications (1) (2) (3) . According to several organizations, serology testing is not useful to diagnose acute infection and to establish immunity correlation claims (4, 5) . Serological testing may be beneficial in epidemiologic studies, in vaccine development and for verification response, selection of convalescent plasma donors, and to aid in the diagnosis of a subset of patients negative by molecular testing and children with the multisystem inflammatory syndrome (4-6). However, the value of serology testing on patient management decisions is still largely unknown (4, 5) supporting the need to further our understanding of clinical performance and baseline characteristics of serologic assays.

We characterized the clinical performance of seven serologic assays with a focus on specificity in a large cohort of samples collected before 2019 and in samples from nucleic acid amplification testing (NAAT)-negative patients, and in acute disease in samples from hospitalized COVID-19positive patients.

This study was approved by the Cleveland Clinic Foundation Institutional Review Board. (7); statistics were also calculated in vice versa. Using the former approach, positive and negative predictive values (PPV, NPV) were calculated from the overall %NA and 14+days %PA. Analysis were carried out using © MedCalc software Ltd (Osten, Belgium).

The IgA test had the lowest %NA at 79.7%, while the other assays demonstrated overall %NA of 97.5-99.8% (n=424-433) ( Table 1 ). The %NA was not appreciably different in the pre-December 2019 subcohorts (n=393; 94 in the cross-reactivity cohort). Ninety-seven specimens had positive/equivocal results by at least 1 assay, 12 by 2 assays (EUROIMMUN IgA, DiaSorin (n=5); EUROIMMUN IgA, EUROIMMUN IgG (n=4)), and 3 by 3 assays (EUROIMMUN IgA, EUROIMMUN IgG, DiaSorin) ( Table 1 and data not shown). Only 23 corresponded to the crossreactivity group (n=94), 14 of which expressed at least 2 known serological/autoimmune makers and 9 expressed more than 2 (Supplementary Table 2 ). Samples with false positives most commonly expressed EBV IgG, followed by cytomegalovirus IgG, varicella zoster IgG, measles IgG and anti-hepatitis B surface antibody (Supplementary Table 1 and Bio-Rad (n=1) assays.

Test positivity increased with days POS only for the Abbott, Roche and Bio-Rad assays ( Table 2 ).

In samples collected 14-21 days POS (n=24), %PA ranged from 50.0-83.3%. The %PA <60% for the Beckman, DiaSorin and EUROIMMUN IgG assays was unexpected. On days 7-21 (n=60),

the Bio-Rad test demonstrated the highest overall %PA. A large seroprevalence study in the US reported variable disease prevalence with most areas still having a prevalence lower than 10% (8).

We therefore calculated PVs for 5% and 10% disease prevalence ( Table 2 ). The Roche assay had the highest PPV at prevalences of 5% and 10% (94.8 and 97.4, respectively), and tied NPVs with the Abbott assay, second to the Bio-Rad assay leading with NPVs of 99.1% and 98.2% for the 5% and 10% prevalences, respectively. The IgA assay had an unacceptably low PPV of 27.9% at a 10% prevalence.

Except the IgA assay, all assays met the >95% specificity criteria required by the FDA for Emergency Use Authorization (EUA) and in agreement with other studies (9) (10) (11) (12) (13) (14) . The %PA for these seven assays did not meet the FDA's sensitivity standard of 90% and was lower than reported (9) (10) (11) (12) . Only the Bio-Rad assay had a %PA above 80% in the 14+days POS samples. It detected one more case than the Roche and Abbott assays and 3 more than the IgA assay, but 6-8 more than the other 3 assays. On samples collected <14 days POS, the Bio-Rad assay detected 2 more cases than the Roche Total assay and 3 more than the IgA assay but 7-10 more than the other assays.

Therrien et al evaluated these assays on samples collected 2-5 weeks POS (n=60) and found the IgA and Bio-Rad assays performed similarly, with sensitivities of 85%;73-95 and 87%;75-94, respectively (9) . They reported performance comparable to our findings for the Roche and Abbott assays but superior %PA for the EUROIMMUN IgG, Beckman and DiaSorin assays ranging from 72-78% (vs 50-58% in our study). In samples collected 14 days+ onset of symptoms, Tang and colleagues reported higher sensitivities by 10 and 14% for the Roche and Abbott assays, respectively, and by almost 40% for the EUROIMMUN IgG assay (10, 11) . Other studies also reported higher sensitivities for the Beckman, DiaSorin, and EUROIMMUN IgG assays (15, 16) .

In the context of many unknown factors in the humoral responses of patients with COVID-19, it is difficult to conclusively determine the reason for the low %PA in our study. An obvious limitation of our study is the low number of samples 14+days POS and only up to 21 days.

Moreover, the timing of sample collection was subjectively determined from chart review. The observed differences may be a factor of the heterogeneity of the populations studied and an uncharacterized role of co-morbidities, present in many hospitalized patients with COVID-19, in seroconversion. A phenomenon of seronegative non-responding patients (i.e. negative by several assays) has been described and although it is not well characterized yet, it may represent at least 5% of 14+days POS samples (14) . In our study, 4/24 samples (16.7%) did not seroconvert by any assay. Lastly, hospitalization rates vary across these studies from approximately 50 to 67% (9, 16), or was not mentioned (10) . Our cohort consisted of hospitalized patients (1-38 days), and 50/53 were symptomatic with mild (n=10), moderate (n=11), critical (n=18), and severe (n=11) disease.

Three strategies are recommended by the CDC to improve PPV (4). First, to use assays with >95% specificity, but preferably >99.5%. The most stringent criteria was only met by the Roche and Beckman assays. Second, to test patients with a high pre-test probability. The third recommendation is orthogonal testing (i.e. testing positive samples using another test preferably with a different format or antigen target). For the IgA assay, 12 samples were also positive on at least another assay targeting the viral spike protein (EUROIMUNN IgG and DiaSorin), but none of the assays targeting the nucleocapsid (Abbott, Roche, Bio-Rad) or the receptor biding domain (Beckman). The IgA assay had the lowest PPV, a concerning number of false-positive results, and should not be used in low prevalence settings. In alignment with a study reporting IgA expression in asymptomatic pregnant women (17), 3 of 10 samples from pregnant women were positive for IgA. Although it has been suggested that IgA appears earlier than IgG in patients with COVID-19 (9, 18) , our data and longitudinal samples did not support this conclusion (Table 1 and data not shown). It is important to mention that our study calculated PVs using the results from the 14+ POS cohort, which has a very low sample size, and is not well powered to assess PPVs, as indicated by the large 95% confidence intervals.

We found that the Roche, Bio-Rad and Abbott assays performed better 14+days POS in hospitalized patients. Using 20-40 samples for EUA assay validation may not be sufficient to establish robust %PA claims (19, 20) . In some cases, extending collection to >7 weeks POS was needed to yield sensitivities near 90% (9) . Nonetheless, serology testing is not recommended for diagnosing acute COVID-19 infection (4, 5). We concluded with high level of confidence that the Roche, Beckman, Abbott and Bio-Rad assays detect few false positives and may be useful in seroprevalence studies.

A. Plaga, provision of study material or patients; D. Payto, administrative support, provision of study material or patients; E. Olson, administrative support; M. Strizzi, administrative support; J. Ko, administrative support, provision of study material or patients. 

SARS-CoV-2 serology: much hype, little data

Rapid roll out of SARS-CoV-2 antibody testing-a concern

Clinical Application of Serologic Testing for Coronavirus Disease 2019 in Contemporary Cardiovascular Practice

Infectious Diseases Society of America Guidelines on the Diagnosis of COVID-19

American College of Rheumatology Clinical Guidance for Multisystem Inflammatory Syndrome in Children Associated With SARS-CoV-2 and Hyperinflammation in Pediatric COVID-19: Version 2

Food and Drug Administration. Independent Evaluations of COVID-19 Serological Tests

Estimated SARS-CoV-2 Seroprevalence in the US as of

Multicenter Evaluation of the Clinical Performance and the Neutralizing Antibody Activity Prediction Properties of ten high throughput serological assays used in Clinical Laboratories

Clinical performance of two SARS-CoV-2 serologic assays

Clinical performance of the Roche SARS-CoV-2 serologic assay

Performance Characteristics of the Abbott Architect SARS-CoV-2 IgG Assay and Seroprevalence in

Food and Drug Administration

Multi-center nationwide comparison of seven serology assays reveals a SARS-CoV-2 non-responding seronegative subpopulation

Antibodies in serum of convalescent patients following mild COVID-19 do not always prevent virus-receptor binding

Comparison of eight commercial, high-throughput, automated or ELISA assays detecting SARS-CoV-2 IgG or total antibody

Fill Malfertheiner S. Peripartal anti-SARS-CoV-2-IgA/IgG in asymptomatic pregnant women during regional SARS-CoV-2-outbreak

Evaluation of SARS-CoV-2 prototype serologic test in hospitalized patients

Laboratory evaluation of SARS-CoV-2 in the COVID-19 pandemic

Application, Verification, and Implementation of SARS-CoV-2 Serologic Assays with Emergency Use Authorization

CI, confidence interval Bolded data represents the %PA that changed if equivocal results were considered positive. Predictive values were calculated using the overall %NA including positive and equivocal results as false positives