key: cord-0723383-rq0ousqz
authors: Kanwar, Neena; Banerjee, Dithi; Sasidharan, Anjana; Abdulhamid, Ayah; Larson, Marissa; Lee, Brian; Selvarangan, Rangaraj; Liesman, Rachael M
title: Comparison of Diagnostic Performance of Five Molecular Assays for Detection of SARS-CoV-2
date: 2021-08-06
journal: Diagn Microbiol Infect Dis
DOI: 10.1016/j.diagmicrobio.2021.115518
sha: 1294138b50b7166c9dc89fe345ed376151185741
doc_id: 723383
cord_uid: rq0ousqz

BACKGROUND: We compared the performance of the Abbott RealTime SARS-CoV-2 assay (Abbott assay), Aptima™ SARS-CoV-2 assay (Aptima assay), BGI Real-Time SARS-CoV-2 assay (BGI assay), Lyra® SARS-CoV-2 assay (Lyra assay), and DiaSorin Simplexa™ COVID assay for SARS-CoV-2 detection. METHODS: Residual nasopharyngeal samples (n=201) submitted for routine SARS-CoV-2 testing by Simplexa assay during June-July 2020 and January 2021 were salvaged. Aliquots were tested on other assays and compared against the CDC 2019-nCoV Real-Time RT-PCR assay. Viral load in positive samples was determined by droplet digital PCR. RESULTS: Among 201 samples, 99 were positive and 102 were negative by the CDC assay. The Aptima and Abbott assays exhibited the highest positive percent agreement (PPA) at 98.9% while the BGI assay demonstrated the lowest PPA of 89.9% with 10 missed detections. Negative percent agreement for all 5 platforms were comparable, ranging from 96.1% to 100%. CONCLUSIONS: The performance of all five assays were comparable.

Consequently, clinical laboratories across the United States faced challenges in identifying reliable diagnostic assays for SARS-CoV-2 detection given the limited availability of published performance and comparison data. Additional challenges include supply chain difficulties, which many laboratories have addressed by implementing multiple assays (6, 7) . Comparative evaluations of analytical performance as well as real world clinical performance of EUA assays are essential to help guide assay selection as laboratories expand testing capacity (8) .

Clinical study data comparing several of the molecular assays have recently been published (9) (10) (11) (12) (13) (14) (15) (16) (17) (18) . To our knowledge, this is the first study comparing the performance of Lyra SARS-CoV-2 Assay (Lyra assay), and the second large scale clinical study with the BGI Real-Time SARS-CoV-2 assay (BGI assay). Our current study is one of a few that evaluated the clinical diagnostic performance of five or more assays (19) (20) (21) . The study objective was to analyze and compare the diagnostic performance of five EUA approved molecular assays, the Abbott Real-time SARS-CoV-2 assay (Abbott assay), Aptima SARS-CoV-2 Assay (Aptima assay), BGI assay, Lyra assay, and DiaSorin Simplexa COVID-19 Direct assay (Simplexa assay), using CDC 2019-nCoV Real-Time RT-PCR assay (CDC assay) as the reference standard on clinical salvage respiratory samples.

Study design: This prospective salvage sample study included 201 nasopharyngeal samples submitted to the University of Kansas Health System (TUKHS) clinical laboratory for routine SAR-CoV-2 testing on the Simplexa COVID-19 Direct assay (DiaSorin Molecular LLC, California, USA) during the months of June/July 2020 and January 2021. Of the 201 specimens analyzed, 100 were positive and 101 were negative on initial testing by Simplexa. Six aliquots of fresh sample were made within 72 hours of the sample collection. Three aliquots were shipped to the Children's Mercy Hospital clinical laboratory on dry ice for testing on three platforms (Aptima assay, CDC assay, and Lyra assay). All aliquots were thawed only once before testing on respective platforms. Testing on Abbott assay and BGI assay were performed at the TUKHS clinical laboratory. Characteristics of all the assays evaluated in the study have been described in Table 1 . Test results from each of the five individual assays were compared to CDC reference standard as reported previously (20, (22) (23) (24) (25) (26) .The Bio-Rad SARS-CoV-2 ddPCR test was performed on all samples that tested positive by any assay. Ct values obtained from each assay were compared with the quantitative results from ddPCR assay.

This study was approved by the Institutional Review Board of both institutions.

Nasopharyngeal swabs (Shenzhen Medico Technology Co., Shenzhen City, China) were transported in 3mL of viral transport media (Jinan Babio Biotechnology Co., Ltd., Shandong, China), stored at 2-8 o C, and tested with the Simplexa assay.

Clinical testing and aliquoting for testing with comparator assays were performed within 72 hours of sample collection. Aliquots were stored at -70˚C until further testing.

Study population: Symptomatic as well as non-symptomatic patients of both genders were included in the study. Samples were submitted from both inpatient and emergency departments at the TUKHS. Testing location and reason for testing (asymptomatic screen, symptomatic testing) were recorded ( Table 2) . (Table 1) . This assay requires detection of the pp1ab gene to report a positive sample. Amplification was performed using 5.5 μL of the easyMAG extract and 16.5 μL Bio-Rad One-

Step RT-ddPCR supermix and loaded on QX200 TM AutoDG Droplet Generator (Bio-Rad) and fractionated into up to 20,000 nanoliter-sized droplets which were collected on a new 96-well plate and heat sealed. One-step reverse transcription and amplification were performed on a C1000 Touch TM Thermal Cycler (Bio-Rad) with 96-deep well reaction module using the following thermal cycling conditions: 50℃ for 60 min (reverse transcription); 95℃ for 10 min; and 40 cycles of 94℃ for 30 sec, and 55℃ for 60 sec followed by 98℃ for 10 min (enzyme deactivation) and 4℃ for 30 min (droplet stabilization). The plate was then transferred to the 

A total of 201 samples were tested. Results from each molecular platform were compared to the CDC reference standard result. Both symptomatic (n=128) and asymptomatic (n=73) subjects were included in the study. There was approximately equal distribution of male and female study subjects (Female: n= 98/201, 48.8 %). The median age of the patients included in the study was 57 years ( A total of 106 samples positive by at least one platform were tested by ddPCR to determine viral load. Of the 106 samples, 21 samples were reported as negative on at least one platform and were included for overall discrepant analysis (Table 4 ). By testing with the CDC assay, 14 of the 21 discordant samples were negative and 7 were positive. The ddPCR result was in agreement with the original CDC result in 18/21 (85.7%) overall discordant samples. The remaining 3 samples were negative by ddPCR (viral load below the 100 copies/ml threshold) and positive by CDC testing. Each of these samples was detected at a high Ct value (or low RLU value) by the CDC assay and most other tested assays and the low viral burden of these samples may account for the negative ddPCR result (Table 4) .

Overall viral load in all positive samples ranged between 150 copies/ml to 98.5 million copies/mL for N1 gene and 162 copies/mL to 93.8 million copies/mL for N2 gene. Positive to the uncapped workflow during the study when capped tubes became unavailable. The RLU values for three of the four positive specimens were 671, 786, and 987, considered to be low positives (cut-off RLU for Aptima assay is 560) (27) ; RLU for the fourth specimen was 1212.

Multiple commercial SARS-CoV-2 molecular assays are currently marketed under FDA EUA without undergoing the typical rigorous clinical trials for obtaining FDA clearance/approval.

Comparative performance studies for SARS-CoV-2 detection are of importance to the clinical community as they provide real world performance data to help guide testing decisions. This study compared the performance of five SARS-CoV-2 NAATs for 201 NP swab specimens collected from both symptomatic and asymptomatic subjects. Other comparator studies recently evaluated four of the six assays included in our study assay and reported comparable performances (6, 10, 11, 18, 25, 26, (28) (29) (30) (31) (32) (33) (34) . To our knowledge, this is the first report of relative clinical performance data of the Lyra assay for SARS-CoV-2 detection and one of the few that compares five platforms simultaneously.

The ddPCR is thought to be more sensitive compared to many RT-PCR assays, as there is less dependence on efficiency of PCR amplification due to end point measurement of nucleic acid quantitation. Additionally, ddPCR is less affected by reaction inhibitors due to microdilution in each droplet (35) . The ddPCR assay results are reported by gene copy, an absolute quantification, rather than Ct value, which varies significantly among different NAAT platforms (36) . Ct value varied among different NAATs evaluated in this study, which may be due to protocol differences (eg, input volume), differences in extraction method (eg, sample-to-answer vs full extraction), amplification technology (eg, TMA vs RT-PCR), differences in target region of amplification, sampling error (especially for specimens with low viral load), and assay design variability (eg, florescence data capture starts after several amplification cycles, as is the case for the Abbott and Lyra assays) ( Table 2 ). The Ct variability across different platforms for samples with the same viral load as demonstrated in this study (Figure 1a and 1b) supports the determination that Ct values should be interpreted with caution (36, 37) . If required, absolute quantification of viral copies by ddPCR may be a more reliable method of determining viral burden. However, ddPCR missed detection of three true positive samples with low viral loads and corresponding high Ct values on other platforms, demonstrating a limitation that viral loads below the limit of detection of the assay (<100 copies/mL) would go undetected by ddPCR.

Few limitations of our study require consideration. Samples were selected based on the Simplexa assay results and repeat testing after specimen freeze-thaw was not performed on the Simplexa platform, unlike comparable platforms. Theoretically, freeze-thaw may lead to nucleic acid degradation or, alternatively, may remove non-specific inhibition, both of which may influence detection rate; nonetheless, all assays were comparable in detection rate. Additionally, this study included predominantly samples collected from adult population with only one sample from a patient <18 years old. Therefore, caution is advised while extrapolating study results to a pediatric population. The CDC PCR assay was used as a reference standard method in this study.

The absence of true reference standard for SAR-CoV-2 detection is well recognized (38, 39) .

Laboratorians have used different approaches for establishing an arbitrary reference standard for diagnostic assay comparison studies such as using composite/consensus reference standard or comparing results to one of the RT-PCR results (39) . We considered the CDC assay as an arbitrary reference method, as it has demonstrated excellent performance characteristics in previous studies (20, 28, 29, 40) . We recognize that a more sensitive assay could alter the study results. We also evaluated results using consensus reference standard, defined as positive result in ≥4 of 6 NAAT assays and consensus negative was a negative result in ≥4 of 6 NAAT assays (data not shown). Results from consensus reference method demonstrated comparable results to CDC reference method; additionally, the CDC assay and consensus reference method were found to have a PPA and NPA of 100% and 99%, respectively (data not shown). Study strengths include direct comparison of five commonly used EUA approved NAAT platforms for SARS-CoV-2 detection. The sample pool came from both symptomatic and non-symptomatic patients.

With the presence of multiple diagnostic assays in the field, the decision to select a SARS-CoV-2 molecular assay depends upon the laboratory setting and infrastructure, resources with respect to staff and cost involved, testing volume, patient population, and indication for testing. Assays detecting only a single gene target pose a theoretical risk for false negative results with the emergence of viral variants, although this was not evaluated in this study. Our study evaluated the performance of five platforms in a controlled study design and our data shows comparable results for SARS-CoV-2 detection across all the platforms tested.

Funding declaration: This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors. 

Coronavirus disease (COVID-19) outbreak

Accessed 07/15/2020

Testing for COVID-19: when, who, and what test

Report from the American Society for Microbiology COVID-19 International Summit

Infectious Diseases Society of America Guidelines on the Diagnosis of COVID-19, on IDSA

Emergency Use Authorization: Emergency Use Authorization (EUA) information, and list of all current EUAs

Comparison of Commercially Available and Laboratory-Developed Assays for In Vitro Detection of SARS-CoV-2 in Clinical Laboratories

SARS-CoV-2 molecular testing: Summary of recent SARS-CoV-2 molecular testing survey

SARS-CoV-2 Testing

Analytical and Clinical Comparison of Three Nucleic Acid Amplification Tests for SARS-CoV-2 Detection

Comparison of Four Molecular In Vitro Diagnostic Assays for the Detection of SARS-CoV-2 in Nasopharyngeal Specimens

Comparison of Abbott ID Now, Diasorin Simplexa, and CDC FDA EUA methods for the detection of SARS-CoV-2 from nasopharyngeal and nasal swabs from individuals diagnosed with COVID-19

Clinical performance of the Luminex NxTAG CoV Extended Panel for SARS-CoV-2 detection in nasopharyngeal specimens of COVID-19 patients in Hong Kong

Evaluation of the commercially available LightMix® Modular E-gene kit using clinical and proficiency testing specimens for SARS-CoV-2 detection

Comparison of Two High-Throughput Reverse Transcription-Polymerase Chain Reaction Systems for the Detection of Severe Acute Respiratory Syndrome Coronavirus 2

Performance of Abbott ID NOW COVID-19 rapid nucleic acid amplification test in nasopharyngeal swabs transported in viral media and dry nasal swabs, in a New York City academic institution

The Detection of SARS-CoV-2 using the Cepheid Xpert Xpress SARS-CoV-2 and Roche cobas SARS-CoV-2 Assays

Clinical Evaluation of the cobas SARS-CoV-2 Test and a Diagnostic Platform Switch during 48 Hours in the Midst of the COVID-19 Pandemic

Clinical evaluation of the BioFire® Respiratory Panel 2.1 and detection of SARS-CoV-2

Comparative performance of SARS-CoV-2 real-time PCR diagnostic assays on samples from

Comparison of Commercially Available and Laboratory-Developed Assays for In Vitro Detection of SARS-CoV-2 in Clinical Laboratories

Evaluation of seven commercial RT-PCR kits for COVID-19 testing in pooled clinical specimens

Rapid Detection of 2019 Novel Coronavirus SARS-CoV-2 Using a CRISPR-based DETECTR Lateral Flow Assay. medRxiv : the preprint server for health sciences

Comparison of Abbott ID Now, DiaSorin Simplexa, and CDC FDA Emergency Use Authorization Methods for the Detection of SARS-CoV-2 from Nasopharyngeal and Nasal Swabs from Individuals Diagnosed with COVID-19

Evaluation of the COVID19 ID NOW EUA assay

Validation and verification of the Abbott RealTime SARS-CoV-2 assay analytical and clinical performance

Comparison of SARS-CoV-2 detection from nasopharyngeal swab samples by the Roche cobas 6800 SARS-CoV-2 test and a laboratory-developed real-time RT-PCR test

Performance Characteristics of a High-Throughput Automated Transcription-Mediated Amplification Test for SARS-CoV-2 Detection

Comparison of two commercial molecular tests and a laboratory-developed modification of the CDC 2019-nCOV RT-PCR assay for the qualitative detection of SARS-CoV-2 from upper respiratory tract specimens

Comparing the analytical performance of three SARS-CoV-2 molecular diagnostic assays

Comparison of a laboratory-developed test targeting the envelope gene with three nucleic acid amplification tests for detection of SARS-CoV-2

Comparison of the analytical sensitivity of seven commonly used commercial SARS-CoV-2 automated molecular assays

Clinical Evaluation and Utilization of Multiple Molecular In Vitro Diagnostic Assays for the Detection of SARS-CoV-2

Rapid and sensitive detection of SARS-CoV-2 RNA using the Simplexa™ COVID-19 direct assay

Comparison of Abbott ID Now and Abbott m2000 Methods for the Detection of SARS-CoV-2 from Nasopharyngeal and Nasal Swabs from Symptomatic Patients

Droplet Digital PCR versus qPCR for gene expression analysis with low abundant targets: from variable nonsense to publication quality data

Challenges and Controversies to Testing for COVID-19

Can the SARS-CoV-2 PCR Cycle Threshold Value and Time from Symptom Onset to Testing Predict Infectivity?

Understanding, Verifying, and Implementing Emergency Use Authorization Molecular Diagnostics for the Detection of SARS-CoV-2 RNA

The estimation of diagnostic accuracy of tests for COVID-19: A scoping review

range