key: cord-1022132-xgosdgh0
authors: Sieker, Jakob T; Horowitz, Coby; Hu, Cheng-Tsung K; Lacombe-Daphnis, Meriane; Chirokas, Bernadette; Pina, Coteia; Heger, Nicholas E; Rabson, Arthur R; Zhou, Ming; Bogen, Steven A; Horowitz, Gary L
title: Analytic Sensitivity of Three Nucleic Acid Detection Assays in Diagnosis of SARS-CoV-2 Infection
date: 2020-10-14
journal: J Appl Lab Med
DOI: 10.1093/jalm/jfaa187
sha: 2716dbef29dbc4ec07d2990e0877a8bf4f351d9f
doc_id: 1022132
cord_uid: xgosdgh0

BACKGROUND: Detection of SARS-CoV-2 by reverse transcriptase polymerase chain reaction is the primary method to diagnose Coronavirus Infectious Disease 2019 (COVID-19). Yet, the analytical sensitivity required is not well defined and it is unclear how available assays compare. METHODS: For the Abbott RealTime SARS-CoV-2 assay (Abbott Molecular Inc.; abbreviated as m2000), we determined that it could detect viral concentrations as low as 26 copies/mL, we defined the relationship between cycle number and viral concentrations, and we tested naso- and oropharyngeal swab specimens from N = 8,538 consecutive individuals. Using the m2000 as a reference assay method, we described the distribution of viral concentrations in these patients. We then used selected clinical specimens to determine the positive percent agreement of two other assays with more rapid turnaround times (Cepheid Xpert Xpress (Cepheid Inc.; GeneXpert, N = 27 specimens) and a laboratory developed test on the Luminex ARIES system (Luminex Corporation; ARIES LDT, N = 50)) as a function of virus concentrations, from which we projected their false negative rates in our patient population. RESULTS: SARS-CoV-2 was detected in 27% (95% confidence interval of 26-28%) of all specimens. Estimated viral concentrations were widely distributed and 17% (16-19%) of positive individuals had viral concentrations below 845 copies/mL. Positive percent agreement was strongly related to viral concentration and reliable detection (i.e. ≥95%), was observed at concentrations >100 copies/mL for the GeneXpert but not the ARIES LDT, corresponding to projected false negative rates of 4% (0-21%) and 27% (11-46%), respectively. CONCLUSIONS: Substantial proportions of clinical specimens have low to moderate viral concentrations and may be missed by methods with lesser analytical sensitivity.

Detection of SARS-CoV-2 by reverse transcriptase polymerase chain reaction is the primary method to diagnose Coronavirus Infectious Disease 2019 . Yet, the analytical sensitivity required is not well defined and it is unclear how available assays compare.

For the Abbott RealTime SARS-CoV-2 assay (Abbott Molecular Inc.; abbreviated as m2000), we determined that it could detect viral concentrations as low as 26 copies/mL, we defined the relationship between cycle number and viral concentrations, and we tested naso-and oropharyngeal swab specimens from N=8,538 consecutive individuals. Using the m2000 as a reference assay method, we described the distribution of viral concentrations in these patients. We then used selected clinical specimens to determine the positive percent agreement of two other assays with more rapid turnaround times (Cepheid Xpert Xpress (Cepheid Inc.; GeneXpert, N=27 specimens) and a laboratory developed test on the Luminex ARIES system (Luminex Corporation; ARIES LDT, N=50)) as a function of virus concentrations, from which we projected their false negative rates in our patient population.

Estimated viral concentrations were widely distributed and 17% (16-19%) of positive individuals had viral concentrations below 845 copies/mL. Positive percent agreement was strongly related to viral concentration and reliable detection (i.e. ≥95%), was observed at concentrations >100 copies/mL for the GeneXpert but not the ARIES LDT, corresponding to projected false negative rates of 4% (0-21%) and 27% (11-46%), respectively.

Substantial proportions of clinical specimens have low to moderate viral concentrations and may be missed by methods with lesser analytical sensitivity.

Although the infectious dose for SARS-CoV-2 is currently unknown, a low viral concentration in respiratory samples may represent poor swabbing technique of a potentially infectious patient or may be capturing a positive patient very early in their clinical course. It is therefore imperative to be able to detect even a low viral concentration. For this, analytical sensitivity of SARS-CoV-2 polymerase chain reaction assays may be key.

caused by SARS-CoV-2, that emerged in late 2019 in Wuhan, China (1). Coronaviruses are enveloped RNA viruses, and SARS-CoV-2 falls into the genus betacoronavirus, which also includes the SARS-CoV identified in 2003 (1). SARS-CoV-2 actively replicates in the upper and lower respiratory tract, and active shedding from the pharynx and sputum is documented (2) . Strong evidence for the detection of SARS-CoV-2 in asymptomatic and minimally symptomatic individuals and for the transmission from them has emerged (3) (4) (5) , explaining in part the rapid spread across the globe. Detection of SARS-CoV-2 by reverse transcriptase polymerase chain reaction (PCR) is the primary method of diagnosis and has played an important role in the initial response to the pandemic (6), but may play an even larger role once current social distancing measures are relaxed (7) .

The reliable detection of individuals with COVID-19, combined with quarantine and contact tracing, may allow the interruption of infection chains and the containment of the disease. As negative test results are commonly used to relax precautional measures, a false negative test may lead to inadequate precautions and disease transmission.

Logically, much attention has been given to the matter, particularly to the technique of swabbing, the timing of swabbing, the viral transport media used, and the storage or processing of specimens prior to testing (8) (9) (10) (11) (12) . Less attention has been paid to the analytic sensitivity, or lower limit of detection, of the assays.

In the context of SARS-CoV-2 testing, the limit of detection is often defined as the lowest concentration of virus at which it can be detected in at least 95% of attempts (13) .

Manufacturers typically provide limits of detection in the packaging inserts, but the use of different viral standards, matrices, or even metrics, can complicate direct comparisons (13) (14) (15) . A method with an already established limit of detection could serve as a reference for comparing new methods to establish their limit of detection using clinical specimens.

For the Abbott RealTime SARS-CoV-2 assay (Abbott Molecular Inc.; here abbreviated as m2000), we determined that it could reliably detect viral concentrations as low as 26 copies/mL. We defined the relationship between cycle number and viral concentrations, we tested naso-and oropharyngeal swab specimens for N=8,538 consecutive individuals, and we described the distribution of patient viral concentrations. We also validated two other assay platforms and compared them to Abbott m2000, namely, the Cepheid Xpert Xpress SARS-CoV-2 assay with Emergency Use Authorization (EUA) (Cepheid Inc,; here abbreviated as GeneXpert) and a laboratory developed test announced by Luminex on their ARIES system (here abbreviated as ARIES LDT; Luminex Corporation) (16) . The objectives of this study were 1) to determine the positive percent agreement (PPA) of the GeneXpert and the ARIES LDT across the spectrum of patient viral concentrations and 2) using the distribution of patient viral concentrations, to estimate the assays' false negative rates in our patient population.

We determined that potentially important proportions of clinical specimens have low to moderate viral concentrations that may be missed by methods with lesser analytical sensitivity. This could significantly impact the diagnosis and containment of SARS-CoV-2 infection.

A retrospective analysis of specimens tested in our laboratory with the m2000 was used to determine the frequency distribution of estimated viral concentrations in clinical specimens. To determine their PPA, select specimens covering a broad range of viral concentrations were tested contemporaneously on the GeneXpert or ARIES LDT and compared to the m2000. The PPA of each assay was determined using four subsets of clinical specimens, i.e. <26 copies/mL, 26 to <100 copies/mL, 100 to <845 copies/mL, and >845 copies/mL. By combining the subset-specific PPA and the relative frequency of specimens in each subset, the assays' false negative ratios in our patient population were projected.

A total of 8,538 tests, performed from March 16 through April 24, 2020, were included in the analysis and correspond to N=8,538 individuals with one specimen each. All specimens were tested with the m2000, while N=27 and N=50 were additionally tested with the GeneXpert and the ARIES LDT, respectively. As the GeneXpert and ARIES LDT were validated sequentially using clinical specimen as they were received in our laboratory, the specimens tested with the two assays are not identical. The patient characteristics are shown in Table 1 .

The Abbott RealTime SARS-CoV-2 assay (here abbreviated as m2000; Abbott Molecular Inc.) is a real-time, reverse transcriptase PCR test targeting the RdRp (RNA-dependent RNA polymerase) and N (nucleoprotein) genes, with a limit of detection of 100 copies/mL per package insert (13) . The Cepheid Xpert Xpress SARS-CoV-2 assay (here abbreviated as GeneXpert; Cepheid Inc.) is a rapid, real-time, reverse transcriptase PCR test targeting the E (envelope) gene and the N2 region of the N gene, with limits of detection of 250 copies/mL and 0.0100 PFU/mL per package insert (14) . shortage of carrier RNA this step was omitted here. The primer/probe set was selected based on sequences provided by the Centers for Disease Control and Prevention (CDC) (19) and manufactured by Integrated DNA Technologies (Integrated DNA Technologies, Inc.). It included primer/probes against the N1 and N3 regions of the N gene, while using RNAse P as control (see supplementary table 1 for primer/probe sequences). The SARS-CoV-2 assay available for the ARIES system under EUA at the time of publication differs from above protocol, as it no longer requires the manual assembly of cartridges and addition of primers. It also uses different primers targeting the ORF1ab and the N genes (15) .

As provided in the package insert, the manufacturer established a limit of detection of 100 copies/mL (13) . This was further analyzed using standards from two different manufacturers including a recombinant virus containing SARS-CoV-2 RNA (SeraCare AccuPlex SARS-CoV-2 Reference Material, batch 10480311, 4,226 copies/mL; SeraCare) and heat inactivated SARS-CoV-2 (NR-52286, SARS-Related Coronavirus 2, Isolate USA-WA1/2020, 1.16 × 10 9 genome equivalents/mL; BEI Resources) diluted in viral transport media. First, the manufacturer's limit of detection of 100 copies/mL was confirmed with 20 replicates. Furthermore, values lower than those were tested including 20 separate dilutions with concentrations between 7 and 26 copies/mL. In all of these, the positive control material was consistently detected (100%) supporting a limit of detection of 26 copies/mL. The diluted standards underwent a nucleic acid extraction process identical to those for clinical specimens.

Although the m2000 is not intended as a quantitative assay (13) , the relation of cycle number (CN) values with viral concentration was established using reference material (SeraCare) diluted in viral transport medium. Using regression analysis, and N=24

replicates ranging from 26 to 845 copies/mL an exponential relationship between CN value and viral concentration was determined, i.e. Estimated viral concentration = 3 × 10 7 × e (-0.503 * CN value) . The corresponding R 2 value was 80%.

After completion of testing by the m2000, select specimens were also tested using the 

Summary statistics, plots, and 95% confidence intervals were generated using RStudio Version 1.0.143 (RStudio, Inc.). The 95% confidence intervals were calculated according to Wilson using the "binom" R software package (20) . oropharyngeal swabs comprised 89% of the 27 specimens tested using the GeneXpert.

The ARIES LDT detected 35 of 50 selected known positive specimens, resulting in an overall PPA of 70% (95% confidence interval 56-81%, Table 2 ). The false negative individuals were outpatients who had a mean age of 43 years (standard deviation ± 17 years) and included 53% males. All of the false negative specimens were oropharyngeal swabs; however, oropharyngeal swabs comprised 92% of the 50 specimens tested using the ARIES LDT. It should be noted that the specimens selected for testing using the GeneXpert differed from the ones selected for testing using the ARIES LDT; thus, a direct comparison of the measured overall PPA of the two assays should be avoided. The assays' PPA was strongly related to the viral concentration, i.e. the GeneXpert detected all specimens with >100 copies/mL as positive, but had a PPA of 50% (95% confidence interval 15-85%) in the subset of samples with 26 to 100 copies/mL, and did not detect specimens <26 copies/mL. The ARIES LDT had a PPA of 38% (14%-69%) in the subset with 100 to 845 copies/mL, of 33% (6%-79%) in the subset with 26 to 100 copies/mL, and did not detect the specimen with <26 copies/mL. The PPA of the GeneXpert and the ARIES LDT across the spectrum of patient viral concentrations is shown in Table 2 .

The GeneXpert's PPA was 0% (0-66%) in the subset with <26 copies/mL, a subset that represented 1.7% (1.2-2.3%) of all positive specimens. The assay's PPA was 50% (15-85%) in the subset with 26 to 100 copies/mL, which represented 4% (3-5%) of all positive specimens. Using the distribution of patient viral concentrations, the predicted false negative rate across the entire distribution, i.e. in our patient population, is estimated to be 4% (0-21%) for the GeneXpert. Similarly, the false negative rate is estimated to be 27% (11-46%) for the ARIES LDT in our patient population.

We determined that potentially substantial proportions of clinical specimens submitted to our laboratory for SARS-CoV-2 testing have low to moderate viral concentrations and that a proportion of these are false negative by methods with lesser analytical sensitivity. For the assays compared here, the PPA was strongly related to the viral concentration, with lower PPA at lower viral concentrations. As lower viral concentrations were intentionally overrepresented in the set of specimens selected for the assay comparison, the distribution of estimated patient viral concentrations established here was used subsequently, to project the assays' false negative rates in our patient population: 4% (0-21%) and 27% (11-46%) for the GeneXpert and ARIES LDT, respectively.

The analytical sensitivity is an important feature to consider when selecting new assays or platforms. Representing a novel infectious agent, the required level of analytic sensitivity for SARS-CoV-2 testing was not well defined. In our patient population, a limit of detection (assuming 100% detection at and above the limit and 0% detection below the limit) of 845 copies/mL is needed to identify 83% (81-84%) of cases positive on our most sensitive platform, 100 copies/mL is needed to identify 95% (94-96%) of cases, and 26 copies/mL is needed to identify 98.3% (97.7-98.8%) of cases. These data, when combined with the intended clinical setting and population, will be useful in making decisions for assay or platform selection and interpretation of results. In settings where it is important to rule out an infection, analytical sensitivity is an important characteristic of SARS-CoV-2 assays.

The analytical sensitivity of assays is typically established by the manufacturer and reported in the package insert. However, the methods, reference standards, matrix, and metric to report the limit of detection may vary considerably, making it at times difficult to compare assays. Here we had the opportunity to compare assays using clinical specimens against a method with a well-established limit of detection. While not always feasible to perform in all laboratories, similar approaches comparing assays not included in our evaluation are becoming increasingly available (21, 22) .

While we report that potentially significant proportions of clinical specimens submitted for SARS-CoV-2 testing have low to moderate viral concentrations, which may be missed by methods with lesser analytical sensitivity, it is not yet known whether low viral concentrations still represent infectivity. As low viral concentrations may also be related to poor swabbing technique or other pre-analytical variability, it is prudent to assume that patients with low viral concentration observed here encompasses infectious individuals.

We established the limit of detection for the m2000 at 26 copies/mL; specimens of viral concentrations <26 copies/mL may thus be underrepresented in our reported distribution of patient viral concentrations. In order to be broadly useful, we provide a frequency distribution using estimated viral concentrations, instead of CN values. This conversion is based on experiments with reference standards ranging from 26 to 845 copies/mL, demonstrating an exponential relation between CN values and viral concentrations with a corresponding R 2 value of 80%, thus there may be some inaccuracies in the estimated viral concentrations. It should be noted that all of the compared assays are intended to be qualitative (13) (14) (15) .

In summary, a potentially substantial proportion of clinical specimens submitted for SARS-CoV-2 testing have low to moderate viral concentrations, often being missed by methods with lesser analytical sensitivity. Although the infectious dose for SARS-CoV-2 is currently unknown, a low viral concentration in respiratory samples may represent poor swabbing technique of a potentially infectious patient. It is therefore imperative to be able to detect even a low viral concentration. For this, analytical sensitivity of SARS-CoV-2 PCR assays may be key. supplementary table 2 and 3 for oropharyngeal and nasopharyngeal specimens only, respectively. a The specimens used to determine the PPA of the GeneXpert and ARIES LDT were chosen to span a large range of viral concentrations, but intentionally overrepresented lower concentrations when compared to the distribution of patient viral concentrations. The PPA is reported as determined in the tested specimens; false negatives are projected by combining the relative frequency of a certain viral concentration with the pertinent assay PPA, they are displayed as rate of the entire patient population tested. Note the resulting difference in tested PPA and projected false negatives in the row describing all concentrations. b The upper limits of the displayed 95% confidence intervals for the GeneXpert's projected false negative rates are driven by the low sample sizes tested in each subset. As the GeneXpert detected all N=21 specimen with viral concentrations >100 copies/mL, it is appropriate to combine these subsets. This combined subset has a projected false negative rate of 0.00 (0.00-0.15), resulting in a projected false negative rate of 0.04 (0.00-0.21) overall. 

A Novel Coronavirus from Patients with Pneumonia in China

Virological assessment of hospitalized patients with COVID-2019

SARS-CoV-2 Viral Load in Upper Respiratory Specimens of Infected Patients

Transmission of 2019-nCoV Infection from an Asymptomatic Contact in Germany

Presymptomatic SARS-CoV-2 Infections and Transmission in a Skilled Nursing Facility

WHO Director-General's opening remarks at the media briefing on COVID-19 -16

Asymptomatic Transmission, the Achilles' Heel of Current Strategies to Control Covid-19

COVID-19: Nasal and oropharyngeal swab

Comparison of nasopharyngeal and oropharyngeal swabs for SARS-CoV-2 detection in 353 patients received tests with both specimens simultaneously

False negative rate of COVID-19 is eliminated by using nasal swab test

Potential false-negative nucleic acid testing results for Severe Acute Respiratory Syndrome Coronavirus 2 from thermal inactivation of samples with low viral loads

Stability issues of RT-PCR testing of SARS-CoV-2 for hospitalized patients clinically diagnosed with COVID-19

Cepheid Xpert Xpress SARS-CoV-2 Instructions for Use (XPRSARS-COV2-10

Luminex Provides Update on SARS-CoV-2 Validation Testing Efforts

ARIES Exo+ Ready Mix Product Sheet

Luminex Corporation. ARIES® Extraction Kit

RT-PCR Primer/Probes Panel

Interval Estimation for a Binomial Proportion

The Detection of SARS-CoV-2 using the Cepheid Xpert Xpress SARS-CoV-2 and Roche cobas SARS-CoV-2 Assays

Comparison of Abbott ID Now and Abbott m2000 methods for the detection of SARS-CoV-2 from nasopharyngeal and nasal swabs from symptomatic patients