key: cord-0740308-0qfup7on
authors: Yang, Jing; Han, Yanxi; Zhang, Runling; Zhang, Rui; Li, Jinming
title: Comparison of analytical sensitivity of SARS-CoV-2 molecular detection kits
date: 2021-08-21
journal: Int J Infect Dis
DOI: 10.1016/j.ijid.2021.08.043
sha: 7a5251af800373b51566afb034a62b90cc6d843f
doc_id: 740308
cord_uid: 0qfup7on

OBJECTIVES: Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection has significantly impacted the global public health system, making nucleic acid detection an important tool in epidemic prevention and control. Detection kits based on real-time reverse transcriptase polymerase chain reaction (rRT-PCR) have been widely used in clinics, but their analytical sensitivity (limit of detection) remains controversial. Moreover, there is limited research evaluating the analytical sensitivity of other molecular detection kits. METHODS: In this study, we used self-developed armored ribonucleic acid reference materials to evaluate the analytical sensitivity of SARS-CoV-2 detection kits approved by the National Medical Products Administration. These were based on rRT-PCR and other molecular detection assays. RESULTS: The percentage re-testing required with rRT-PCR kits is as follows: 0%, 7.69%, 15.38%, and 23.08% for samples with concentrations ranging from 50,000 to 781 copies/mL. In total, 93% of rRT-PCR kits had a limit of detection (LOD) <1000 copies/mL. Only one kit had an LOD >1000 copies/mL. The LOD of other molecular detection kits ranged from 68 to 2,264 copies/mL. CONCLUSIONS: Our findings can help pharmaceutical companies optimize and improve detection kits, guide laboratories in selecting kits, and assist medical workers in their daily work.

The outbreak of coronavirus disease 2019 , caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), was first reported in December 2019. To date, more than 207 million people have been infected with SARS-CoV-2, and over 4 million people have died from COVID-19 globally (WHO, 2021) .

Asymptomatic infection in individuals has accelerated the transmission of the virus, making containment and mitigation difficult (Han et al., 2020) . To prevent the situation from worsening, governments need to implement stronger prevention and control strategies to test and track patients, suspected cases, and asymptomatic cases.

Currently, the diagnostic methods for COVID-19 include not only traditional molecular tests, serology tests, and computed tomography but also field-effect transistor-based sensing and plasmonic sensing as well as the use of high-throughput sensors (Taleghani N et al., 2021 , Seo G et al., 2020 , Ahmadivand A et al., 2021 .

Among these, molecular detection is recommended as the gold standard for SARS-CoV-2 detection (Green et al., 2020) . Real-time reverse transcriptase (rRT) polymerase chain reaction (PCR) is a robust technology with a high specificity and sensitivity (Yuce et al., 2021) . However, false-negative results from rRT-PCR have substantial risks and public health implications. Many factors can lead to false-negative results, especially low viral loads (Yu et al., 2020 , Kucirka et al., 2020 , Wikramaratna et al., 2020 . Accurate detection can help effectively identify infected 5 individuals and limit the spread of the virus. Therefore, it is necessary to improve the analytical sensitivity to ensure the accuracy and reliability of the test results. The limit of detection (LOD) is the lowest concentration of SARS-CoV-2 RNA at which the positivity rate of the detection kit is ≥95%, known as the analytical sensitivity. It is an important performance parameter for evaluating a detection kit. At the beginning of the pandemic, only a few kits were evaluated for detection performance (Alcoba-Florez et al., 2020 , Wang B. et al., 2020 , Wang X. et al., 2020 , Yan et al., 2021 . Currently, the number of kits approved by the NMPA has increased significantly, and previously approved kits have been optimized. Notably, some of these kits are CE-certified and FDA-approved for use in various countries. In addition, the evaluation of detection kits has mainly focused on the rRT-PCR method, with little focus on other molecular detection assays.

Therefore, this study aimed to evaluate the analytical sensitivity of kits currently approved by the NMPA, including those using rRT-PCR and those using other molecular detection techniques, to assess whether a SARS-CoV-2 detection kit meets the LOD claimed by their manufacturers and to provide a theoretical basis for laboratories in selecting kits. 6 As shown in Figure 1a , SARS-CoV-2 ORF1a, RdRP, ORF1b/S, and N/E virus-like particles (VLPs) were prepared in our laboratory and used as evaluation samples for this study. Four gene fragment sequences were synthesized (GENEARY, Shanghai, China). These fragments were cloned into the pACYC-MS2 vector, respectively. The recombinant plasmid was transformed into BL21 (DE3) cells, and protein expression was induced using isopropyl-β-D-thiogalactopyranoside. The VLPs were purified and digested with DNase and RNase to remove free nucleic acids from the surface. Specific primers and probes were used to identify the four target sequences contained in the VLPs by quantitative PCR (qPCR) and were quantified by droplet digital PCR (Supplementary Material Table S1 ). According to the results, four VLPs were mixed in equal proportions at a concentration of 1×10 8 copies/mL.

The four-fold serially diluted mixture was used as the evaluation sample at concentrations of 50, 000, 12,500, 3,125, 781, 195, 49, and 12 .25 copies/mL using Dulbecco's Modified Eagle Medium. The samples were subpackaged into small tubes and stored in a refrigerator at 80°C until testing.

The LODs of the SARS-CoV-2 detection kits approved by the NMPA were evaluated in this study. The 13 commercial detection kits for rRT-PCR were as follows: Sansure, Da an (Da an 1 and Da an 2), BioGerm, Liferiver, Maccura, EasyDiagnosis, Bioperfects, Applied Biological, Fosun Long March, Kinghawk, Geneodx, and BGI. The following five kits based on other molecular detection assays were also evaluated in this study: CapitalBio, Rendu, Zhongzhi 1, Zhongzhi 2, and Anbio. Evaluation of the 13 kits using the rRT-PCR method was performed in our laboratory. Biopharmaceutical companies were entrusted to complete the evaluation of the remaining five detection kits.

Da an has two rRT-PCR kits approved by the NMPA, namely, Da an 1 and Da an 2. Although the detection principles of these two kits are the same, the RNA preparation method, thermal cycling conditions, and cycling number of these two kits are different. The method used by Anbio is a hybrid capture immunofluorescence assay. Zhongzhi 1 uses a dual amplification assay method, whereas Zhongzhi 2 uses a ribonucleic acid (RNA) isothermal amplification gold probe chromatography assay.

The method used by Rendu is an RNA capture probe assay. CapitalBio uses an integrated isothermal amplification chip assay.

As shown in Figure 1b , the rRT-PCR evaluation samples were grouped according to seven concentrations: 50, 000, 12,500, 3,125, 781, 195, 49, and 12.25 copies/mL. Each concentration group included 21 samples. For each detection kit, there were 2 negative and 147 positive samples. The procedure was performed according to the manufacturer's instructions. RNA was obtained by extraction and purification of nucleic acid, or lysis release. The viral RNA extraction kit (QIAGEN, Shanghai, China) was used for RNA extraction and purification in rRT-PCR-based kits, except in Da an 2, for which the lysis method for RNA preparation was used.

Results were interpreted according to the manufacturer's instructions (Supplementary Material Table S6 ). In cases in which false positivity was suspected, the sample was re-tested. The results of the re-test were interpreted according to the interpretation criteria of re-test result.

As shown in Figure 1b , five detection kits were tested by biopharmaceutical companies. All companies were provided evaluation samples with concentrations of 50,000, 12,500, 3,125, 781, 195, and 49 copies/mL. Each concentration group included 21 samples. There were 3 negative and 126 positive samples. The order of all samples was disrupted before handing them to the biopharmaceutical company, and the samples were transported through cold-chain transportation to maintain a low-temperature environment. The evaluation samples were tested according to the manufacturer's instructions. All companies were required to report results within a week.

As shown in Figure 1c , probit regression analysis was used to evaluate the LOD of the test results on MedCalc Statistical Software version 19.6.1 (MedCalc Software Ltd, Ostend, Belgium). Pearson's chi-squared test was used to evaluate the positivity rate of target genes using SPSS version 19.0 software (SPSS Inc., Chicago, IL). 

Four VLPs were prepared, and specific probe primers for ORF1a, RdRP, ORF1b/S, and N/E were used to detect VLPs containing the target sequence Table S1 ). The four VLPs were quantified by droplet digital PCR using specific probe primers. The results showed ORF1a, RdRP, ORF1b/S, and N/E VLPs at concentrations of 5.44×10 11 , 1.273×10 12 , 4.48×10 11 , and 1.9×10 12 copies/mL, respectively (Supplementary Material Table S2 ). The four VLPs were diluted and mixed to form a high-concentration sample with a concentration of 1×10 8 copies/mL. Three commercial kits that could detect ORF1a, RdRP, ORF1b/S, and N/E fragments were used to confirm that all target sequences were present in the mixture (Supplementary Material Table 3, Table 4 , and Table   5 ).

Serially diluted samples were detected using commercial detection kits, and the results were presented as positive or negative according to the manufacturer's instructions. As described in some of the instructions, samples that were suspected to be positive in the first test were re-tested. The percentage of rRT-PCR kits that required re-testing was 0% (0/13), 7.69% (1/13), 15.38% (2/13), and 23.08% (3/13) for samples with concentrations ranging from 50,000 to 781 copies/mL, respectively.

With the decrease in sample concentration, the number of kits that required re-testing increased. Therefore, the percentage of rRT-PCR kits re-tested was 69.23% (9/13), 92.31% (12/13), and 61.54% (8/13) for sample with concentrations from 195 to 12.25 copies/mL, respectively (Table 2, Figure 2a) .

The re-test rate was calculated as follows: number of re-test/ number of replicates (re-test rate, %) ( Table 2) . The re-test rate of the Da an 2 detection kit was the highest. The re-test rates were 14.29% (3/21), 57.14% (12/21), 71.43% (15/21), 57.14% (12/21), 23.81% (5/21), and 9.52% (2/21) of the samples of concentrations 12,500, 3,125, 781, 195, 49, and 12 .25 copies/mL, respectively. The Maccura kit had the second highest re-test rate, which were as follows: 23.81% (5/21), 14.29% (3/21), and 71.43% (15/21) for concentrations of 3,125, 781, and 195 copies/mL, 11 respectively. For samples with low concentrations, the re-test rates were zero because all test results were considered negative. In Sansure and kits based on other molecular detection assays, re-testing of samples of each concentration was not required; therefore, the re-test rates of these kits were not calculated (Figure 2a) . 

All kits were designed to simultaneously detect different target genes. Hence, the positivity rate for each gene was calculated. The positivity rate was calculated as follows: number of positive/ number of replicates (positivity rate, %) ( Table 3) .

Except for BGI, the manufacturer's instructions did not indicate its target gene ( Figure 2b) . The positivity rate of the ORF gene in the Da an 2 kit was much higher than that of the N gene (10/21 vs. 19/21, Pearson's chi-squared test, P=0.003), which affected the positivity rate of the 781 copies/mL sample. In contrast, the difference in the positivity rates of the 195 copies/mL samples showed that the N and E genes were more sensitive than the ORF gene (17/21 vs. 21/21 vs. 8/21, P<0.001) in the Maccura kit. This phenomenon was more obvious in the low-concentration samples. However, Da an 1 showed similar positivity rates for the N and ORF genes (20/21 vs. 20/21) .

Probit regression analysis of the positivity rate of 147 samples was performed to obtain the LOD of the rRT-PCR kit ( 195-775 copies/mL), 331 (95% CI: 229-725 copies/mL), 392 (95% CI: 239-999 copies/mL), 439 (95% CI: 235-1,385 copies/mL), 480 (95% CI: 275-1,320 copies/mL), and 711 copies/mL (95% CI: 420-1,853 copies/mL), respectively. The LODs of Maccura, Genedox, and Da an 2 were 724 (95% CI: 478-1,776 copies/mL), 827 (95% CI: 432-28,953 copies/mL), and 1,217 copies/mL (95% CI: 756-3,039 copies/mL), respectively. The results of the probit regression analysis are shown in 

The LODs of kits using other detection methods were obtained to analyze the feedback from the biopharmaceutical companies. Similarly, probit regression analysis was performed on the positivity rate of 126 samples to obtain the LODs of other detection assay kits ( Table 3) . As shown in Figure 4 , the LOD of Zhongzhi 2 was 761 copies/mL (95% CI: 457-2,056 copies/mL); that of Anbio was 803 copies/mL (95% CI: 365-5,930 copies/mL); that of Zhongzhi 1 and CapitalBio was 820 copies/mL (95% CI: 477-2,365 copies/mL) and 2,264 copies/mL (95% CI: 1,204-7,041 copies/mL), respectively; and that of Rendu was 68 copies/mL. Notably, the Rendu kit reported 12 samples as positive, with a concentration of 49 copies/mL.

Other positive samples with other concentrations were also completely detected. Therefore, CI was not obtained after the probit regression analysis (Figure 4 ). 

During the process of RNA preparation, the volumes of the sample and elution buffer were different (Table 1) . They are summarized as follows: (1) 140 μL of sample for extraction and 60 μL of buffer for elution; (2) the volume of the sample is 200 μL; (3) the elution volume is 80 μL. This difference could affect the RNA concentration. For instance, RNA obtained from (2) was two-fold more concentrated than that from (3). Similarly, the volumes of the reaction system and RNA template were also different. The discrepancy in the amount of template may affect the detection performance of a kit (Wang B. et al., 2020) . The RNA templates of the kits tested were 2/5, 1/5, 1/2, and 1/3 of the total reaction system. The largest difference between them was up to 2.5-fold. Remarkably, the LOD of Da an 2 was much worse than that of Da an 1, and the method of RNA preparation may be the reason for this huge disparity. However, considering the differences in the reaction solutions of the two kits, further studies are needed to determine the effect of RNA extraction on kit detection performance.

Moreover, the interpretation criteria for these detection kits and the sensitivity of each target gene affected re-testing. The interpretation criteria consist of two parts, the first test result and the re-test result. In terms of the interpretation criteria for the first test result, the criteria for diagnosis of positivity in Da an 1 was more stringent than that in Sansure and Liferiver. A single positive result for the target gene needed to be re-tested in the case of a Da an 1 kit. This undoubtedly increased the re-test rate, especially when the target gene had poor sensitivity. The sensitivities of each target gene in the multiple rRT-PCR detection kits were not identical. Three studies reported that the primer and probe sets of the US Centers for Disease Control and Prevention (CDC) and WHO have different LODs (Afzal, 2020 , Nalla et al., 2020 , Vogels et al., 2020 . In this study, we calculated the positivity rates of the different target genes (ORF1ab, N, and E) of these kits (Figure 2b) . The performance of the N gene in the Da an 2 kit and the ORF gene in the Maccura and Bioperfect kits was poor, which may have been caused by technical deficiencies, including primer design, reagent instability, or inappropriate reagent ratio (Wang X. et al., 2020) . Therefore, Da an 2 and Maccura kits have a high re-testing rate for high-concentration samples. A high re-testing rate delayed the test results, increased the workload of medical workers, and led to the requirement of more detection reagents. However, re-testing reminds medical workers to pay more attention to suspicious positive samples and avoid false-negative results.

Furthermore, the interpretation criteria for these detection kits and the sensitivity of each target gene also affect SARS-CoV-2 detection. Taking the interpretation criteria of re-test results as an example, the possibility of successful diagnoses of weakly positive samples is higher with the Da an 1 kit than with the EasyDiagnosis kit. The sensitivity of each target gene determines the sensitivity of the kit (Vogels et al., 2020 , Wan et al., 2020 . Additionally, primers are key to determining the sensitivity of target genes. The positivity rate of SARS-CoV-2 detection is related to the selection of target regions and amplification efficiency of the primers. These 16 results suggest that target gene selection, kit optimization, and primer validation are important for improving detection performance. The amplification of primers is greatly affected by characteristics such as internal stability, melting temperature, secondary structure, or mutual interference. Better-performing primers preferentially amplify the target fragments, leading to unbalanced amplification and even differences in LOD between different targets in multiple PCR reactions (Sint et al., 2012) . Hence, reducing the competition between primers and improving the efficiency of primer amplification is necessary for multiple rRT-PCR.

In contrast to the rRT-PCR method, other molecular detection assays use the isothermal amplification method, except the Anbio kit. The Anbio kit adopts a hybrid capture immunofluorescence method, which involves high speed, simple operation, and low requirement for personnel and equipment. This kit is suitable for communities and for other scenarios outside hospitals. The reverse transcription recombinase-aided amplification and loop-mediated isothermal amplification used by CapitalBio have higher requirements for instruments than other kits. Both Zhongzhi 1 and 2 use the transcription-mediated amplification method, but the detection methods for the amplified products are different. Compared to the operation process of Zhongzhi 2 that of Zhongzhi 1 is very complicated. The principle of Rendu is the same as that of Zhongzhi, but with a high analytical sensitivity. Among other molecular detection assays, only Rendu and Zhongzhi 2 reached their claimed detection limits.

In general, it is necessary to ensure that the amplification efficiency of primers for different target genes is similar and is >90% to reduce the discrepancy in the sensitivity of different target genes and reduce the re-test rate. The detection region of the target gene should be carefully selected to avoid competition between primers during the test. The kit instructions should clearly indicate the RNA extraction method and the corresponding kit. If the lysis method is used for RNA preparation, it should be indicated by the manufacturer of the sample preservation solution. All recommended kits, reaction reagents, and procedures in the instruction manual should be strictly verified. The kit should be fully verified with accurate quantitative international standard reference materials to determine the LOD of the kit. In the application process, weak positive quality control materials should be used to regularly evaluate the detection performance of the kit. Although we tried to perfect our research, there are limitations to this study. Compared with VLPs, real samples may contain more interference or inhibitory substances, and clinical samples should be used for sensitivity and specificity evaluations.

Although the measured LODs of several kits were inferior to those claimed, the analytical sensitivity of kits approved by NMPA can meet the need for COVID-19 diagnosis in clinics. Next, biopharmaceutical companies should focus on the pitfalls of their detection kits and improve the detection performance of the kits. Laboratories 18 need to emphasize the importance of quality control in daily work. All of these measures are essential for ensuring reliable test results. 

None declared. (0) 

Molecular diagnostic technologies for COVID-19: Limitations and challenges

Functionalized terahertz plasmonic metasensors: Femtomolar-level detection of SARS-CoV-2 spike proteins

Sensitivity of different RT-qPCR solutions for SARS-CoV-2 detection

Clinical Performance of SARS-CoV-2 Molecular Tests

COVID-19: Insight into the asymptomatic SARS-COV-2 infection and transmission

Variation in False-Negative Rate of Reverse Transcriptase Polymerase Chain Reaction-Based SARS-CoV-2 Tests by Time Since Exposure

Comparative Performance of SARS-CoV-2 Detection Assays Using Seven Different Primer-Probe Sets and One Assay Kit

Rapid Detection of COVID-19 Causative Virus (SARS-CoV-2) in

Human Nasopharyngeal Swab Specimens Using Field-Effect Transistor-Based Biosensor

Advances in multiplex PCR: balancing primer efficiencies and improving detection success

Diagnosis of COVID-19 for controlling the pandemic: A review of the state-of-the-art

Analytical sensitivity and efficiency comparisons of SARS-CoV-2 RT-qPCR primer-probe sets

Current practice and potential strategy in diagnosing COVID-19

Evaluation of seven commercial SARS-CoV-2 RNA detection kits based on real-time polymerase chain reaction (PCR) in China

Limits of Detection of 6 Approved RT-PCR Kits for the Novel SARS-Coronavirus-2 (SARS-CoV-2)

WHO COVID-19 Explorer. Geneva: World Health Organization

Estimating the false-negative test probability of SARS-CoV-2 by RT-PCR

Comparison of Seven Commercial Severe Acute Respiratory Syndrome Coronavirus 2 Nucleic Acid Detection Reagents with Pseudovirus as Quality Control Material

Quantitative Detection and Viral Load Analysis of SARS-CoV-2 in Infected Patients

COVID-19 diagnosis -A review of current methods

The author would like to thank agent manufacturers that provided the real-time RT-PCR kits for SARS-CoV-2 detection in this study.