key: cord-0773634-8fceyff0
authors: Cubas-Atienzar, Ana I.; Kontogianni, Konstantina; Edwards, Thomas; Wooding, Dominic; Buist, Kate; Thompson, Caitlin R.; Williams, Christopher T.; Patterson, Edward I.; Hughes, Grant L.; Baldwin, Lisa; Escadafal, Camille; Sacks, Jilian A.; Adams, Emily R.
title: Limit of detection in different matrices of 19 commercially available rapid antigen tests for the detection of SARS-CoV-2
date: 2021-09-15
journal: Sci Rep
DOI: 10.1038/s41598-021-97489-9
sha: 7a78a249f8761591b647bf1ba8d41b4025980116
doc_id: 773634
cord_uid: 8fceyff0

In the context of the coronavirus disease 2019 (COVID-19) pandemic there has been an increase of the use of antigen-detection rapid diagnostic tests (Ag-RDT). The performance of Ag-RDT vary greatly between manufacturers and evaluating their analytical limit of detection (LOD) has become high priority. Here we describe a manufacturer-independent evaluation of the LOD of 19 marketed Ag-RDT using live SARS-CoV-2 spiked in different matrices: direct culture supernatant, a dry swab, and a swab in Amies. Additionally, the LOD using dry swab was investigated after 7 days’ storage at − 80 °C of the SARS-CoV-2 serial dilutions. An LOD of ≈ 5.0 × 10(2) pfu/ml (1.0 × 10(6) genome copies/ml) in culture media is defined as acceptable by the World Health Organization. Fourteen of 19 Ag-RDTs (ActiveXpress, Espline, Excalibur, Innova, Joysbio, Mologic, NowCheck, Orient, PanBio, RespiStrip, Roche, Standard-F, Standard-Q and Sure-Status) exceeded this performance criteria using direct culture supernatant applied to the Ag-RDT. Six Ag-RDT were not compatible with Amies media and a decreased sensitivity of 2 to 20-fold was observed for eleven tests on the stored dilutions at − 80 °C for 7 days. Here, we provide analytical sensitivity data to guide appropriate test and sample type selection for use and for future Ag-RDT evaluations.

www.nature.com/scientificreports/ Ag-RDTs have been recently used for screening asymptomatic people in high prevalence areas and frontline workers to quickly identify persons with a SARS-CoV-2 infection to adapt infection prevention and control measures, thus preventing transmission in the community [18] [19] [20] . A mass testing program screening asymptomatic people in Slovakia using Ag-RDTs was shown to reduce the prevalence of SARS-CoV-2 infections by > 50% within 2 weeks 18 .

Despite the increased adoption of Ag-RDTs as an alternative of RT-PCR, independent analytical sensitivity data is currently lacking for many rapid antigen tests. Evaluation of Ag-RDTs using spiked samples in the laboratory before proceeding on clinical specimens is of paramount importance because the sensitivity of Ag-RDTs is highly variable depending on the manufacturer, ranging 0-95% in respiratory samples 7, [21] [22] [23] [24] .

Here we describe a single-center, manufacturer-independent analytical validation of 19 commercially available Ag-RDTs. The aims of the study were to assess the limit of detection (LOD) using viral culture in different sample matrices: direct culture supernatant, dry swab and swabs in Amies. The effect on the LOD of one freeze-thaw cycle following storage at − 80 °C was also explored; demonstration of adequate performance using this sample type could support future rapid evaluation of Ag-RDTs with stored material.

LOD using different matrices. The LOD was evaluated in three matrices: direct culture supernatant, dry swab and swab in Amies. Direct viral culture supernatant was used as it is the standardized protocol for the evaluation of LOD in Ag-RDTs 25 . Dry swab matrix using the proprietary swab kit was selected to evaluate the LOD in the sample type as defined in the instructions for use (IFU). Finally, swab in Amies was used to assess the use of the same swab used for RT-PCR as a sample type for Ag-RDTs.

A predefined performance criterion of an analytical LOD of ≤ 5.0 × 10 2 plaque forming units (pfu)/ml (≈ 10 6 genome copy numbers (gcn)/ml) using direct culture supernatant was selected based on current WHO and national standards 25, 26 . Fourteen of 19 Ag-RDTs evaluated in this study had an LOD of ≤ 5.0 × 10 2 pfu/ml (ActiveXpress, Espline, Excalibur, Innova, Joysbio, Mologic, NowCheck, Orient, PanBio, RespiStrip, Roche, Standard-F, Standard-Q and Sure-Status) using direct culture supernatant and the remaining had an LOD of 1.0-5.0 × 10 3 pfu/ml (Biocredit, Genedia, iChroma and Wondfo) (Fig. 1) . Espline, iChroma, Innova Panbio and Roche had the lowest LOD using direct culture supernatant reaching 0.5-1.0 × 10 2 pfu/ml, followed by ActiveXpress, Excalibur, Mologic, Orient, Standard-Q, Sure-Status and Respi-Strip, with an LOD of 2.5-5.0 × 10 2 pfu/ ml. Biocredit, Genedia, Standard-F and Wondfo and were the least sensitive with an LOD of 1.0-5.0 × 10 3 pfu/ ml. See Fig. 1 www.nature.com/scientificreports/ The LOD using dry swab was poorer in all tests when compared to direct culture supernatant with the exception of Mologic, where the LOD was the same using direct culture supernatant and dry swab. The LOD using dry swab was ≤ 5.0 × 10 2 pfu/ml only in four Ag-RDT (Espline, Mologic, Roche and Sure-Status), 1.0-5.0 × 10 3 pfu/ ml in seven (Bioeasy, Innova, NowCheck, Orient, Panbio, Respi-Strip and Standard-Q) and ≥ 1.0 × 10 4 pfu/ml in seven (ActiveXpress, Biocredit, Genedia, iChroma, Joysbio, Standard-F and Wondfo). The least sensitive using dry swab matrix was Joysbio with an LOD of 2.5 × 10 5 pfu/ml (Fig. 1) .

None of the Ag-RDTs evaluated specifically indicate compatibility with swabs in Amies media. However, four tests recommend the use of universal or viral transport media (UTM/VTM) (Biocredit, Respi-Strip, Roche and Wondfo), three tests do not recommend the use of UTM/VTM (NowCheck, Standard-F and Standard-Q), and the remaining kits do not mention the use of any transport media. Six Ag-RDTs (Excalibur, Joysbio, NowCheck, Orient, Sure-Status and Wondfo) were found to be incompatible with the Amies media, as these showed a positive test line with the negative control sample. Of these, Wondfo is the only kit which recommends the use of transport media. LODs using swabs in Amies media was poorer than using dry swabs except in two tests where the LOD was the same as with the dry swabs (PanBio and Standard-Q).

Effect of swab absorbance and volume of extraction buffer in the LOD of dry swabs. We investigated whether the absorbance of the proprietary swabs provided with the Ag-RDT kits affected the LOD compared with direct culture supernatant i.e. if a less absorbent swab resulted in a poorer LOD in dry swab compared with the LOD obtained in direct culture supernatant for the same test. We also investigated the effect of the volume of the extraction buffer as this varied per Ag-RDT kit i.e. if larger volumes of extraction buffer resulted in poorer LODs due to the dilution factor. The volume recovered by swabs per tests and volumes of extraction buffer per kit are shown in Supplementary Table S2 in Supplementary Information. Spearman's correlation coefficient did not show any statistically significant correlation between the LOD and the volume recovered by swabs (P = 0.421, ρ = − 0.50), volume of extraction buffer (P = 0.483, ρ = − 0.011) and a combination of both (P = 0.460, ρ = − 0.025).

LOD one freeze-thaw cycle after 7 days at − 80 °C. Ag-RDTs are intended to be POC tests and thus the majority are recommended for use with freshly collected specimens. To validate test performance, use of stored material is much easier. Therefore, we performed this experiment to understand whether LOD is impacted following sample storage. The LOD of each of the tests using SARS-CoV-2 serial dilutions after 7 days storage at − 80 °C and one freeze-thaw cycle is shown in Table 1 .

The LOD for 6 tests (ActiveXpress, Bioeasy, iChroma, NowCheck, RespiStrip and Wondfo) was equivalent using stored, frozen viral dilutions compared to fresh preparations. Three tests showed increased sensitivity (lower LOD by two-to five-fold) (Joysbio, PanBio and Standard-Q), and eleven showed poorer performance with a higher LOD of twofold (Biocredit, Espline, Roche and Sure-Status), 4-5 (Excalibur, Mologic and Orient), ten (Innova) and twenty (Genedia and Standard-F). www.nature.com/scientificreports/

Here, we present the analytical performance of 19 antigen rapid tests, which are currently on the market and in use in multiple countries. Analytical LODs are a useful proxy of clinical sensitivity, and the most standardized way to evaluate multiple antigen tests head-to-head, as each test requires a separate swab from patients. An approximate LOD of ≤ 5.0 × 10 2 pfu/ml (≈ 1.0 × 10 6 copies/ml) calculated using direct culture supernatant, has been proposed as the minimal analytical sensitivity by the WHO and the Department of Health and Social Care (DHSC, U.K.) 25, 26 . Fourteen of the 19 marketed Ag-RDTs evaluated in this study fulfill this requirement (ActiveXpress, Bioeasy, Espline, Innova, Mologic, NowCheck, PanBio, Excalibur, RespiStrip, Joysbio, Roche, Standard-F, Standard-Q, Sure-Status and Orient). Evaluation of the LOD using the kit-specific swabs immersed in the viral culture dilutions offers a more representative comparison to the level of sensitivity for clinical samples than using direct viral culture as these Ag-RDT kits are used by applying the kit-specific swabs to obtain a respiratory sample and the swab is immersed into the kit's extraction buffer. Four of the 19 tests detected samples with concentrations ≤ 5.0 × 10 2 pfu/ml (≈ 1.0 × 10 6 gcn/ml) when using the dry swabs (Mologic, Espline, Roche, Sure-Status) and none of the tests met that LOD target when using swabs in Amies, likely due to the dilution factor with the addition of 1 ml Amies buffer as well as potential chemical interactions between the media and the kit-specific buffers.

It is expected that the more absorbent swabs will absorb more viral material and the lower volume of extraction buffer will result in more concentrated sample within the test, and so we investigated if the volume recovered by the swab and the volume of extraction buffer had any bearing on the LOD of each test when compared with the LOD achieved with direct culture supernatant. We took into account the volume recovered from the swab and the volume of proprietary buffer provided, but no correlation was found, this suggests that other factors may reduce the sensitivity when using swabs such as differences in the formulation of the proprietary buffers and efficiency of sample recovery from the swab (in natural conditions). A reduction in sensitivity using clinical samples may be observed compared to swabs in viral culture, as clinical samples are more viscous than culture media, potentially resulting in less viral material being absorbed onto the swab. The efficiency of the recovery is also likely to be increased by the centrifugation method used in our protocol.

We also evaluated the LOD and compatibility of the Ag-RDTs using a swab placed in Amies media, as these are routinely used to collect upper-respiratory samples in SARS-CoV-2 suspected individuals for diagnosis using RT-PCR 27, 28 . If the same swab proves to be suitable for both RT-PCR and rapid antigen testing, one swab can be used for both tests as part of a serial algorithm. As well, the frozen leftover Amies media/swab from RT-PCR testing could be used for future Ag-RDT evaluations. Either approach could simplify clinical and/or evaluation workflows. None of the Ag-RDT manufacturers specifically recommend the use of Amies media, and we demonstrated here that this 'off-label' sample preparation should be used with caution: six tests had false positive results (Espline, Excalibur, Joysbio, Sure-Status and Orient) and sensitivity was also reduced due to the additional volume of Amies.

The effect of storage at − 80 °C and one freeze-thaw cycle was evaluated, with eleven Ag-RDTs showing a loss of sensitivity by up to 20-fold. A small decrease in sensitivity has been reported in in SARS-CoV-2 RT-PCR testing (< 1 RT-PCR cycle threshold) after one and two freeze-thaw cycles 29 but there are no studies so far that have reported the effect of freeze-thaw on antigen detection. Results here highlight that the use of frozen material with Ag-RDTs should be performed with caution. The fact that three tests showed two-to-five-fold better sensitivity after an additional freeze thaw-cycle could not be explained in here, a further investigation is required with a larger sample size to rule out whether this phenomenon was within the margin of error of the experiment.

Three out of 19 Ag-RDTs (Bioeasy, iChroma and Standard-F) rely on detection of a fluorescent signal using a reader. Though this may enable quantitative detection and potentially more consistent result interpretation, we did not find any improved sensitivity for this test format. Furthermore, Ag-RDTs that rely on a device may limit testing throughput if only one test can be read at a time. The reader also presents additional costs, as well as potential technical and maintenance issues which can be a barrier to implementation.

This analytical study has some limitations, as only a single isolate (REMRQ0001/Human/2020/Liverpool) was used to assess the LODs but our results are consistent with other recently-published analyses 30, 31 . To the authors knowledge, all 19 tests evaluated here detect the nucleoprotein, presumably chosen for abundance and relative low mutation rate and therefore hypothesized to pick up all currently known variants 32, 33 . Another limitation is that only one lot per kit was evaluated.

There is a growing number of studies suggesting that although antigen detection is less analytically sensitive than nucleic acid amplification techniques, it may strongly correlate with culturable virus, which may be a proxy for transmissibility. Hence Ag-RDTs could be informative for test, trace, isolate processes for the most infectious individuals [10] [11] [12] [13] [14] [15] . Viral loads have been estimated to range from 10 8 to 10 11 gcn/ml in the most infectious patients [34] [35] [36] . The majority of Ag-RDTs evaluated here have an LOD predicted to successfully diagnose infected individuals with higher viral loads in this range across all matrices, except Joysbio that had an LOD of 5.4 × 10 8 gcn/ml in dry swab. Further, Biocredit, iChroma, Standard-F and Genedia tests had LOD greater than 1.0 × 10 8 gcn/ml when using swabs placed in Amies.

In conclusion, the most sensitive tests with an LOD ≤ 5.0 × 10 2 pfu/ml (≈ 1.2 × 10 6 gcn/ml) on dry swabs and direct culture supernatant were Espline, Mologic, Sure-Status and Roche and the least sensitive on all matrices were Biocredit, iChroma, Standard-F and Genedia. The differences of LODs found here between tests and/or matrices ranged between 2-3 logs (i.e. 100-1000 fold). Some tests showed impaired performance when using freeze-thaw material and/or Amies media. These findings highlight the importance of understanding assay specific performances and the need to select the appropriate sample matrix and the right test for each intended use, particularly for laboratories and evaluation programs that seek a rapid validation of Ag-RDT using frozen stored samples and 'off-label' specimen preparations. The LODs obtained in this comprehensive assessment of www.nature.com/scientificreports/ analytical sensitivity are consistent with rapidly emerging clinical performance data 9,11,37 demonstrating the high clinical accuracy of Ag-RDTs for rapid detection of individuals with high viral loads, which can be very impactful for initiation of isolation and tracing measures.

Ethical statement. As the study involves only the data based on SARS-CoV-2 spiked samples and no clinical and tissue samples are used, ethical approval is not required for this study.

Evaluated Ag-RDTs. Nineteen Ag-RDT based on lateral flow principle were evaluated in this study ( www.nature.com/scientificreports/ SARS-CoV-2 serial dilutions and quantification of copy numbers. The SARS-CoV-2 isolate REMRQ0001/Human/2020/Liverpool was propagated in Vero E6 cells (C1008; African green monkey kidney cells), maintained in DMEM with 2% fetal bovine serum (FBS) and 0.05 mg/ml gentamycin. Ten-fold serial dilutions of SARS-CoV-2 stock were made starting from 1.0 × 10 6 pfu/ml to 1.0 × 10 2 pfu/ml using culture media as a diluent (DMEM with 2% FBS % and 0.05 mg/ml gentamycin). Two-fold dilutions were made below the ten-fold LOD dilution to refine the LOD. For quantification, viral RNA was extracted using QIAmp Viral RNA mini kit (Qiagen, Germany) according to the manufacturer's instructions. The genome copies/ml (gcn/ml) were calculated using the COVID-19 Genesig RT-qPCR kit (PrimerDesign, UK). RT-qPCR testing was carried out using the Rotor-Gene Q (Qiagen, Germany), with a ten-fold serial dilution of using quantified specific in vitro-transcribed RNA 38 . A total of five replicates were tested for each standard curve point and extracted RNA from each culture dilution was tested in triplicate, and the gcn/ml was calculated from the mean Ct value of these replicates.

Preparation of SARS-CoV-2 sample matrices and LOD testing protocol. Three types of sample matrices were tested (1) direct viral culture supernatant, (2) spiked dry swabs and (3) spiked wet swabs in Amies media.

For the direct viral culture matrix, a specific volume of the serial dilutions was added directly to the extraction buffers at a 1:10 ratio except for Respi-Strip which was added at 1:1 ratio with the extraction buffer following the IFU.

For dry swab testing, the proprietary nasopharyngeal (NP) or nasal (N) swabs included in each individual kit was used except for Respi-Strip, which does not include swabs, and the recommended Eswab (Copan, Italy) was used instead. To prepare the dry swab matrix, the swab was soaked in 1 ml of the virus culture dilution series for 6-8 s, followed by immersion in the prescribed amount of proprietary reaction buffer solution.

For the preparation of spiked wet swabs, Eswab in Amies media (Copan, Italy) was used across all tests. The swab was first immersed in the serial viral dilutions for 6-8 s, then placed into the Amies media to mimic the sample collection stage. Ag-RDTs were evaluated by then immersing the same swab into the extraction buffer, except for test Respi-Strip where 100 µl of the Amies was mixed at 1:1 with the extraction buffer following its IFU.

For all Ag-RDTs and matrices, the sample volumes applied, and procedures were performed as specified in the test specific IFUs.

The LOD was defined as the lowest dilution at which all three replicates were positive. Every dilution was tested in triplicate and non-spiked culture media and Amies were used as negative controls. Results were interpreted by two operators, each blinded to the result of the other. If a discrepant result was obtained, a third operator read any discrepant tests for a 2/3 result.

Effect of swab absorbance and volume of extraction buffer in the LOD of dry swabs. We investigated the effect of the absorbance of the proprietary swabs and extraction buffer provided with the Ag-RDT kit in the LOD using dry swab, i.e. if a less absorbent swab and larger volumes of extraction buffer resulted in a poorer LOD on dry swab compared with direct culture supernatant. To compare the effectiveness of each NP and N swab to recover sample, the amount of liquid absorbed by the swabs was measured. Five replicates of each swab brand were immersed in culture media for 6-8 s, then taped on the inside of a 50 ml centrifuge tube. These were then centrifuged for 5 min at 1000g and the amount of liquid released was measured using a micropipette.

The degree of correlation of the difference between LOD of dry swabs and direct culture for the same Ag-RDT with the volume recovered by swab type and volume of proprietary were investigated by Spearman's correlation coefficient rho (ρ). Statistical significance was set at P < 0.05. LOD after 7 days at -80 °C and one freeze-thaw cycle. After performing the LOD experiments, the viral culture dilutions were stored at − 80 °C for 7 days and then the LOD experiments were performed again using the dry NP and N swabs. This would help to assess the use of stored clinical samples could be used to facilitate evaluation of Ag-RDTs.

All data generated during this study is presented in an analysed format in this manuscript. Raw datasets are available from the corresponding author on reasonable request.

Diagnostic detection of 2019-nCoV by real-time RT-PCR

RNA Extraction Kits for COVID-19 Tests Are in Short Supply in US. The Scientist

Severe shortages of swabs and other supplies hamper coronavirus testing -CNN

Reagents hold up European COVID-19 tests

Impact of delays on effectiveness of contact tracing strategies for COVID-19: A modelling study

Antigen-detection in the diagnosis of SARS-CoV-2 infection using rapid immunoassays. Interim Guide

Rapid development of COVID-19 rapid diagnostics for low resource settings: Accelerating delivery through transparency, responsiveness, and open collaboration

Evaluation of the accuracy, ease of use and limit of detection of novel, rapid, antigen-detecting point-of-care diagnostics for SARS-CoV-2

Rapid antigen testing and non-infectious shedding of SARS-Cov2

Clinical evaluation of the Roche/SD Biosensor rapid antigen test with symptomatic, non-hospitalized patients in a municipal health service drive-through testing site

Rethinking Covid-19 test sensitivity-A strategy for containment

European Centre for Disease Prevention and Control. Options for the use of rapid antigen tests for COVID-19 in the EU/EEA and the UK

Antigen-based testing but not real-time PCR correlates with SARS-CoV-2 virus culture

Duration of infectiousness and correlation with RT-PCR cycle threshold values in cases of COVID-19

Laboratory diagnosis of SARS-CoV-2: Available approaches and limitations

Speed versus coverage trade off in targeted interventions during an outbreak

Covid-19: Mass testing in Slovakia may have helped cut infections

Mass testing for covid-19 in the UK

Covid-19: Government ramps up "Moonshot" mass testing

Evaluation of rapid antigen test for detection of SARS-CoV-2 virus

Clinical evaluation of self-collected saliva by quantitative reverse transcription-PCR (RT-qPCR), Direct RT-qPCR, reverse transcription-loop-mediated isothermal amplification, and a rapid antigen test to diagnose COVID-19

Low performance of rapid antigen detection test as frontline testing for COVID-19 diagnosis

Head-to-head comparison of four antigen-based rapid detection tests for the diagnosis of SARS-CoV-2 in respiratory samples

Protocol for evaluation of rapid diagnostic assays for specific SARS-CoV-2 antigens (lateral flow devices) GOV

Center of Disease Control and Prevention. Interim Guidelines for Collecting, Handling, and Testing Clinical Specimens for COVID-19

SARS-CoV-2 detection in setting of viral swabs scarcity: Are mrsa swabs and viral swabs equivalent

Influence of storage conditions on SARS-CoV-2 nucleic acid detection in throat swabs

Comparison of seven commercial SARS-CoV-2 rapid Point-of-Care Antigen tests

Evaluation of three rapid lateral flow antigen detection tests for the diagnosis of SARS-CoV-2 infection

Sars-cov-2 (Covid-19) by the numbers

The nucleocapsid protein of SARS-CoV-2: A target for vaccine development

Transmissibility of COVID-19 depends on the viral load around onset in adult and symptomatic patients

SARS-CoV-2 viral load in upper respiratory specimens of infected patients

SARS-CoV-2 detection, viral load and infectivity over the course of an infection

Performance evaluation of a SARS-COV-2 rapid antigentest: Test performance in the community in the Netherlands

Saliva offers a sensitive, specific and non-invasive alternative to upper respiratory swabs for SARS-CoV-2 diagnosis

The study and design of the study was conceived by A.I.C.A., J.S. and E.R.A. Laboratory testing was performed by A.I.C.A., K.K., T.E., C.W., D.W., K.B., C.T. Data analysis and interpretation were conducted by A.I.C.A. The initial manuscript was prepared by A.I.C.A. All authors edited and approved the final manuscript.

The study was supported by FIND. The funders of the study had no role in data collection and data analysis.

The authors declare no competing interests.

Supplementary Information The online version contains supplementary material available at https:// doi. org/ 10. 1038/ s41598-021-97489-9.Correspondence and requests for materials should be addressed to A.I.C.-A.Reprints and permissions information is available at www.nature.com/reprints.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http:// creat iveco mmons. org/ licen ses/ by/4. 0/.