key: cord-0329198-y70gbu4w
authors: Deerain, J. M.; Tran, T.; Batty, M. B.; Yoga, Y.; Druce, J.; Mackenzie, C.; Taiaroa, G.; Taouk, M.; Chea, S.; Zhang, B.; Prestedge, J.; Ninan, M.; Carville, K.; Fielding, J.; Catton, M.; Williamson, D. A.
title: Assessment of twenty-two SARS-CoV-2 rapid antigen tests against SARS-CoV-2: A laboratory evaluation study
date: 2021-12-16
journal: nan
DOI: 10.1101/2021.12.15.21267691
sha: b6c1dcd5b10792c796cffa3511eb82bc27fe08aa
doc_id: 329198
cord_uid: y70gbu4w

Background Rapid antigen testing is widely used as a way of scaling up population-level testing. To better inform antigen test deployment in Australia, we evaluated 22 commercially available antigen tests, including an assessment of culture infectivity. Methods Analytical sensitivity was evaluated against SARS-CoV-2 B.1.617.2 (Delta), reported as TCID50/mL, cycle threshold (Ct) value and viral load (RNA copies/mL). Specificity was assessed against non-SARS-CoV-2 viruses. Clinical sensitivity and correlation with cell culture infectivity was assessed using the Abbott PanBio COVID-19 Ag test. Results Nineteen kits consistently detected SARS-CoV-2 antigen equivalent to 1.3x10^6 copies/mL (5.8x10^3 TCID50/mL). Specificity for all kits was 100%. Compared to RT-PCR the Abbott PanBio COVID-19 Ag test was 52.6% (95% CI, 41.6% to 63.3%) sensitive, with a 50% detection probability for infectious cell culture at 5.9 log10 RNA copies/mL (95% CI, 5.3 to 6.5 log10 copies/mL). Antigen test sensitivity was 97.6% (95% CI, 86.3% to 100.0%) compared to positive infectious in cell culture. Conclusions Antigen test positivity correlated with positive viral culture, suggesting antigen test results may determine SARS-CoV-2 transmission risk. Sensitivity varied considerably between test kits and highlights the need for ongoing systematic post-market evaluation, providing valuable information to help guide antigen test selection and deployment.

The global COVID-19 pandemic has profoundly altered the way that societies live and function. To date, a combination of vaccination, local and national lockdowns, and contact tracing ('test, trace, isolate and quarantine') has been used to mitigate the transmission of SARS-CoV-2, with reverse-transcriptase polymerase chain reaction (RT-PCR) testing forming the cornerstone of testing in many countries (1, 2) . In addition to molecular detection of SARS-CoV-2, point-of-care rapid antigen tests have been used to complement laboratory-based PCR testing. Deployment of rapid antigen testing has been suggested as a potential means of upscaling population-level testing to facilitate the safe re-opening of societies, and to enable access to COVID-19 testing in resource-poor settings (3) (4) (5) (6) . In theory, large-scale testing of asymptomatic populations using rapid antigen tests (in conjunction with other public health measures) could detect infectious individuals with pre-symptomatic or asymptomatic COVID- 19 and rapidly interrupt transmission networks and is a conceptually attractive option for facilitating a 'COVID Normal' return to daily activities.

Because COVID-19 antigen tests detect viral protein (usually nucleocapsid) rather than amplified nucleic acid, they are inherently less sensitive than RT-PCR assays (7) . However, their advantages include rapid results, portability and reduced cost compared to RT-PCR.

Further, recent studies suggest that detection of viral antigen may correlate with higher viral load and hence, a greater likelihood of cell culture infectivity (8, 9) . Although higher viral load does not necessitate a greater probability of infectiousness, regular antigen testing may serve to identify high risk individuals who are infectious and should be quarantined (10) .

Australia has one of the highest per capita SARS-CoV-2 RT-PCR testing rates in the world; these high rates of testing have, along with major public health interventions, contributed to a . CC-BY-NC-ND 4.0 International license It is made available under a perpetuity.

is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint 

A laboratory evaluation study was conducted at the Victorian Infectious Diseases Reference Laboratory (VIDRL) at The Peter Doherty Institute for Infection and Immunity (Melbourne, Australia). VIDRL is the public health virology laboratory for the state of Victoria, serving a population of approximately 6.24 million people. The study was designed in two parts: (i) an evaluation of analytical sensitivity and specificity of 22 antigen tests, including two saliva antigen tests and two fluorescent immune assays (Supplementary Table 1 is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint 

To evaluate analytical sensitivity and specificity, we used a dilution series of a representative isolate of the Delta variant currently circulating in Victoria (VIC/18440/2021). SARS-CoV-2 lineage classification was based on previously described methods (13) . Virus was grown in Calu-3 cells (Supplementary Appendix) and inactivated by a 50kGy dose of gamma radiation.

Gamma irradiation is commonly used in virology to inactivate viruses for subsequent use in the development and evaluation of laboratory assays, whilst preserving the structural integrity of surface antigens (14) . To confirm that gamma irradiation had no impact on antigen kit evaluation, we compared the analytical sensitivity of an antigen test kit pre-and post-irradiation using VIC/18440/2021. Viral load was quantified by an in-house assay for the N-gene, described previously (15) , and by droplet digital PCR (ddPCR; BioRad QX200) (Supplementary Appendix). Infectious virus was also quantified prior to gamma-irradiation by Table 2 ). Based on manufacturer's reported sensitivities for assessed antigen test kits, we estimated that the limit of detection (LoD) for all devices would fall between 3 x 10 1 and 2 x 10 3 TCID50/mL. Samples with a Ct value between 21 and 23 (2 x 10 4 and 1 x 10 5 TCID50/mL, and > 33 (≥2 x 10 1 TCID50/mL) were assessed in duplicate, while samples with a Ct between 24 and 32 were assessed in quadruplicate. is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint For the twenty nasal/nasopharyngeal antigen tests, testing was performed according to the manufacturers provided instruction for use (IFU) for assessing swabs eluted in VTM. Where no specific protocol was provided, a 1:1 dilution of sample to extraction buffer was prepared and added to the test kit using provided plasticware, as per manufacturer's instructions (Supplementary Table 1 ).

To assess analytical specificity, panels of 'distractor' viruses commonly found in respiratory samples were prepared using ten stored reference isolates from VIDRL: Table 3 ). Each sample was tested in duplicate (n=20 samples) using the same procedure as the sensitivity panels described above.

Testing duration and interpretation of antigen test results were performed as per manufacturer's instructions. Results of all tests were recorded independently by two scientists. Results were read by a third independent scientist where there was discordance between initial readings.

Clinical sensitivity and correlation with cell culture infectivity was assessed using the Abbott PanBio COVID-19 Ag test (Abbott Rapid Diagnostics Jena GmbH, Germany). This device was specifically chosen because clinical studies using this kit are ongoing in our setting.

Residual VTM from combined naso-oropharyngeal swabs that were submitted to VIDRL for SARS-CoV-2 RT-PCR testing between 01/07/2021 and 22/08/2021 were stored at 4 o C up to 1

week prior to testing. Samples were not subjected to a freeze-thaw cycle to facilitate . CC-BY-NC-ND 4.0 International license It is made available under a perpetuity.

is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint

The copyright holder for this this version posted December 16, 2021. ; https://doi.org/10.1101/2021.12.15.21267691 doi: medRxiv preprint preservation of virus for cell culture infectivity assessment. All samples reported as detected for SARS-CoV-2 and with sufficient volume remaining were included and assessed for the following: (i) antigen test reactivity; (ii) cell culture infectivity, and (iii) viral RNA load quantification using a quantitative standard curve (Supplementary Appendix). Lineage designation of SARS-CoV-2 was performed using previously described methods (13) . Clinical specificity of the LFD was assessed using residual VTM from 100 SARS-CoV-2 RT-PCR negative combined naso-oropharyngeal swabs.

SARS-CoV-2 RT-PCR positive clinical samples were assessed for antigen test reactivity using the Abbott PanBio TM COVID-19 Ag Rapid Test Device (Nasal) (Abbott 41FK11). Samples were prepared and tested according to methods described in the 'Guidance For Use of Alternative Protocol' documentation provided by Abbott. Samples were diluted 1:1 in 150 µL extraction buffer and 130µL of the mixed solution added to the rapid test device. Results were recorded between 15-20 minutes according to manufacturer's IFU by two independent scientists. Results were read by a third independent scientist where there was discordance between initial readings.

Linear regression was performed to determine the relationship between N-gene RT-PCR Ct value and either SARS-CoV-2 viral load (reported as RNA copies/mL) or log10 TCID50/mL. Analytical sensitivity was defined as all samples in a dilution series positive for SARS-CoV-2 antigen, except in the logistic regression analysis, in which each sample was treated as an independent positive (*1*) or negative (*0*) result. These regression models were used to determine the 50% antigen test detection rate and 95% confidence intervals (CI) at a given viral load (log10 RNA copies/mL). Median value (and corresponding 95% CI) comparing clinical . CC-BY-NC-ND 4.0 International license It is made available under a perpetuity.

is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint

The copyright holder for this this version posted December 16, 2021. ; https://doi.org/10.1101/2021.12.15.21267691 doi: medRxiv preprint sensitivity and specificity of antigen test positivity to culture infectivity and N-gene RT-PCR Ct values were generated using the Wilson-Brown method. Analysis was performed using GraphPad Prism (v 9.0) and Stata, and data visualisation was performed using GraphPad Prism (v 9.0) and ggplot (v3.3.5) in Rstudio (v1.4.1717) (16) .

This study was approved by the Royal Melbourne Hospital Research Ethics Committee (QA2020085).

Overall, 22 rapid antigen kits were assessed for LoD and specificity (Table 1) . Nineteen kits consistently detected SARS-CoV-2 antigen at dilutions equivalent to 5.8 x 10 3 TCID50 /mL (1.3

x 10 6 copies/mL). Six kits consistently detected antigen in samples at a dilution equivalent to 7.2 x 10 2 TCID50 /mL (1.6 x 10 5 RNA copies/mL), and a further six kits showed a limit of detection at 3.6 x 10 2 TCID50/mL (8.3 x 10 4 RNA copies/mL) ( Table 1) . Linear regression analysis confirmed a significant agreement (R 2 = 0.9796, P <0.0001) between N-gene Ct and ddPCR-quantified viral load (log10 RNA copies/mL) for each sample dilution (Supplementary Figure 2) .

Logistic regression analysis demonstrated that eleven kits had a 50% detection probability for viral loads >10 6 RNA copies/mL, and seven kits had 50% detection probabilities >10 5 RNA copies/mL, equivalent to approximately 3.7 x 10 3 TCID50/mL (Figure 1 and Supplementary   Figure 1) . When compared to the manufacturer's stated sensitivity in the accompanying IFU documentation, 12 kits were less sensitive, although of these, five kits were within a log10 is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint

The copyright holder for this this version posted December 16, 2021. ; https://doi.org/10.1101/2021.12.15.21267691 doi: medRxiv preprint difference (Table 1 ). Three kits provided a limit of detection in ng/mL for recombinant antigen or PFU/mL and therefore could not be compared. None of the kits displayed cross-reactivity with other respiratory viruses (Supplementary Table 4 ) and no difference in analytical sensitivity was observed between SARS-CoV-2 samples pre-and post-gamma irradiation (Supplementary Table 5 ). Logistic regression demonstrated a 50% detection probability of cell culture infectivity at a Ct value of 26.6 (95% CI, 24.8 to 28.5) and viral load of 5.9 log10 RNA copies/mL (95% CI, 5.3 to 6.5 log10 RNA copies/mL) ( Figure 2 ). Viral loads were significantly higher in positive cell cultures and antigen test positive samples compared to culture and antigen test negative samples ( Figure 2 ). The highest Ct value for a positive sample by both antigen test and recoverable infectious virus was 33.4 (Supplementary Table 7) .

To assess clinical specificity, we tested 100 SARS-CoV-2 negative samples. None of these 100 samples tested positive using the LFD, resulting in 100% specificity compared with RT-PCR (95% CI 96.3% -100%). is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint 

Here, we provide information on the comparative analytical performance of twenty-two commercially available LFDs in Australia. As the use of antigen tests increases in Australia, including recent legislative changes allowing home-based antigen testing for COVID-19, knowledge of the performance characteristics is required to help public health agencies, laboratories and the public inform decisions around the selection and implementation of these tests.

All tests were able to detect 1. pooled clinical specimens (15) . Similar to our findings, these authors demonstrated that the most sensitive kits were able to detect approximately 7.5 x 10 4 RNA copies/mL, and the least sensitive kit approximately 2.3 x 10 7 RNA copies/mL (15) . In our study, detection limits ranged from approximately 8.3 x 10 4 (six kits) to 2.6 x 10 6 RNA copies/mL (all 22 kits). Further,

were able to detect between 2.07 x 10⁶ and 2.86 x 10⁷ RNA copies/mL (18) . Importantly however, studies reporting analytical sensitivity of LFDs vary across several parameters, including sample matrix (e.g. VTM or saline) (19) , antigenic source (e.g. recombinant antigen; virus inactivation method; swab type) (20) ; reporting format (e.g. PFU/mL; TCID50; copies/mL) and different SARS-CoV-2 variants (19, 20) . Here, we correlate antigen test is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint

The copyright holder for this this version posted December 16, 2021. ; https://doi.org/10.1101/2021.12.15.21267691 doi: medRxiv preprint positivity with: (i) Ct values; (ii) RNA copies/mL; and (iii) TCID50/mL to provide a systematic comparison of quantitative laboratory parameters. Our finding of differential analytical sensitivity of antigen kits using standardised panels reiterates the need for ongoing post-market validation data, including regular assessment of batch variation, to help guide the deployment of these tests.

Similar to other studies (21), we found high analytical specificity of antigen tests, with no cross-reactivity for any antigen tests against a panel of other respiratory viruses. Specificity was also high (100%) for the Abbott PanBio TM COVID-19 Ag test kit against clinical samples, in keeping with other findings, and with WHO recommendations of a minimum specificity of ≥97% for rapid antigen tests (7, 17) . Importantly however, even with high specificity, the positive predictive value of antigen testing will be low when there is minimal community transmission; in these circumstances, confirmatory RT-PCR testing is strongly recommended

Previous work has demonstrated a correlation between respiratory tract specimens with higher SARS-CoV-2 viral loads and infectivity in cell culture (8, 23, 24, 25) . For example, Pickering et al. found that approximately 95% of clinical samples that yielded a positive viral culture had a Ct value of less than 25 (8) . Further, when culture positivity was used as the reference standard, rather than RT-PCR, the sensitivity of antigen testing increased to at 94.7% (8) .

Similarly, Pekosz et al., found that antigen testing (using the BD Veritor system) demonstrated a higher positive percent agreement (PPA) (~90%) than RT-PCR (~70%) when using positive viral culture as the reference gold standard (26). In our study, 40/41 (98%) of samples with a positive viral culture also had a positive antigen test, with 95% of these samples having a Ct of less than 30. However, we also observed 8/40 (20%) samples that had a positive viral culture . CC-BY-NC-ND 4.0 International license It is made available under a perpetuity.

is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint

The copyright holder for this this version posted December 16, 2021. A limitation of our study, and indeed a limitation of several other laboratory studies assessing antigen test kits, include the use of spiked virus in VTM for kit evaluation, rather than the use of swabs directly placed in manufacturer-provided buffer. However, to try and maximise yield from samples, we tested samples without a freeze-thaw step, as freezing has been shown to reduce yield of infectious virus (18, 28) . Although we assessed the analytical characteristics of over twenty assays, a further limitation is that we only evaluated the clinical sensitivity of one antigen test, namely the Abbott PanBio TM COVID-19 Ag test assay. However, recent work has highlighted the widespread use of this kit, with 39 published datasets utilising the Abbott PanBio TM COVID-19 Ag test assay (21), meaning that our findings will have broad applicability and relevance.

In summary, our data describe the performance characteristics of 22 antigen test kits against the SARS-CoV-2 Delta variant using a standardised evaluation panel. We demonstrate marked variability between test kits, and variability between reported and observed sensitivities, is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint

The copyright holder for this this version posted December 16, 2021. ; https://doi.org/10.1101/2021.12.15.21267691 doi: medRxiv preprint although most (86.4%) were able to meet WHO recommended minimum standards for detection. In addition, we further corroborate the hypothesis that antigen test positivity generally correlates with positive viral culture and by extension with presence of infectious viral particles) and that a positive antigen test result could be used as an adjunct to determine SARS-CoV-2 transmission risk. Collectively, our data provide valuable information to help guide antigen test selection and deployment and highlight the need for ongoing systematic postmarket evaluation of antigen test kits, ideally using standardised reagents and protocols.

This work was supported by the Department of Health Victoria, Australia. We thank the study participants for their contributions. is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint

The copyright holder for this this version posted December 16, 2021. ; https://doi.org/10.1101/2021.12.15.21267691 doi: medRxiv preprint

Inferring change points in the spread of COVID-19 reveals the effectiveness of interventions

Estimating the effects of non-pharmaceutical interventions on COVID-19 in Europe

Clarifying the evidence on SARS-CoV-2 antigen rapid tests in public health responses to COVID-19

Rethinking Covid-19 Test Sensitivity -A Strategy for Containment

COVID-19: Rapid antigen detection for SARS-CoV-2 by lateral flow assay: A national systematic evaluation of sensitivity and specificity for mass-testing

Antigen-detection in the diagnosis of SARS-CoV-2 infection: Interim guidance

Rapid, point-of-care antigen and molecular-based tests for diagnosis of SARS-CoV-2 infection

Comparative performance of SARS-CoV-2 lateral flow antigen tests and association with detection of infectious virus in clinical specimens: a single-centre laboratory evaluation study

Performance characteristics of the

Abbott BinaxNOW SARS-CoV-2 antigen test in comparison to real-time RT-PCR and viral culture in community testing sites during

Test sensitivity is secondary to frequency and turnaround time for COVID-19 screening

Coronavirus Pandemic (COVID-19)

Genomics-informed responses in the elimination of COVID-19 in Victoria, Australia: an observational, genomic epidemiological study

Gamma Irradiation as an Effective Method for Inactivation of Emerging Viral Pathogens

Establishment of a specimen panel for the decentralised technical evaluation of the sensitivity of 31 rapid diagnostic tests for SARS-CoV-2 antigen

Elegant Graphics for Data Analysis

Technical specifications for selection of essential In vitro diagnostics for SARS-CoV-2

CoV-2 rapid point-of-care antigen tests: a single-centre laboratory evaluation study

Limit of detection in different matrices of 19 commercially available rapid antigen tests for the detection of SARS-CoV-2

The impact of sample processing on the rapid antigen detection test for SARS-CoV-2: Virus inactivation, VTM selection, and sample preservation

SARS-CoV-2 Variants with T135I

Nucleocapsid Mutations may Affect Antigen Test Performance

A SARS-CoV-2 Nucleocapsid Variant that 22

Multi-site assessment of rapid, point-of-care antigen testing for the diagnosis of SARS-CoV-2 infection in a low-prevalence setting: A validation and implementation study

Predicting infectious SARS-CoV-2 from diagnostic samples

Duration and key determinants of infectious virus shedding in hospitalized patients

SARS-CoV-2 infectivity correlates with high viral loads and detection of viral antigen and is terminated by seroconversion

Antigen-Based Testing but Not Real-Time Polymerase Chain Reaction Correlates With Severe Acute Respiratory Syndrome Coronavirus 2 Viral Culture

In Vitro Rapid Antigen Test Performance with the SARS-CoV-2 Variants of Concern B.1.1.7 (Alpha), B.1.351 (Beta), P.1 (Gamma), and B.1.617.2 (Delta)

Factors that Influence the Reported Sensitivity of Rapid Antigen Testing for SARS-CoV-2

Genomics-informed responses in the elimination of COVID-19 in Victoria, Australia: an observational, genomic epidemiological study

Tracking the COVID-19 pandemic in Australia using genomics

Virus isolation and quantitation

is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprintThe copyright holder for this this version posted December 16, 2021. ; https://doi.org/10.1101/2021.12.15.21267691 doi: medRxiv preprint