key: cord-257140-ge15qrqg authors: Perkmann, T.; Perkmann-Nagele, N.; Ozsvar-Kozma, M.; Koller, T.; Breyer, M.-K.; Breyer-Kohansal, R.; Burghuber, O. C.; Hartl, S.; Aletaha, D.; Sieghart, D.; Quehenberger, P.; Marculescu, R.; Mucher, P.; Radakovics, A.; Strassl, R.; Leitner, G.; Wagner, O. F.; Binder, C. J.; Haslacher, H. title: Increasing both specificity and sensitivity of SARS-CoV-2 antibody tests by using an adaptive orthogonal testing approach date: 2020-11-07 journal: nan DOI: 10.1101/2020.11.05.20226449 sha: doc_id: 257140 cord_uid: ge15qrqg Background SARS-CoV-2 antibody tests have undergone a remarkable improvement in performance. However, due to the low seroprevalence in several areas, very high demands are made on their specificity. Furthermore, the low antibody-response in some individuals requires high test sensitivity to avoid underestimating true seroprevalence. Optimization of testing has been reported through lowering manufacturer cut-offs to improve SARS-CoV-2 assay sensitivity or by combining two tests to improve specificity at the cost of sensitivity. However, these strategies have thus far been used in isolation of each other. Methods To increase sensitivity, cut-offs of three commercially available SARS-CoV-2 automated assays (Roche, Abbott, and DiaSorin) were reduced according to published values in a pre-pandemic specificity cohort (n=1117) and a SARS-CoV-2 positive cohort (n=64). All three testing systems were combined in an orthogonal approach with a confirmatory test, which was one of the remaining automated assays or one of two commercial ELISAs directed against the spike protein receptor binding-domain (RBD) or the nucleocapsid antigen (NP). Results The modified orthogonal test strategy resulted in an improved specificity of at least 99.8%, often even 100%, in all 12 tested combinations with no significant decline in sensitivity. In our cohort, regardless of whether the assays were used for screening or confirmation, combining Roche and Abbott delivered the best overall performance (+~10% sensitivity compared to the single tests and 100% specificity). Conclusion Here we propose a novel orthogonal assay strategy that approaches 100% specificity while maintaining or even significantly improving the screening test's sensitivity. Serological tests detecting SARS-CoV-2 specific antibodies became available shortly after the beginning of the COVID-19 pandemic. Over the last few months, SARS-CoV-2 serology has experienced incredible diversity and technical development. Only weeks after the first ELISA protocol for SARS-CoV-2 antibody testing had been made publicly available (1), several fully automated serologic tests received emergency-use authorization by the FDA (2). It should be noted that these tests can perform very different diagnostically, which has been described in detail recently (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) (13) (14) (15) . Due to the continuing overall low seroprevalence of SARS-CoV-2 (16), the need for highly specific (≥99.5%) test systems to provide adequate positive predictive values has become evident (17) . Additionally, contrary to the manufacturers' data and the first publications on this subject, the test systems have not proven sufficiently sensitive (4-6, 11, 18, 19) . This discrepancy could be explained because the COVID-19 positive cohorts in the validation studies were mainly hospitalized patients and not cases with mild to moderate or even asymptomatic disease (20) . However, there is increasing evidence that, especially in non-hospitalized cases with RT-PCR confirmed SARS-CoV-2 infection, very low levels of specific antibodies can be observed frequently (21) (22) (23) (24) . Inadequate antibody responses can eventually lead to a negative result when the manufacturers' cut-offs are applied, resulting in suboptimal sensitivities of the test systems. of other infectious diseases and was recently also recommended by U.S. national public health officials to overcome the specificity issue. In the case of SARS-CoV-2, falsepositive samples are usually not simultaneously reactive in different test systems (4, 15) . Thus, an orthogonal test approach, in which positive results are confirmed with another test, can maximize specificity by excluding false-positive results. The disadvantage of this test strategy, which was recently shown in a study for the SARS-CoV-2 antibody tests from Abbott, Roche, and DiaSorin (19) , is that the gain in specificity usually comes at the expense of sensitivity and therefore increases the number of false negatives of individual tests. Thus, depending on the seroprevalence, undesired changes from over-to underestimating the actual number of SARS-CoV-2 seropositive samples may occur. To overcome this limitation, a fundamentally modified orthogonal test strategy was necessary: by maximally reducing the cut-offs of the individual tests, sensitivity was markedly increased, and consequently reduced specificity was compensated by orthogonal testing. In the present work, we show for the first time an orthogonal test algorithm based on real-life data for the SARS-CoV-2 antibody tests of Roche, Abbott, DiaSorin, and two commercial SARS-CoV-2 ELISAs modified according to Krammer et al. (27) intending to maximize both specificity and sensitivity at the same time. All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. In brief, the Roche Elecsys® Anti-SARS-CoV-2 assay detects total antibodies against the nucleocapsid (NC) antigen in a sandwich electrochemiluminescence immunoassay (ECLIA) on a cobas® e801 analyzer (Roche Diagnostics, Rotkreuz, Switzerland). According to the manufacturer, results exceeding 1.000 COI are considered positive. The Abbott SARS-CoV-2 assay detects IgG-antibodies against NC in a chemiluminescence microparticle assay (CMIA) on Abbott ARCHITECT® i2000sr platforms (Abbott Laboratories, Chicago, USA). The cut-off suggested by the manufacturer is 1.40 index (S/C). The DiaSorin LIAISON® SARS-CoV-2 S1/S2 IgG test detects IgG-antibodies against the S1/S2 domains of the virus' spike protein on LIAISON® XL analyzers (DiaSorin S.p.A., Saluggia, Italy) employing chemiluminescence immunoassays (CLIA). Results >12.0 AU/mL are considered All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted November 7, 2020. ; https://doi.org/10.1101/2020.11.05.20226449 doi: medRxiv preprint positive (the manufacturer suggests retesting of borderline results between 12.0 -15.0 AU/mL; however, this approach is not practicable when using historical samples). Commercially available ELISAs based on the protocols developed in the Krammer Lab (27) are directed against the nucleoprotein (NP) (Technozym® anti SARS-CoV-2 NP IgG) or the RBD (Technozym® anti SARS-CoV-2 RBD IgG) domain of the spike protein (Technoclone, Vienna, Austria). Calibrators and controls of both tests are traceable to a specific antibody against the SARS-CoV-2 receptor-binding domain (CR3022). These quantitative assays were processed manually according to the manufacturer's instructions. IgG-antibodies were quantified on a Filtermax F5 plate reader (Molecular Devices, San José, USA). The reduced cut-offs for these assays were modeled in-house and set at 4000 U/ml (data not shown). Statistics Potential orthogonal testing approaches were identified by a combination of a screening test performed on an automated platform (Roche, Abbott, or DiaSorin) and a confirmation test, which could be one of the remaining automated assays or an ELISA test (Technozym® NP or Technozym® RBD) that can be easily performed without additional large-scale equipment. As suggested in the literature, the cut-off levels for Roche, Abbott, and DiaSorin were reduced to increase sensitivities (3, 10, 31) . Values above the higher cut-off (i.e., no false positives above this range) were considered positive without additional confirmation tests. All results that fell between the lower cutoff value and the higher cut-off value were retested with one of the confirmation tests ( Figure 1 ). All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted November 7, 2020. ; https://doi.org/10.1101/2020.11.05.20226449 doi: medRxiv preprint Unless otherwise indicated, categorical data were given as counts (percentages), and continuous data were presented as median (interquartile range). 95% Confidence Intervals (CI) for sensitivities and specificities were calculated according to Wilson, 95% CI for predictive values were computed according to Mercaldo-Wald unless otherwise indicated. Sensitivities and specificities were compared by z-score testing. All calculations were performed using Analyse-it 5.66 (Analyse-it Software, Leeds, UK). All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted November 7, 2020. ; https://doi.org/10.1101/2020.11.05.20226449 doi: medRxiv preprint Using the cut-offs as suggested by the manufacturers in conventional orthogonal test settings (i.e., confirmation of positives by a second, confirmatory test system), a specificity of about 100% (99.9-100%) was obtained for all tested combinations. The sensitivities, on the other hand, reached at most the initial value of the screening test depending on the confirmatory test used (Supplemental Table 1 ). Interestingly, for the Roche test, no significant further improvement in specificity could be achieved by orthogonal testing using the cut-offs suggested by the manufacturers, as the initial specificity was already close to 100%. On the other hand, it significantly lost sensitivity in 3 out of 4 tested combinations. Furthermore, of all tested combinations, only the retesting of Abbott results by Roche achieved a statistically significant improvement in specificity while maintaining the screening test's initial sensitivity. In the next step, we aimed to increase sensitivities by lowering the cut-offs for positivity. Increasing sensitivity of automated SARS-CoV-2 antibody assays by reducing the cut-offs Table 2 ). Hence, sensitivity increased significantly for all automated assays: Roche +9.4%, Abbott +12.5%, and DiaSorin +6.3% (Table 1 ). In contrast, the increase in false positives (Roche +15, Abbott +18, and DiaSorin +11) significantly All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted November 7, 2020. ; https://doi.org/10.1101/2020.11.05.20226449 doi: medRxiv preprint reduced the specificity of the tests: Roche -1.3%, Abbott -1.6%, DiaSorin -1.0% (Table 1 ). This loss of specificity caused consistently low positive predictive values at a presumed seroprevalence of 2% (39.6-55.5%). IgG based assays showed strikingly Table 2 ). In the next step, we aimed to restore specificity by an orthogonal testing approach. All specimens with results between the reduced cut-offs derived from the literature and the high cut-offs (where false positives were no longer expected) in the screening assay were retested with a second method (see retesting range in Figure 2 ). Of the total 1181 specimens (1117 pre-COVID and 64 post-COVID samples, 5.4% positivity rate), 27 (2.3%) were retested for Roche, 45 (3.8%) for Abbott and 87 (7.4%) for DiaSorin. The Roche Elecsys® total antibody assay could be combined with all remaining tests to reach a specificity of 100.0% (99.7-100.0), with no false positives left. However, this increase in specificity was not statistically significant compared to the single test approach using the manufacturer's cut-offs, as the initial specificity was already comparably high (99.7%). Simultaneously, sensitivities remained significantly increased when combined with the Abbott, the DiaSorin, and the Technozym® RBD assay (all p<0.05). All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted November 7, 2020. ; https://doi.org/10.1101/2020.11.05.20226449 doi: medRxiv preprint The Abbott Anti-SARS-CoV-2 IgG test, in contrast, reached 100.0% specificity only when followed by the Roche assay; in the other combinations, it remained slightly below, as a few false-positives persisted (99.8-99.9%). Pairing the Abbott with the Roche assay or the Technozym® RBD ELISA also raised the sensitivity significantly (p<0.01). The specificity of the DiaSorin LIAISON Anti-Sars-CoV-2 IgG assay could be significantly improved (99.9-100.0%) when all samples with results >9.0 AU/mL were retested with any of the remaining assays. However, in all combinations except for that with the Roche assay (p<0.05), sensitivity returned to previous levels, but did not further decrease, as it was the case when applying a conventional orthogonal testing algorithm using the manufacturers' cut-offs (results of all combinations are displayed in Table 2 ). Orthogonal testing with reduced cut-offs resulted in improved specificity (between 99.8 and 100% specificity) in any of the 12 combinations tested. The greatest increase in specificity was achieved for the DiaSorin assay (from 98.2% to 99.9 or 100%). Besides, when the Roche assay was used as the primary test, all combinations except with the NP ELISA showed a statistically significant gain in sensitivity (from 89.1% to 95.3% or 96.9%). The Roche assay yielded the highest sensitivity increases when used as a confirmatory test (from 84.4% to 96.9% for Abbott and 82.9% to 88.9% for DiaSorin). In our test cohort, Roche and Abbott's combination led to the best overall performance, no matter whether Roche or Abbott was used as a screening or confirmatory test (specificity 100% and sensitivity 96.9%). The second-best solution was achieved using Roche as screening and RBD ELISA or DiaSorin as a confirmatory test (specificity 100% All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted November 7, 2020. ; https://doi.org/10.1101/2020.11.05.20226449 doi: medRxiv preprint and sensitivity 95.3%). Table 3 provides an overview of the improvements in specificity and sensitivity of all assay combinations. All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted November 7, 2020. ; https://doi.org/10.1101/2020.11.05.20226449 doi: medRxiv preprint The detection of specific antibodies can confirm a previous infection with SARS-CoV-2, even if the infection was not diagnosed by PCR or was asymptomatic in the first place. For this reason, serological SARS-CoV-2 testing is the method of choice for estimating the prevalence of previously infected persons and closing the gap between PCR confirmed cases and unreported cases (17) . Due to the very variable, but globally still low seroprevalence rates for SARS-CoV-2 (16), high demands are placed on the specificity of the antibody assays (Supplemental Figure 1 ). Many commercially available SARS-CoV-2 antibody tests do not provide sufficient specificity to achieve acceptable positive predictive values (PPVs), for example at a seroprevalence rate of 1-5% (4, 11) . So far there have been three different approaches to solve the problem of specificity: a) the use of assays with very high specificity (>99.5%); b) to increase the pre-testing probability by pre-selecting high-risk populations; c) the use of conventional orthogonal testing, whereby initial positive results are confirmed by a second method to exclude false positives. There are indeed already some highly specific assays (14), but their application might not be possible everywhere due to regional differences in availability or lack of technical equipment. This solution, therefore, cannot be implemented in every case. Preselection of populations could lead to a biased picture of the overall situation in seroprevalence studies. A conventional orthogonal test strategy can undoubtedly increase the specificity massively and even bring it very close to 100%, but a decisive disadvantage can arise as recent publications have shown: i.e., a significant loss of sensitivity (19, 26) . We All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted November 7, 2020. ; https://doi.org/10.1101/2020.11.05.20226449 doi: medRxiv preprint could previously show that the sensitivity of SARS-CoV-2 antibody tests can drop below 90%, especially when non-hospitalized patients are included (4). In the meantime, suboptimal sensitivities of SARS-CoV-2 antibody tests have been described in numerous subsequent publications (4, 7, 12, 32). Thus, an orthogonal test strategy with a certain further loss of sensitivity does not seem appropriate for seroprevalence studies, especially considering the dynamic development of infection rates we are facing. A-priori estimates of the expected prevalence could therefore be misleading. If we experience a shift from low prevalence to high prevalence due to the overall trend of infections or regional clustering ("hot spots"), adequate sensitivity is just as important as adequate specificity of the test system used. Based on real data related to SARS-CoV-2, which are in principle generalizable for all types of serological tests, we, therefore, propose a entirely new test strategy and thus a fourth approach to solve the problem: The combination of improved sensitivity through cut-off modeling and an orthogonal test strategy to restore specificity. This algorithm offers the opportunity to significantly improve the overall performance of SARS CoV-2 antibody tests. For this purpose, the recommended manufacturer cut-offs were reduced according to the results of other independent studies (3, 10, 31). These reduced cut-offs to increase sensitivity are close to the modeled Youden indices, and indeed, although described in completely independent studies, they can obviously be applied to our study collective. Therefore, we anticipate that certain generalizability can be assumed in this context and that cut-offs for other test systems can be modeled in a similar way based on the Youden indices. All 12 combinations tested in the present study (3 screening tests with 4 All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted November 7, 2020. ; https://doi.org/10.1101/2020.11.05.20226449 doi: medRxiv preprint different confirmatory tests each) achieved specificities of 99.8%-100.0%. Consequently, a significant increase in specificity was achieved for Abbott, and a massive increase in specificity was seen for DiaSorin, due to the very moderate initial specificity of 98.2%. For DiaSorin, this means a doubling of the PPV at an assumed seroprevalence of 2%. In contrast to a classical orthogonal test approach (19, 26) , no statistically significant decrease in sensitivity below the screening test's initial level was detected in any combination tested. However, the combinations Roche + RBD ELISA or Abbott or DiaSorin, as well as Abbott + RBD ELISA or Roche or DiaSorin, and DiaSorin + Roche showed significant increases in sensitivity. In this context, it should be noted that the maximum achievable sensitivity in the orthogonal test approach depends very much on the sensitivity performance of the screening assay. In our study, the Roche test clearly outperformed the DiaSorin test with a substantial difference in initial sensitivity, and reduced cut-offs of the Roche test, allowing for a wider range of inclusion of additional positives. All additional potential false-positives were ruled out by the confirmation tests. We propose a novel orthogonal test strategy for SARS-CoV-2 serology (but generally applicable to serological testing), which makes it possible to maintain or even significantly improve sensitivity while approaching 100% specificity. All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted November 7, 2020. ; https://doi.org/10.1101/2020.11.05.20226449 doi: medRxiv preprint Captions Fig. 1. A) Testing algorithm utilizing a screening test on an automated platform (ECLIA/Roche, CMIA/Abbott, CLIA/DiaSorin) and a confirmation test, either on one of the remaining platforms or tested by means of ELISA (Technozym RBD, NP) . B) All test results between a reduced cut-off suggested by the literature, and a higher cut-off, above which no more false-positive were observed, were subject to confirmation testing. *…for DiaSorin, no reasonable high cut-off could be defined, as false-positives were present nearly over the total range of the test (see Figure 2 ). § … suggested as a cut-off for seroprevalence testing; ¶ … determined by in-house modeling; 1 … see (31) ; 2 … see (10) ; 3 … see (3) . (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted November 7, 2020. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted November 7, 2020. ; https://doi.org/10.1101/2020.11.05.20226449 doi: medRxiv preprint *…for DiaSorin, no reasonable high cut-off could be defined, as false-positives were present nearly over the total range of the test (see Figure 2 ). § … suggested as a cut-off for seroprevalence testing; ¶ … determined by in-house modeling; 1 … see (31) ; 2 … see (10) ; 3 All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted November 7, 2020. ; https://doi.org/10.1101/2020.11.05.20226449 doi: medRxiv preprint All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted November 7, 2020. ; https://doi.org/10.1101/2020.11.05.20226449 doi: medRxiv preprint SARS-CoV-2 Seroconversion in Humans: A Detailed Protocol for a Serological Assay, Antigen Production, and Test Setup Clinical and Analytical Performance of an Automated Serological Test That Identifies S1/S2-Neutralizing IgG in COVID-19 Patients Semiquantitatively Side by side comparison of three fully automated SARS-CoV-2 antibody assays with a focus on specificity Evaluation of Six Commercial Mid-to High-Volume Antibody and Six Point-of-Care Lateral Flow Assays for Detection of SARS-CoV-2 Antibodies Quantification of SARS-CoV-2 antibodies with eight commercially available immunoassays Analytical and Clinical Evaluation of the Automated Elecsys Anti-SARS-CoV-2 Antibody Assay on the Roche cobas e602 Analyzer Performance of an automated chemiluminescence SARS-CoV-2 IG-G assay Differences in Performance Characteristics Among Four High-Throughput Assays for the Detection of Antibodies Against SARS-CoV-2 Using a Common Set of Patient Samples Clinical Performance of the Roche SARS-CoV-2 Serologic Assay Evaluation of three fully-automated SARS-CoV-2 antibody assays SeroTracker: a global SARS-CoV-2 seroprevalence dashboard Canadian society of clinical chemists (CSCC) interim consensus guidance for testing and reporting of SARS-CoV-2 serology Association between SARS-CoV-2 neutralizing antibodies and commercial serological assays Evaluation of Three Commercial SARS-CoV-2 Serologic Assays and their Performance in Two-Test Algorithms Antibody tests for identification of current and past infection with SARS-CoV-2 Antibodies in serum of convalescent patients following mild COVID-19 do not always prevent virus-receptor binding. Allergy. 2020. 22 COVID-19 confirmed patients with negative antibodies results Clinical performance of a semi-quantitative assay for SARS-CoV2 IgG and SARS-CoV2 IgM antibodies Evaluation of Orthogonal Testing Algorithm for Detection of SARS-CoV-2 IgG Antibodies A serological assay to detect SARS-CoV-2 seroconversion in humans Usage Data and Scientific Impact of the Prospectively Established Fluid Bioresources at the Hospital-Based MedUni Wien Biobank The LEAD (Lung, Heart, Social, Body) Study: Objectives, Methodology, and External Validity of the Population-Based Cohort Study A serological assay to detect SARS-CoV-2 seroconversion in humans effects on sensitivity and specificity.  significant increase, or a specificity of approximately 100.0% that could not be further improved; Ʇ… no significant change.