key: cord-1021456-hk3mcsf7 authors: Xu, Gang; Emanuel, Anthony J; Nadig, Satish; Mehrotra, Shikhar; Caddell, Brittany A; Curry, Scott R; Nolte, Frederick S; Babic, Nikolina title: Evaluation of Orthogonal Testing Algorithm for Detection of SARS-CoV-2 IgG Antibodies date: 2020-09-07 journal: Clin Chem DOI: 10.1093/clinchem/hvaa210 sha: 87342fad58bd0031c3797e9186cddaa4baa861eb doc_id: 1021456 cord_uid: hk3mcsf7 BACKGROUND: Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) antibody testing is an important tool in assessment of pandemic progress, contact tracing, and identification of recovered coronavirus disease 2019 (COVID-19) patients. We evaluated an orthogonal testing algorithm (OTA) to improve test specificity in these use cases. METHODS: A two-step OTA was applied where individuals who initially tested positive were tested with a second test. The first-line test, detecting IgG antibodies to the viral nucleocapsid protein was validated in 130 samples and the second-line test, detecting IgG antibodies to the viral spike protein in 148 samples. The OTA was evaluated in 4,333 clinical patient specimens. The seropositivity rates relative to the SARS-CoV-2 PCR positivity rates were evaluated from our entire patient population data (n = 5,102). RESULTS: The first-line test resulted in a clinical sensitivity of 96.4% (95% CI; 82.3% to 99.4%), and specificity of 99.0% (95% CI; 94.7% to 99.8%), whereas the second-line test had a sensitivity of 100% (95% CI; 87.7% to 100%) and specificity of 98.4% (95% CI; 94.2% to 99.5%). Using the OTA, 78/98 (80%) of initially positive SARS-CoV-2 IgG results were confirmed with a second-line test, while 11/42 (26%) of previously diagnosed COVID-19 patients had no detectable antibodies as long as 94 days post PCR diagnosis. CONCLUSION: Our results show that an OTA can be used to identify patients who require further follow-up due to potential SARS CoV-2 IgG false positive results. In addition, serological testing may not be sufficiently sensitive to reliably detect prior COVID-19 infection. Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), a novel coronavirus that originated in the city of Wuhan, China in late 2019 has spread rapidly throughout the world resulting in pandemic of coronavirus disease 2019 . Despite the relatively short timeframe of this outbreak, a considerable number of laboratory tests targeting some aspect of the virus or disease state have been developed (1) . As with other viruses, nucleic acid amplification tests (NAAT) have been employed as a sensitive marker of virus presence within various clinical samples, most notably in respiratory specimens. The development of such assays has had a positive impact on clinical diagnosis and epidemiologic endeavors. In addition to diagnostic NAATs, serological tests increasingly are being employed in the populations to assess the progress of the pandemic and in individuals to identify recovered cases of COVID-19 in previously undiagnosed and/or asymptomatic individuals (2, 3) . One of the early challenges with serological testing was that, due to the initial lack of regulatory oversight from US Federal Drug Administration (FDA), several serological tests with inadequately verified performance characteristics were marketed (4, 5) . This led to reporting of inaccurate results. Some early serological tests reported high rates of false positive results due to assay cross-reactivity with other antibodies, such as those against the other endemic human coronaviruses (6, 7) . Fortunately, automated serology tests developed by reputable in vitro diagnostic (IVD) vendors are now readily available and are being evaluated by clinical laboratories (8) (9) (10) (11) (12) . Given that current utility of serological testing is mostly for epidemiological studies, proper evaluation of the clinical sensitivity and specificity of these tests within the tested population is 4 very important. The test specificity directly affects its positive predictive values (PPV) and thus the reliability of a positive result. One of the testing strategies recommended by Center for Disease Control (CDC) to improve PPV of a test result is an orthogonal approach, a two-step algorithm where individuals who initially test positive are tested with a second test (13) . We evaluated an orthogonal testing approach utilizing two immunoassays validated clinically at the Medical University of South Carolina (MUSC) clinical laboratories. The first-line test is Abbott SARS-CoV-2, targeting nucleocapsid (N) protein. The second-line test is a laboratory-developed enzyme-linked immunosorbent assay (ELISA), targeting spike (S) protein. With this strategy involving testing for antibodies against two different immunogenic targets, we aimed to improve the specificity of a positive test, thus providing important information to patients and public health officials. All method validation specimens were collected from apparently healthy volunteers or derived from de-identified, residual clinical specimens. Per MUSC IRB policy, this project was classified as a clinical laboratory quality improvement project and was exempted from IRB review and approval. Clinical specificity samples were a combination of residual clinical specimens sent to our laboratory before the COVID-19 outbreak (prior to December 2019) and sera collected from asymptomatic volunteer donors in April 2020. The sensitivity samples were either de-identified, residual, serum specimens collected from patients at least 14 days following PCR diagnosis of 5 COVID-19 or pre-screened convalescent plasma donors provided by Vitrologic (Charleston, SC). Most specimens were available in sufficient quantities to use in evaluation of both first and second-line tests. The two-step algorithm was evaluated using de-identified, residual serum specimens received in our laboratory for SARS-CoV-2 IgG testing from mid -April through May 2020. The first-line test in our algorithm is the automated Abbott SARS-CoV-2 IgG assay (Abbott Laboratories). This assay has been granted Emergency Use Authorization (EUA) by the US Food and Drug Administration (FDA). It is a two-step chemiluminescent microparticle immunoassay 6 Antibodies that bind to the surface glycoprotein of the virus, termed spike (S) protein were determined using the serological assay described earlier (14) . Briefly, the 96 well microtiter plates were coated overnight at 4.0 °C using the commercially available S-protein (LakePharma, Cat # 46328) at 2 µg/ml. Plates were then washed thrice and blocked for 1 hr with PBS-Tween plus 3% milk powder (weight/volume). The diluted samples were then added to each well and incubated for 2 hr, after which the plates were washed and anti-human IgG (Fab specific) HRP labeled secondary antibody 1:3000 in PBS-T containing 1% milk was added for 1 hr. Subsequently, plates were washed and substrate (o-phenylenediamine dihydrochloride, OPD solution; Sigma-Aldrich) was added. The reaction was stopped by adding 50 µl of 3 mol/L HCl to all wells before reading at 490 nm to record the data. Inactivated human AB serum was used as a negative control, while monoclonal antibody CR3022 was used as a positive control to ensure assay reproducibility. This ELISA test was validated using a total of 148 samples. Clinical sensitivity was assessed using 26 known seropositive samples. All positive samples with sufficient quantities were shared between the two assays. Clinical specificity was determined using a total of 122 specimens collected from patients from pre-COVID-19 or with no evidence of SARS-CoV-2 infection. and N2, and the human RNase P gene (RP) as an internal control. This test has a reported LOD of 1,000 copies/ml. University of Chicago. They used the Roche cobas SARS-CoV-2 assay on the 6800 system and the Cepheid Xpert Xpress SARS-CoV-2 assay as previously described (10) . In two patients reported to be positive for SARS-CoV-2 RNA the test and performing laboratory could not be determined. Clinical sensitivities and specificities along with corresponding Wilson 95% confidence intervals (CI) were calculated using a clinical method validation software package (Analyze-it for Microsoft Excel). Patient demographics (gender and age), PCR and serology testing statistics were provided by MUSC Health Analytics using Tableau software. All serology results including S/C index values and final interpretative results were extracted from our laboratory information system (Cerner Millennium). The paired analysis of discordant results was performed using univariate distribution plots (Analyze-it software). 9 Clinical sensitivity and specificity characteristics for the first-line test (Abbott SARS-CoV-2 IgG) are summarized in Table 1 The two-step testing algorithm was evaluated using a total of 4,333 individual clinical patient specimens analyzed April through May 2020. The age distribution in our population was ≤ 18 years (2%), 19-49 (49%), 50-59 (18%), 60-69 (17%), 70-79 (12%) and ≥ 80 (2%). Of the patients tested, 98 were initially positive by a first-line test, resulting in 2.3% seroprevalence rates. We were able to confirm 78/98 (80%) of seropositive results using the two test approach. The resulting discordances in 20 samples indicate that our true seroprevalence rates may be <2%. The distribution of all initially positive specimens (Abbott test S/C of 1.4) is shown in Figure 1 . As the figure shows, all but one of the discordant samples were below Abbott SARS-CoV-2 IgG S/C index of 3.5. The distribution of all specimen results tested at RL N is shown in Figure 2 . Following the analysis of 20 discordant specimens by RL N (Figure 2A) Figure 2B) . Finally, within the total tested population (n=5,102), a record of COVID-19 diagnosis by PCR was available for 42 patients. Of those, 11 patients (26%) had no detectable SARS-CoV-2 IgG antibodies and one patient had discordant antibody results (see Table 3 ). The seronegative patients were tested at 14-94 days post-diagnosis. The current outbreak of SARS-CoV-2, as well as the global response efforts aimed at controlling the spread of the virus, has significantly impacted daily life worldwide. Although there are uncertainties that still remain regarding utility of antibody testing -namely duration, strength, and specificity of the immune response -it is a valuable tool that can help monitor disease progress and determine disease prevalence. Our testing strategy and experience provide data from a large cohort of patients in South Carolina, a state that has experienced a low burden of disease from SARS-CoV-2 as of late May 2020. Based on our data, seroprevalence in our population sample is 1.8% (78/4,333). Of note, this population sample is susceptible to selection bias among a relatively healthy, predominantly Caucasian group of healthcare workers in a health system with exceptionally low numbers of documented COVID-19 infections from lowincidence areas of South Carolina. Analysis of population data revealed that virtually all discordant results were among the patients without prior positive PCR diagnosis (Table 3) . Surprisingly, our data analytics also revealed that approximately a quarter of patients with prior COVID-19 PCR diagnosis had no detectable antibodies 14-94 days post diagnosis, which raises a question whether current serology tests are sensitive enough to detect low levels of SARS-CoV-2 IgG antibodies or whether serologic response to SARS-CoV-2 wanes quickly in some recovered patients. We did not perform follow-up serological testing of seropositive individuals in our cohort to assess for the latter possibility. It also may reflect some false-positive PCR test results since the true diagnostic accuracy of none of the many molecular tests for SARS-CoV-2 RNA currently in use under EUA is known and is likely to vary between different tests and laboratories (17) . Of note, at this time only symptomatic patients were tested by PCR. Using the orthogonal testing approach, with a highly specific first-line test and a more sensitive second-line test, we were able to confirm 80% of positive SARS-CoV-2 IgG results. The potential false positive rate of 15-20% is to be expected in low-prevalence population and given our first-line test sensitivity and specificity. The side-by-side analysis of the Abbott test, MUSC ELISA and RL N revealed that the Abbott SARS-CoV-2 IgG test results with S/C Index >3.5 are likely true positive results and would not require additional follow-up (see Figure 1 ). Patient results with an S/C Index value between 1.4 and 3.5 (about 50% of all initially positive results for our institution) require additional follow-up to exclude a possibility of discordance due to difference in the kinetics of antibody response. Unfortunately the RL N test could not effectively adjudicate MUSC discordant results, most likely due to its inferior sensitivity. As shown in Figure 2B , we were not able to confirm 5/20 MUSC positive results with the RL N assay. One was indeterminate and four samples were negative by the RL N test. All had Abbott S/C Index value <3.5. This suggests that sensitivity of the RL N is not sufficient to detect antibodies at these levels, since the likelihood of false positive in these samples is low. Given that all but one MUSC discordant results had Abbott S/C Index value <3.5, most (18/20) were either indeterminate or negative by RL N assay (Figure 2A) . Based on seroconversion studies reported in the literature (18) (19) (20) and studies conducted by IVD vendors (12) , most patients seroconvert 14 days following the infection and can be detected by serological tests directed against both N and S proteins. Thus, it is reasonable to conclude that, in the absence of PCR data, if the secondline test is still negative following a re-test at 14 days, the original first-line positive test result may be a false positive. In our institution we report all discordant results with an interpretive comment stating that the antibodies against N protein were detected and antibodies against S protein were not detected and recommend repeat testing. In conclusion, we were able to show that the orthogonal testing strategy is a useful tool that can be used to help identify potentially false positive SARS-CoV-2 IgG serology results, particularly in populations with low disease prevalence. While the advantage of this approach is clear, one of the caveats is that the false positive serological results still cannot be definitively characterized. In addition, this approach may be not feasible for many clinical laboratories, due to costs associated or lack of appropriate tests. EUA Authorized Serology Test Performance Serological assays for SARS-CoV-2 infectious disease: benefits, limitations and perspectives Development and clinical application of a rapid IgM-IgG combined antibody test for SARS-CoV-2 infection diagnosis Double-edged spike: are SARS-CoV-2 serologic tests safe right now Test performance evaluation of SARS-CoV-2 serological assays SARS-CoV-2 serology: much hype, little data Immunoglobulin M for acute infection: true or false? Clin Vaccine Immunol Performance characteristics of the Abbott Architect SARS-CoV-2 IgG assay and seroprevalence in Validation of a chemiluminescent assay for specific SARS-CoV-2 antibody Evaluation of the EUROIMMUN Anti-SARS-CoV-2 ELISA Assay for Detection of IgA and IgG Antibodies Emergency Use Authorization (EUA) document Food and Drug Administration (FDA) Emergency Use Authorizations for Medical Devices Centers for Disease Control and Prevention (CDC). Interim Guidelines A serological assay to detect SARS-CoV-2 seroconversion in humans Validation and verification of the Abbott RealTime SARS-CoV-2 assay analytical and clinical performance CDC 2019-Novel Coronavirus (2019-nCoV) Real-Time RT-PCR Diagnostic Panel False negative tests for SARS-CoV-2 infectionchallenges and implications Clinical evaluation of serological IgG antibody response on the Abbott Architect for established SARS-CoV-2 infection Performance characteristics of four highthroughput immunoassays for detection of IgG antibodies against SARS-CoV-2 Clinical performance of two SARS-CoV-2 serologic assays We thank Colleen Cloud for her invaluable assistance in specimen analysis. We also thank MUSC Data Analytics team for their help with data analysis. Finally, we thank Dr. Vera Tesic