key: cord-0903050-0dgmfeak
authors: Meyer, B.; Torriani, G.; Yerly, S.; Mazza, L.; Calame, A.; Arm-Vernez, I.; Zimmer, G.; Agoritsas, T.; Stirnemann, J.; Spechbach, H.; Guessous, I.; Stringhini, S.; Pugin, J.; Roux-Lombard, P.; Fontao, L.; Siegrist, C.-A.; Eckerle, I.; Vuilleumier, N.; Kaiser, L.
title: Validation of a commercially available SARS-CoV-2 serological Immunoassay
date: 2020-05-06
journal: nan
DOI: 10.1101/2020.05.02.20080879
sha: df1f8d510066c8d95a8482ed526fddcef4e0aced
doc_id: 903050
cord_uid: 0dgmfeak

Aims: To validate the diagnostic accuracy of a Euroimmun SARS-CoV-2 IgG and IgA immunoassay for COVID-19 disease. Methods: In this unmatched (1:1) case-control validation study, we used sera of 181 laboratory-confirmed SARS-CoV-2 cases and 176 negative controls collected before the emergence of SARS-CoV-2. Diagnostic accuracy of the immunoassay was assessed against a whole spike protein-based recombinant immunofluorescence assay (rIFA) by receiver operating characteristic (ROC) analyses. Discrepant cases between ELISA and rIFA were further tested by pseudo-neutralization assay. Results: COVID-19 patients were more likely to be male and older than controls, and 50.3% of them were hospitalized. ROC curve analyses indicated that IgG and IgA had a high diagnostic accuracy with AUCs of 0.992 (95% Confidence Interval [95%CI]: 0.986-0.996) and 0.977 (95%CI: 0.963-0.990), respectively. IgG assays outperformed IgA assays (p=0.008). Considering optimized cut-offs taking the 15% inter-assay imprecision assessed into account, an IgG ratio cut-off > 1.5 displayed a 100% specificity (95%CI: 98-100) and a 100% positive predictive value (95%CI: 97-100). A 0.5 cut-off displayed a 97% sensitivity (95%CI: 93-99) and a 97% negative predictive value (95%CI: 93-99). Adopting these thresholds, rather than those of the manufacturer, improved assay performance, leaving 12% of IgG ratios ranging between 0.5-1.5 as indeterminate. Conclusions: The Euroimmun assay displays a nearly optimal diagnostic accuracy using IgG against SARS-CoV-2 in a samples of patients, without any obvious gains from considering IgA serology. The optimized cut-offs are fit for rule-in and rule-out purposes, allowing determination of whether individuals have been exposed to SARS-CoV-2 or not in our study population. They should however not be considered as a surrogate of protection at this stage.

High throughput and reliable serological assays detecting antibodies against SARS-CoV-2 are 68 essential to determine the proportion of SARS-CoV-2 infected individuals and estimate the 69 current seroprevalence in the general population or in high-risk groups, such as health care 70 workers. Serological assays can complement diagnostic strategies focusing on the identification 71 of the infectious agent during the acute phase of disease. Unlike RT-PCR, they can identify 72 infected individuals that remained asymptomatic or undiagnosed, which are both frequent 73 conditions during SARS-CoV-2 infection, long after the initial infection. Validated serological 74 assays are also key to understanding the (immuno)-pathophysiology of COVID-19 in various 75 patients' groups and will be critical to characterize responses elicited by the numerous vaccine 76 candidates in development (1) . 77

Designing serological testing strategies with high sensitivity and specificity and with acceptable 78 positive (PPV) and negative predictive values (NPV) is far from trivial, and requires taking two 79 major analytical aspects into account: analytical specificity and sensitivity (2) , (3). The former 80 may largely be determined by the degree of cross-reactivity with other CoVs, which frequently 81 cause common colds in humans (i.e. HCoV-229E, -NL63, -OC43 and -HKU1) (4) resulting in 82 seroprevalence rates usually above 90% in adults (5). 83

This cross-reactivity occurs when virus-specific antigenic epitopes are highly similar and 84 recognized by the same B cells. It is best defined by the proportion of "false-positive" SARS-85

CoV-2 results in individuals who were never exposed to this pathogen. In contrast to common 86 cold CoVs, the seroprevalence for MERS-CoV is low even in endemic countries (6). Therefore, 87 cross-reactivity between MERS-CoV and SARS-CoV-2 is not a critical factor when assessing 88 population seroprevalence. Previous studies have shown that antibodies against common cold 89

CoVs can cause considerable cross-reactivity in serological assays, depending on the type of 90 assay and antigens used. Particularly, whole virus-or nucleocapsid protein-based assays 91 showed a higher cross-reactivity compared to whole spike or S1 domain-based assays resulting 92 in lower specificity (4). During the early phase of an outbreak, when SARS-CoV-2 93 seroprevalence is low, serological testing strategies must have a very high specificity to reach a 94 high positive predictive value (PPV) and avoid false positive results. 95

On the other hand, analytical sensitivity is strongly influenced by the epidemic course, the 96 disease biology, and numerous analytical factors. All these interrelated items are primarily 97 influenced by the intrinsic immunogenicity of the SARS-CoV-2 antigens and the magnitude and 98 duration of B cell responses elicited by infection, be it asymptomatic, benign, moderate or 99 severe (2) , (3). 100 . CC-BY-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted May 6, 2020. . https://doi.org/10.1101/2020.05.02.20080879 doi: medRxiv preprint Last but not least in the context of a pandemic, the availability of high throughput and reliable 101 diagnostic platforms is key for health care systems to effectively handle the testing demand, 102 while respecting clinically compatible diagnostic turnaround times (TAT). In this study, we 103 performed an extensive validation of a high throughput SARS-CoV-2 commercial serological 104 platform quantifying both serum IgG and IgA against the S1 protein. As reference method, we 105 used a whole spike-based recombinant immunofluorescence assay (rIFA) (4) , (7) , (8). Selected 106 sera from SARS-CoV-2-infected patients were assessed for their neutralization capacity using a 107 pseudovirion assay (see below). 108 109

Aims 111

The aim of this study is to validate and define the operational cut-off values of a commercially 112 available ELISA-based SARS-CoV-2 serological assay that could be applied at large scales to 113 reliably determine the presence of specific IgG as a marker of SARS-CoV-2 infection. This study 114 used RT-PCR confirmed cases, but the goal was to be able to identify exposure to SARS-CoV-2 115 by immunoassay alone. Therefore, the ELISA results were compared against recombinant 116 immunofluorescence assay (rIFA), which was considered as the reference method due to its 117 demonstrated high specificity for serology of other CoVs such as MERS-CoV (6). The 118 secondary goal is to assess the potential added value of IgA in patients recently infected with 119 SARS-CoV-2. 120 121

Negative control serum samples (n=176) were collected for various serological testing in our 123 routine laboratory and stored for analytical validation. These sera were collected in 2013, 2014 124 and 2018 before the start of the outbreak and thus have not been exposed to SARS-CoV-2. 125 Sera (n=181) of PCR-confirmed COVID-19 patients were collected at the University Hospitals of 126 Geneva from hospitalized patients (n=91) as well as from patients from the outpatient clinic 127 (n=90). Ethical approval for all sera used in this study was waived by the local ethics committee 128 of the HUG that approves usage of leftover of patient serum collected for diagnostic purposes. 129

The number of days from symptom onset to blood collection was based on patient history 130 whenever this information was available or could be retrieved in a reliable way; otherwise, we 131 used the date of RT-PCR positivity as a surrogate for onset of symptoms. Serum samples from 132 unmatched PCR-confirmed COVID-19 hospitalized patients were collected for routine diagnostic 133 purposes under a general informed consent and outpatients were asked if they were willing to 134 . CC-BY-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

The copyright holder for this preprint this version posted May 6, 2020. Inter-assay variation was 15.6% for IgG at a ratio of 2.09 (n=17) and 17.7% for IgA at a ratio of . CC-BY-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

The copyright holder for this preprint this version posted May 6, 2020. . https://doi.org/10.1101/2020.05.02.20080879 doi: medRxiv preprint

The IgG antibody response against the spike protein of SARS-CoV-2 was assessed by rIFA as 168 described and previously validated for MERS-CoV (6) , (7). Briefly, Vero B4 cells were 169 transfected with the mammalian expression vector pCG1-SCoV2-S (kindly provided by M. 170

Hoffmann and S. Pöhlmann, DPZ, Göttingen, Germany) using Fugene HD (Promega, #E2311). 171

After 24h of incubation cells were detached and residual trypsin was removed by centrifugation 172 at 300x g for 5min. 50µl of transfected cells were seeded at a density of 2x10 5 /ml on multi-test 173 glass slides (DUNN Labortechnik GmbH #40-412-05) and incubated for 6h at 37°C, 5% CO2. 174

Afterwards slides were washed 2x with PBS and fixed for 10min using ice-cold 175 Acetone/Methanol (ratio 1:1). For rIFA staining, slides were rehydrated for 10min using PBS + 176 0.1% Tween20 (PBS-T) and blocked with 5% milk in PBS-T for 30min at RT. Sera were diluted 177 1:40 in blocking buffer, 30µl were applied on each spot and incubated for 60min at 37°C or RT. . CC-BY-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted May 6, 2020. . CC-BY-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted May 6, 2020. . https://doi.org/10.1101/2020.05.02.20080879 doi: medRxiv preprint putative negative and 14 (7.7%) negative samples, indicating an overall detection rate of 91.2%. 236

We found that rIFA seropositivity was low at 0-10 dpos/dpd (12.5%) but increased to 92.0% and 237 100% in sera collected at 11-20 and 21-39 dpos/dpd, respectively (table 2) indeterminate and 13 (7.2%) negative samples for IgA. This resulted into an overall significantly 250 higher seropositivity for IgA (90.6%) than IgG (85.1%) in the S1-based ELISA (p= 0.041). 251 252 Next, we analysed the seropositive rates of both ELISAs at different dpos or dpd ( Figure 1E ) 253 according to the EI cut-offs. Sera collected 21 dpos/dpd had a similarly high seropositivity for 254 both IgG (96.7%) and IgA (96.7%) (p >0.99). A higher seropositivity was observed for IgA 255 compared to IgG for sera collected at 11-20 dpos/dpd (91.1% vs 84.8%, p= 0.0264) but not 0-10 256 dpos/dpd (37.5% vs 0%, p= 0.2). No significant difference was found between hospitalized and 257 outpatients in IgG or IgA ELISA (p= 0.833, and p= 1.000, respectively). 258 259 ROC curve analysis (Fig 1; C rIFA and that IgA did not improve diagnostic value. In a subgroup analysis considering only sera 267 harvested before 21 days after symptoms onset (n=120), ROC curve analyses displayed similar 268 . CC-BY-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted May 6, 2020. IgG ratio cut-off, ROC curve analyses indicated that the SE was 86%, the SP 100%, and the 289 NPV 89%. Thus, selecting a 1.5 cut-off, rather than the recommended 1.1 cut-off for IgG 290 seropositivity, allows the securing of a PPV of 100% despite a 15% imprecision. Higher VCs 291 would translate into a higher seropositivity cut-off to secure an identical PPV. For rule-out 292 purposes (i.e. the seronegativity lower cut-off), the best trade-off IgG ratio cut-off was found to 293 be < 0.5. At this value, the SE was 97%, the SP 87%, the NPV 97% and the PPV 86% (table 4) . 294

This defines an indeterminate range between IgG ratios of 0.5 and 1.5, which represented 43 295 cases (12%) of our samples (including 20 control and 23 COVID-19 samples). In this 296 indeterminate zone, all 20 sera from controls were confirmed as negative (n=19) or putative 297 negative (n=1) by rIFA. For indeterminate samples of COVID-19 patients, five were negative, 298 one was a putative negative and 18 were positive by rIFA. 299

In the subgroup of patients whose samples were taken before 21 dpos, using the manufacturer 300

IgG ratio seropositivity cut-off (1.1), the SE was 91%, the SP 100%, the PPV 100%, and the 301 . CC-BY-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted May 6, 2020. . https://doi.org/10.1101/2020.05.02.20080879 doi: medRxiv preprint NPV 62%. At the manufacturer seronegative cut-off (<0.8), the SE was 91%, the SP 100%, the 302 PPV 100%, and the NPV 64% (table 4) . Importantly, no patient displayed a ratio between 0.8 303 and 1.1 in this subgroup, preventing us from estimating the importance of the indeterminate 304 cases. 305

Using the aforementioned optimized IgG ratio cut-off for IgG seropositivity (1.5), the SE was 306 82%, the SP 100%, the PPV 100%, and the NPV 46%. At the cut-off for IgG seronegativity 307 (<0.5), the SE was 96%, the SP 69%, the PPV 95%, and the NPV 73% (table 4) The second notable finding of this study is that the current manufacturer cut-offs are prone to 331 misinterpretations and should not be used without proper evaluation before routine testing. 332

Keeping in mind the challenges of developing a serological assay for a new viral disease in an 333 emergency situation; securing both rule-in and rule-out cut-offs is key to mitigate the 334 unavoidable risk of false positive and false negative results due to the combination of a highly 335 . CC-BY-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted May 6, 2020. . https://doi.org/10.1101/2020.05.02.20080879 doi: medRxiv preprint dynamic pre-test probability variation, with a suboptimal seroconversion time when studies are 336 undertaken at the peak of an epidemic. Our analyses revealed the following limitations of the 337 manufacturer's seropositivity cut-off. First, with an inter-assay imprecision of 15% assessed at 338 an IgG ratio of 2.09 (above than the 1.5 two cut-off value selected) translating into a LSC of 0.42 339 IgG ratio, our results indicate that the analytical imprecision is higher than the range of the 340 indeterminate zone proposed by the manufacturer, which encompasses a delta of 0.3 IgG ratio. 341

This implies that any result within the 0.8-1.1 IgG ratio range could be randomly either above, 342 within or below these values just because of analytical imprecision. Secondly, with a LSC of 343 0.42 our results indicated that a higher IgG ratio cut-off value was needed to secure an optimal 344 specificity and PPV. Adding the 0.42 LSC to the 1.1 cut-off yielded a 1.5 ratio as the IgG 345 seropositivity cut-off with a PPV of 100%, the lower end of the 95%CI still being compatible with 346 a 97% rule-in strategy. Notably, using this cut-off in the subgroup of patients under 21 dpos/dpd, 347 the SP and PPV were still 100%, however, with broader confidence intervals. Similarly, in an 348 attempt to maximize the negative predictive value at the rule-out cut-off, the cut-off had to be 349 decreased from 0.8 to 0.5 IgG ratio in order to reach an overall NPV of 97%, with a 93% at the 350 lower end of the 95CI. In the subgroup of patients under 21 dpos/dpd this interval was found to 351 be substantially larger (95%CI: 45-91). Taken together, these results indicate that at this stage 352 the optimal rule-in cut-off should be set at >1.5 of IgG ratio for seropositivity and at <0. Regarding potential limitations, we need to highlight several additional points. We evaluated an 368 ELISA assay measuring antibodies against the S1 domain of the spike protein and not against 369 . CC-BY-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted May 6, 2020. . https://doi.org/10.1101/2020.05.02.20080879 doi: medRxiv preprint the full protein, which may contain other highly relevant epitopes from the S2 domain. Such 370 factors could potentially explain why some samples were negative by ELISAs but positive by 371 rIFA, as the whole-spike protein is used in rIFA. Secondly, we strongly emphasize two important 372 cut-off limitations related to the analytical imprecision. At the level of the analytical imprecision, 373 the SARS-CoV-2 IgG intra-individual biological variation being unknown, we had to use the LSC 374 instead of the reference change value (22), which most likely would have translated into a 375 higher seropositivity cut-off. Along the same line, because the intra-lot imprecision of the 376 reagents is still undetermined, but expected to be higher than 15%, this could have a similar 377 impact on cut-off determination. Nevertheless, these cut-offs implemented in routine testing at 378 the Geneva University Hospitals (GE-cut-offs) proved useful in the management of a number of 379 clinically compatible COVID-19 patients with negative PCR results. 380 Importantly, this study is a diagnostic accuracy validation study and not a seroprevalence study. 381

This implies that the current seropositivity cut-off has to be considered with caution in population 382 studies. Indeed, the 100% PPV achieved in this study was due to combined effect of a 100% 383 specificity and a 1:1 distribution of control and samples form patients with positive SARS-CoV-2 384 PCR. Therefore, in population seroprevalence studies with a lower expected proportion of 385 COVID-19, the PPV at the 1.5 cut-off will likely decrease. In this context, increasing the rule-in 386 IgG cut-off to a higher value or using a secondary specific confirmatory assay may become 387 necessary. Due to this limitation, at this stage we recommend the confirmation of all positive 388 ELISA results (including ratios above 1.5) using a second serological assay such as rIFA for 389 seroprevalence studies with a low pre-test probability, and the confirmation of doubtful ELISA 390 results in settings with a pre-test probability around 50%. A summary of our proposed testing 391 strategy is given in figure 2 . 392 393

In conclusion, in this validation study performed on 357 sera, of which 50.7% came from 394 patients with COVID-19, we demonstrate a close to optimal diagnostic accuracy of IgG SARS-395

CoV-2 serology of the Euroimmun assay, without any obvious gains from IgA serology. Taking  396 analytical imprecision into account, we propose optimized cut-offs allowing a PPV of 100% and 397 a NPV of 97% to be secured with an indeterminate zone comprising about 12% of the results, 398 for which additional rIFA analyses are currently necessary. Ongoing seroprevalence studies will 399 be instrumental to further refine the optimal rule-in and rule-out cut-offs, as well as the optimal 400 testing strategy, which may require a highly specific confirmatory assay. For the time being, this 401 assay seems to be fit for the purpose of enabling authorities to make informed decisions 402 regarding measures to be taken in order to manage the SARS-CoV-2 pandemic. Using the GE 403 . CC-BY-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted May 6, 2020. CC-BY-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted May 6, 2020. Tables 500 Table 1 Negative CC-BY-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted May 6, 2020. . https://doi.org/10.1101/2020.05.02.20080879 doi: medRxiv preprint Table 4 Cut-Offs Sensitivity (95%) Specificity (95%) Positive predictive value (95%)

Negative predictive value (95%) . CC-BY-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

The copyright holder for this preprint this version posted May 6, 2020. . https://doi.org/10.1101/2020.05.02.20080879 doi: medRxiv preprint

SARS-CoV-2 Vaccines: Status Report. Immunity

Serological Testing for the Diagnosis of Emerging Infectious Diseases

Dengue and Zika: Control and Antiviral Treatment Strategies

Guidelines of the Office International des Epizooties for laboratory quality evaluation, for 430 international reference standards for antibody assays and for laboratory proficiency testing. 431 Rev -Off Int Epizoot

Serological assays for emerging coronaviruses: 433 Challenges and pitfalls. Virus Research

Prevalence of antibodies to four human 435 coronaviruses is lower in nasal secretions than in serum. Clin Vaccine Immunol

Middle East respiratory syndrome coronavirus antibodies in Saudi Arabia: a nationwide, 439 cross-sectional, serological study

Assays for 441 laboratory confirmation of novel human coronavirus (hCoV-EMC) infections

Acute Respiratory Syndrome Coronavirus 2-Specific Antibody Responses in Coronavirus 445 Disease

Detection of 2019 447 novel coronavirus (2019-nCoV) by real-time RT-PCR

The kappa statistic in reliability studies: use, interpretation, and sample 449 size requirements

A vesicular stomatitis virus replicon-based bioassay for the 451 rapid and sensitive determination of multi-species type I interferon

Clotrimazole Derivatives as Specific Inhibitors of Arenavirus Fusion

Vesicular 456 stomatitis virus pseudotyped with severe acute respiratory syndrome coronavirus spike 457 protein

Glüer CC. Monitoring skeletal changes by radiological techniques

Long-term variability of bone turnover 461 markers in patients with non-metastatic breast cancer

Comparing the areas under two or more 463 correlated receiver operating characteristic curves: a nonparametric approach

466 Evaluation of nine commercial SARS-CoV-2 immunoassays. medRxiv

A human SARS-CoV neutralizing 469 antibody against epitope on S2 protein

B-cell responses in patients 472 who have recovered from severe acute respiratory syndrome target a dominant site in the 473 S2 domain of the surface spike glycoprotein

Amino Acids 1055 to 475 1192 in the S2 Region of Severe Acute Respiratory Syndrome Coronavirus S Protein 476 Induce Neutralizing Antibodies: Implications for the Development of Vaccines and Antiviral 477 Agents

All samples (n=357) IgG ratio