key: cord-0976065-c67y4tx4 authors: Nilsson, Anna Christine; Holm, Dorte Kinggaard; Justesen, Ulrik Stenz; Gorm-Jensen, Thøger; Andersen, Nanna Skaarup; Øvrehus, Anne; Johansen, Isik Somuncu; Michelsen, Jens; Sprogøe, Ulrik; Lillevang, Søren Thue title: Comparison of six commercially available SARS-CoV-2 antibody assays – choice of assay depends on intended use date: 2020-12-10 journal: Int J Infect Dis DOI: 10.1016/j.ijid.2020.12.017 sha: 96aa53d1d844f1c0785aab12592c9da30ce8d968 doc_id: 976065 cord_uid: c67y4tx4 Objectives Evaluate six commercial serological assays for detection of IgA, IgM or IgG SARS-CoV-2 antibodies in different disease severities. Methods Three lateral flow tests (LFTs) (Acro IgM/IgG, CTK IgM/IgG, Livzon IgM/IgG) and three ELISA assays (Euroimmun IgA and IgG, Wantai IgM) were included. Application was evaluated in samples from 57 SARS-CoV-2 RT-PCR positive patients stratified according to disease severity. Specificity was assessed using historical samples from 200 blood donors. Results While IgM LFTs failed to detect SARS-CoV-2 antibodies in 37–84% of non-hospitalised cases, the Wantai IgM ELISA detected antibodies in 79%. The Euroimmun IgG ELISA detected antibodies in 95% of non-hospitalised subjects. IgA, IgM and IgG ELISA levels were initially low, increased over time and correlated with disease severity. LFT sensitivity declined in samples taken >28 days after symptom onset/resolution. The Livzon IgG LFT had the highest specificity (98.5%), followed by the Euroimmun IgG ELISA (96.2%). The specificity for Euroimmun IgA ELISA improved (≥97.5%) using a custom cut-off value (4.0). Conclusions The sensitive and semi-quantitative ELISA assays are most appropriate for serologic detection of SARS-CoV-2 infection in mild cases. Livzon LFT and Euroimmun ELISA had the highest specificity among the IgG assays, making them most suitable for seroprevalence studies. Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is the cause of coronavirus disease 2019 (COVID-19), a novel infectious disease primarily manifesting as pneumonia. The clinical spectrum ranges from asymptomatic cases to severe or even fatal infections. However, in the majority of cases patients are not critically ill (Guan et al., 2020 , Long et al., 2020b . Diagnosis of COVID-19 can be made by direct detection of SARS-CoV-2 RNA, typically by reverse transcription polymerase chain reaction (RT-PCR) methods on respiratory tract samples. Several studies indicate that asymptomatic infections are frequent (Hoehl et al., 2020 , Nishiura et al., 2020 Antibody responses during and after SARS-CoV-2 infection are still poorly understood, especially in asymptomatic or mild cases and the clinical usefulness of serological testing in early stages of COVID19 disease remains uncertain. In general, it appears that SARS-CoV-2 antibodies can be detected within several days to weeks after debut of clinical disease. IgM and IgA appears first and shortly after IgG antibodies to SARS-CoV-2 can be detected in most subjects (Guo et al., 2020 , Long et al., 2020a , Zhang et al., 2020 . However, solid data on the magnitude and duration of IgM, IgA and IgG responses in relation to disease severity are still lacking. Furthermore, accuracy and analytical sensitivity vary with the method and assay used (Lassaunière et al., 2020) . Serological testing in a number of ways complement genomic methods and several testing modalities are available (Okba et al., 2020) . Enzyme-linked immunosorbent assays (ELISA) offer several advantages compared to other serological assays in the way of reduced workload, higher throughput, shorter turnaround time and lower costs. Lateral flow tests (LFTs) have the advantage of being easy to implement and carry the potential of being used in point-of-care testing (Li et al., 2020) . However, concerns have been raised regarding both the sensitivity and specificity of LFTs (Van Elslande et al., 2020) . Reports are emerging that serological testing using validated assays might add diagnostic information in patients suspected for COVID-19 and a negative test for SARS-CoV-2 RT-PCR (Guo et al., 2020 , Zhao et al., 2020 and, further, for the identification of actual or recent asymptomatic infections (Long et al., 2020a , Zhao et al., 2020 . Serological testing might also be used to assess the communal burden of disease and could potentially be an indicator of individual immunity. The purpose of this study was to evaluate six commercially available CE-marked SARS-CoV-2 antibody assays and to evaluate the usefulness of these assays in different disease severities. A gold standard for the assessment of antibodies to SARS-CoV-2 is not available, hence a "true positive" antibody test was defined according to RT-PCR positivity. The inclusion criteria were a positive SARS-CoV-2 RT-PCR test on a respiratory tract sample and the availability of a blood sample for SARS-CoV-2 antibody testing. There were no exclusion criteria. As part of a routine setting during late March to late April 2020, blood samples from all participants had been sent to the Department of Clinical Immunology and Department of Clinical Microbiology at Odense University Hospital for SARS-CoV-2 antibody testing. A total of 57 COVID-19 patients were retrospectively included comprising 98 samples tested for SARS-CoV-2 antibodies. Participants were stratified into three groups. The first group consisted of 19 non-hospitalised cases with mild symptoms, defined as combinations of mild flu-like symptoms, cough and fever. They were active blood donors diagnosed with COVID-19 and recruited for a convalescent plasma project. Samples from these participants were collected at least four weeks after symptom resolution. The other two groups consisted of COVID-19 patients admitted to Odense University Hospital during the study period. One group of 27 patients with moderate COVID-19 disease but requiring hospitalisation at a ward specialising in infectious diseases and one group of 11 patients with severe COVID-19 disease requiring admission to an intensive care unit (ICU). Sensitivity was defined as the proportion of patients correctly identified as having SARS-CoV-2 infections, as initially diagnosed by RT-PCR detection of SARS-CoV-2 in respiratory samples. Receiver operating characteristics (ROC) analyses were performed on the semiquantitative ELISA assays, using all samples from both patients and healthy controls. The detection limit of IgG and IgM assays was compared using a serum sample containing IgM and IgG SARS-CoV-2 antibodies, two-foldly diluted (1 -1,024). The highest titer which yielded a positive reaction was recorded. This study sought to adhere to the STARD criteria for reporting diagnostic accuracy studies (Bossuyt et al., 2015) . Three SARS-CoV-2 antibody ELISAs and three LFTs were evaluated. Selection of assays and tests were according to delivery availability on the Danish market at the time of the study (late March to late April 2020). All testing and evaluation was performed in tertiary care hospital laboratories at the Department of Clinical Immunology and the Department of Clinical Microbiology, Odense University Hospital. Both laboratories are accredited according to the EN:ISO 15189 standard. All samples were analysed using the same lot number for each of the six different assays. WANTAI SARS-CoV-2 IgM ELISA is a CE/IVD-marked assay (Beijing Wantai Biological Pharmacy Enterprise Co., Ltd., China). The IgM ELISA is a solid-phase antibody capture ELISA assay in which polystyrene microwell strips are pre-coated with anti-human immunoglobulin M (anti-μ chain). Any IgM-class antibodies will be captured inside the wells and subsequently detected by the addition of recombinant SARS-CoV-2 antigen (spike protein including the receptor binding domain) conjugated with horseradish peroxidase, thus catalysing a colour reaction proportional with concentration of IgM antibody specific for SARS-CoV-2. The assay was performed according to the manufacturer's instructions. Calibrations were performed in semiquantitative ratios, which are relative measures for the concentration of antibodies in the sample. Results were interpreted using the cut-offs recommended by the manufacturer; ratios <0.9 were interpreted as negative, ≥0.9 to <1.1 as borderline and ratios ≥1.1 as positive. The ELISA tests are classic sandwich ELISAs. Samples containing anti-SARS-CoV-2 IgG, IgA and IgM will bind to the antigen which is a recombinant structural protein (S1-domain including the receptor binding domain) of SARS-CoV-2. Bound antibodies are detected during a second incubation, using enzyme-labelled anti-human IgG or IgA, thus catalysing a colour reaction proportional with concentration of IgA of IgG antibody specific for SARS-CoV-2. Both assays were performed according to the manufacturer's instructions. Calibrations were performed in semiquantitative ratios, which are relative measures for the concentration of antibodies in the sample. Results were interpreted using the cut-offs recommended by the manufacturer; ratios <0.8 were interpreted as negative, ≥0.8 to <1.1 as borderline and ratios ≥1.1 as positive. Three different serological LFTs were evaluated. These were the Acro IgM/IgG LFT (2019-nCoV IgG/IgM Rapid Test Cassette (Acro Biotech, Rancho Cucamonga, CA, USA)), Livzon IgM/IgG LFT (Diagnostic Kit for IgM/ IgG Antibody to Corona Virus (Zhuhai Livzon Diagnostics, Inc., Zhuhai, China) and CTK IgM/IgG LFT (OnSiteTM COVID-19 IgG/IgM Rapid Test (CTK Biotech, Poway, CA, USA)). All tests are CE-IVD approved and they all apply a colloidal gold immuno-chromatography technology to detect either IgM or IgG SARS COV-2 antibodies. All three assays were performed according to the manufacturer's instructions. The LFTs were read by visual inspection 15 minutes after application of test material (one observer per test, but tests were performed by multiple laboratory technicians). Only tests in which the control line was visible were regarded as valid. If a line for IgM and/or IgG was observed, the test was considered positive for that isotype of antibody. Data on analytical sensitivity and specificity were processed by Microsoft Excel (Microsoft, Redmond, WA, USA) and confidence intervals were calculated by using the engine for calculation of exact confidence intervals for binomially distributed data on Statpages (statpages.info/confint.html). Further statistical analysis and generation of boxplot figures was carried out using JMP for Mac ver. 14.2 (SAS Institute, Cary, NC, USA). Comparisons of antibody levels across patient groups were carried out using one-way ANOVA, with a statistically significant difference within a group followed by pairwise post-hoc Tukeys HSD test. Receiver operating characteristics (ROC) charts were created using Prism 8 for macOS (GraphPad Software, LLC San Diego, CA, USA) and used to assess the performance of the ELISA tests. Statistical tests were performed as two-sided tests and p<0.05 was considered statistically significant. One group consisted of non-hospitalised individuals with mild symptoms (n=19, 24 samples), a second group comprised patients with moderate symptoms of COVID19 requiring hospitalisation at a ward specialising in infectious diseases (n=27, 51 samples), and finally, the third group consisted of patients with severe disease requiring admission to an ICU (n=11, 23 samples). For 25/57 (44%) patients a median of 2 (range 2-5) samples were available. For the remainder, 32/57 (56%), only one sample was available. For non-hospitalised individuals, samples were drawn at a median of 45 (range 34-51) days after symptom debut. For patients with moderate COVID-19, samples were drawn median 18 (range 6-31) days after symptom debut. For patients with severe COVID-19, samples were drawn at a median of 13 (range 5-26) days after symptom debut. Diagnostic accuracy was highest for the Wantai IgM ELISA with an area under curve (AUC) of 0.993, 95% confidence interval (CI) 0.98-1.00 (Figure 1. ). In comparison, the Euroimmun assays, IgA ELISA (AUC 0.959, 95% CI: 0.93-0.98) and IgG ELISA (AUC 0.945, 95% CI: 0.92-0.97), had lower AUCs, however, still above 0.94. Assay specificities are shown in Table 1 . For the IgM assays, the highest specificities were found using the Livzon IgM LFT and CTK IgM LFT (1/200, i.e., 97.6% one-sided 95%-upper confidence limit). The Wantai IgM ELISA had a specificity of 2/200 equal to 96.9% (one-sided 95%-upper confidence limit). The Livzon IgG LFT showed the highest specificity of all assays (0/200, i.e., 98.5% one-sided 95%-upper confidence limit). The Euroimmun ELISA IgG had a specificity of 3/200, i.e. 96.2% (one-sided 95%-upper confidence limit). The Acro IgG LFT and CTK IgG LFT both had lower specificities (Table 1. ). Using the manufacturer's recommended cut-off, the specificity of the Euroimmun IgA ELISA assay was 73.2% (one-sided 95%-upper confidence limit). A new cut-off was calculated and established at 4.0 in order to yield a specificity of ≥ 97.5%. When grouping samples according to days since symptom debut, the Wantai IgM ELISA had the highest sensitivity in all time intervals ranging from 86-100% (Table 2a. ). All the IgM LTFs were clearly inferior to the Wantai IgM ELISA, and showed peak detection rates in the time intervals ranging from day 15 -28. In nine samples, SARS-CoV-2 IgM antibodies could not be detected by LFT. Using ELISA, four samples were negative in all assays (Table 2a) . Regarding the Euroimmun IgG ELISA assay the sensitivity increased in relation to days since debut, reaching 96% in samples drawn after > 28 days (Table 2b .). A similar pattern was also seen with the IgG LFTs, however for samples drawn >28 days after symptom onset, there was a decline in sensitivity, especially so for the Livzon IgG LFT (Table 2b. ). In Figure 2 a-c the antibody optical density (OD) ratios for the three semi-quantitative ELISA assays are shown in relation to time of symptom onset. For all isotypes, antibody levels were initially low and then increased steadily over the following weeks. For the Euroimmun IgA ELISA, a plateau was reached already after a few weeks, primarily due to the fact that many samples reached the maximum OD ratio of the IgA ELISA assay. Overall IgM and IgG antibody response in relation to disease severity was assessed on the last sample available from each included patient. Overall the Wantai IgM ELISA appears to be the most sensitive assay for detection of SARS-CoV-2 IgM antibodies across all three patient groups, with detection rates ranging from 79-100% (Table 3a. ). In non-hospitalised subjects with samples taken a median of 45 days after symptom debut, the Wantai IgM ELISA detected antibodies in 79%. The LFTs all showed low sensitivity and failed to detect IgM antibodies in the majority of nonhospitalised cases. The Livzon IgM LFT and Acro IgM LFT both only detected IgM SARS-CoV-2 antibodies in 16% in this group. In samples from patients treated at the ICU (taken a median of 16 days after symptom debut) the Wantai IgM ELISA, Livzon IgM LFT and CTK IgM LFT detect SARS-CoV-2 antibodies in 100% of the samples. The Acro IgM LFT had the weakest performance and only detected antibodies in 82% of severely ill patients (Table 3a. ). Overall, SARS-CoV-2 IgM antibodies could not be detected by LFT in eight samples. Using ELISA, four samples were negative in all assays (Table 3a) . Comparing the IgG assays the Euroimmun IgG ELISA and the Acro IgG LFT show equal sensitivity in the non-hospitalised category detecting antibodies in 95% of subjects (Table 3b. ). The Livzon IgG LFT failed to detect IgG antibodies in more than half of non-hospitalised patients, with a sensitivity at 42%. The CTK IgG LFT also had a low sensitivity in this group, detecting antibodies in 79% of non-hospitalised subjects. Both the Livzon IgG LFT and the Acro IgG LFT detected SARS-CoV-2 antibodies in all severely ill patients, while the Euroimmun IgG ELISA and CTK IgG ELISA detected antibodies in 91% of ICU patients (Table 3b. ). When comparing detection rates using a subset comprising ten hospitalised patients (five from the Department of Infectious Diseases, five from the ICU) for whom two samples taken ≥ 7 days apart were available, the Wantai IgM ELISA showed excellent sensitivity, detecting antibodies against SARS-CoV-2 in all samples (Table 4. ). The earliest sample was taken five days after symptom onset and the last sample were taken after 31 days. The IgM LTFs however, were not able to detect antibodies in all patients. The Livzon IgM LTF and the CTK IgM LTF were able to detect antibodies in all of the late samples, but the Acro IgM LTF failed to detect antibodies in a late sample from an ICU patient (Table 4. ). The Euroimmun IgG ELISA and IgG LFTs, except the CTK assay, were able to detect IgG antibody in all of the late samples (Table 4. ). However, detection ability with regard to the early samples varied (Euroimmun IgG ELISA 5/10, Livzon IgG LFT 6/10, Acro IgG LTF 9/10 and CTK IgG LFT 7/10). Using the manufacturer's recommended cut-off for the Euroimmun IgA ELISA, all samples tested positive. When assessing anti-SARS-COV-2 IgM, IgG and IgA by ELISA OD ratios across the three patient categories (using all 98 samples), the general pattern of mean antibody level (OD ratio) for all three ELISA assays were in the order non-hospitalised subjects < hospitalised patients < ICU patients (Figures 3a-c. ). In detail, for IgM, mean antibody levels determined by OD ratios were 9.2 (non-hospitalised subjects), 18.6 (hospitalised patients) and 27.2 (ICU patients) with all differences being statistically significant (p<0.0001 -p=0.021) (Figure 3a. ). For IgG, mean antibody levels were 4.0 (non-hospitalised), 5.4 (hospitalised) and 6.3 (ICU), but not statistically significantly different (p=0.14) (Figure 3b. ). Finally, for IgA, mean antibody levels were 4.4 (non-hospitalised), 7.4 (hospitalised) and 9.5 (ICU) with all differences being statistically significant (p<0.0001 -p=0.016) (Figure 3c.). The Wantai IgM ELISA had the lowest detection limit, yielding a positive reaction at 64-fold sample dilution. The IgM LFTs all had higher (i.e., inferior) detection limits; the CTK IgM LFT showing a positive reaction at a maximum of 16-fold sample dilution, the Livzon IgM LFT at eight-fold sample dilution, and the Acro IgM LFT at four-fold sample dilution. Regarding the IgG assays, the Acro IgG LFT showed the lowest detection limit, detecting a positive reaction at 64-fold sample dilution. Euroimmun IgG ELISA and CTK IgG LFT both showed positive reactions at 32-fold sample dilutions. Livzon IgG LFT, yielded a positive reaction at eight-fold sample dilutions. Overall, number of samples presenting antibodies against SARS-CoV-2 are related to disease severity and to days since symptom debut, but differ amongst type of assay. The Wantai IgM ELISA was the best assay for detection of SARS-CoV-2 antibodies in all three disease severities, with detection rates ranging from 79-100% and showed excellent diagnostic accuracy (AUC 0.993). While all IgM LFTs failed to detect SARS-CoV-2 antibodies in the majority of non-hospitalised subjects, the Wantai IgM ELISA detected antibodies in 79% of these cases. In particular, the Livzon IgM LFT and Acro IgM LFT both had unacceptably low detection rates at 16%. The Euroimmun IgG ELISA detected SARS-CoV-2 antibodies in 95% of non-hospitalised subjects, as did the Acro IgG LTF. The Livzon IgG LFT had the lowest detection rate (42%) in non-hospitalised subjects. To our knowledge this is one of the first accuracy studies including samples from both nonhospitalised subjects with mild disease and patients with more severe disease, thus, providing the opportunity to assess test performance in different disease severities. Samples from nonhospitalised individuals were taken 45 days (median, range 34-51) after symptom debut. In a real life setting, this is typically when there is a need for serologic testing. High sensitivity, as seen with the Wantai IgM ELISA and the Euroimmun IgA and IgG ELISA, is important in late presenting cases with milder symptoms and in patients suspected of COVID-19 despite negative SARS-CoV-2 RT-PCR (Sethuraman et al., 2020) . Furthermore, all ELISA assays tested have the advantage of being semi-quantitative, providing an estimate of antibody levels through OD ratios. All LFTs have the inherent limitations of being qualitative tests and being subject to inter-observer variation. The performance and applicability of each test depends on the setting in which it is used. In a low prevalence screening situation, an assay with high specificity is required. Otherwise the positive predictive value will be unacceptably low. This can be achieved by using a highly specific assay, or, when suitable, raising the cut-off limit for the assay used. The Livzon IgG LFT showed a high specificity (98.5%) and an adequate sensitivity in the moderately and severely ill patients, but low sensitivity when applied in subjects with only mild symptoms. Hence none of the LFTs are suitable in a low-prevalence setting where each individual result is important, i.e. diagnostics, however if the aim is communal seroprevalence, the Livzon IgG test could be a suitable assay due to the high specificity (Erikstrup et al., 2020) . Antibody levels (OD ratios) were initially low and then increased steadily over time. Also, the antibody levels of IgM, IgG and IgA correlated very well with clinical disease severity. Seow et al. has demonstrated that the magnitude of the neutralizing antibody response is dependent on disease severity (Seow et al., 2020) . In our study, samples from non-hospitalised cases were taken a median of 27 or 29 days later than samples from hospitalised patients, but this fact is not the sole explanation for the difference in antibody strength. The longevity of SARS-CoV-2 spike IgG antibodies, also in cases with mild disease, have been described (Wajnberg et al., 2020) . The general pattern of mean antibody level for all three assays were in the order non-hospitalised subjects < hospitalised patients < ICU patients, significantly so for IgA and IgM, but not for IgG antibodies. In particular, IgA antibody levels are significantly higher for severely ill patients treated in the ICU. High anti-SARS-CoV-2 antibodies have previously been linked to disease severity (Hsueh et al., 2004 , Tan et al., 2020 , Zhao et al., 2020 . We speculate that a highly positive IgA antibody could be an indicator of disease severity. Importantly, whether SARS-CoV-2 antibody levels correlate to protective immunity is yet to be determined. IgA antibodies are detected in early and late stages of SARS-CoV-2 infection, and the detection of IgA antibodies can have an added value in diagnosing SARS-CoV-2 (Guo et al., 2020) . Importantly, the Euroimmun IgA ELISA should not be used for screening purposes, due to a high rate of false positives when using the manufacturers recommended cut-off. As we have shown, multiple thresholds for positivity can be considered for different settings, giving some flexibility in use of the assay. The specificity for Euroimmun IgA ELISA was improved (≥ 97.5%) using a custom cut-off value (4.0). However, in high-prevalence settings, i.e. among hospitalised patients, test sensitivity should be prioritised and the manufacturers cut-off (1.1) can be used instead. This study has several limitations. At the start of the pandemic, serologic assays were pushed on the market in a haste. Not all assays performed adequately and some of these assays obviously need be optimized further, independently validated, and probably used in algorithm formats to achieve the highest possible accuracy for decision making. The retrospective nature means that samples were not collected at pre-determined time intervals and clinical data on patients are scarce. Moreover, number of subjects and samples included are few. Routine serologic testing using validated anti-SARS-CoV-2 assays could prove beneficial when used in the correct settings. The three LFTs evaluated in this study are not suitable for diagnosis or monitoring of SARS-CoV-2 infection, especially not in mild cases. Instead, sensitive ELISA assays will perform better in these settings. For evaluation of seroprevalence, IgG tests with high specificity, either ELISA or LFT, would be appropriate. Table 4 . Changes in reactivity of SARS-CoV-2 antibody assays in 10 hospitalised patients (#6-10 represent ICU patients). Samples were drawn at least 7 days apart. For the lateral flow tests (LFTs), the intensity of the test band was read and reported as "-", "(+)" or "+". # Euroimmun IgA assay cut-off set at 1.1 (manufacturer's instructions). ELISA: enzyme linked immunosorbent assay, LFT: lateral flow test. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies Estimation of SARS-CoV-2 infection fatality rate by real-time antibody screening of blood donors Clinical Characteristics of Coronavirus Disease 2019 in China Profiling Early Humoral Response to Diagnose Novel Coronavirus Disease (COVID-19) Infection in Returning Travelers from Wuhan, China Chronological evolution of IgM, IgA, IgG and neutralisation antibodies after infection with SARS-associated coronavirus Evaluation of nine commercial SARS-CoV-2 immunoassays Development and clinical application of a rapid IgM-IgG combined antibody test for SARS-CoV-2 infection diagnosis Antibody responses to SARS-CoV-2 in patients with COVID-19 Clinical and immunological assessment of asymptomatic SARS-CoV-2 infections Estimation of the asymptomatic ratio of novel coronavirus infections (COVID-19) Severe Acute Respiratory Syndrome Coronavirus 2-Specific Antibody Responses in Coronavirus Disease Longitudinal observation and decline of neutralizing antibody responses in the three months following SARS-CoV-2 infection in humans Interpreting Diagnostic Tests for SARS-CoV-2 Viral Kinetics and Antibody Responses in Patients with COVID-19 Diagnostic performance of seven rapid IgG/IgM antibody tests and the Euroimmun IgA/IgG ELISA in COVID-19 patients Robust neutralizing antibodies to SARS-CoV-2 infection persist for months Longitudinal Change of SARS-Cov2 Antibodies in Patients with COVID-19 Antibody responses to SARS-CoV-2 in patients of novel coronavirus disease 2019 This research did not receive any specific grant from funding agencies in the public, commercial or not-for-profit sectors. Not required. The authors declare no conflict of interest. The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.