key: cord-0806619-1r7ej0g2
authors: Cota, Gláucia; Freire, Mariana Lourenço; de Souza, Carolina Senra; Pedras, Mariana Junqueira; Saliba, Juliana Wilke; Faria, Verônica; Alves, Líndicy Leidicy; Rabello, Ana; Avelar, Daniel Moreira
title: Diagnostic performance of commercially available COVID-19 serology tests in Brazil
date: 2020-10-09
journal: Int J Infect Dis
DOI: 10.1016/j.ijid.2020.10.008
sha: 92a21db16150efbf200ff50fbf4b16de2ff954c8
doc_id: 806619
cord_uid: 1r7ej0g2

Timely and accurate laboratory testing is essential to manage the global COVID-19 pandemic. Reverse transcription-polymerase chain reaction remains the gold-standard for SARS-CoV-2 diagnosis, but several practical issues limit the test use. Immunoassays have been indicated as an alternative for individual and mass testing. Objectives: To access the performance of twelve serological tests for COVID-19 diagnosis. Methods: We conducted a blind evaluation of six lateral flow immune assays (LFIAs) and six enzyme-linked immunosorbent assays (ELISA) commercially available in Brazil to detect anti-SARS-CoV-2 antibodies. Results: Considering patients with seven or more days of symptoms, the sensitivity ranged from 59.5% to 83.1% for LFIAs and from 50.7% to 92.6% for ELISAs. For both methods, the sensitivity increased with clinical severity and days of symptoms. The agreement between LFIA performed in digital blood and serum was moderate. Specificity was, in general, higher for LFIAs than for ELISAs. Infectious diseases prevalent in the tropics, such as HIV, leishmaniasis, arboviruses and malaria, represent conditions with the potential to cause false positive results, which significantly compromises their specificity. Conclusion: The performance of immunoassays was only moderate, affected by the duration and clinical severity of the disease. Absence of discriminatory power between IgM/IgA and IgG has also been demonstrated, which prevents the use of acute phase antibodies for decisions on social isolation.

City, Hubei Province, China, in December 2019, followed by an outbreak across the world. On March 12, World Health Organization (WHO) declared COVID-19 a global pandemic (1) and, four months later, more than 12 million cases and 550,000 deaths have already been reported worldwide. In July 2020, Brazil exceeded the mark of 1,800,000 cases notified (2) , at the same time exceeding the total of 70,000 deaths confirmed by the disease, becoming the second country in the Americas with more cases, after the United States, and the epicenter of the pandemic in Latin America (3) .

Preventing transmission to control the spread of SARS-CoV-2, from symptomatic and asymptomatic individuals (4) , is the main objective of any containment strategy. The approach of testing, tracking and tracing has become a central tool to achieve this J o u r n a l P r e -p r o o f objective (5) . However, the response to the coronavirus disease pandemic has been hampered by a lack of aggressive testing for the infection in several regions of world.

To date, assays based on the reverse-transcription polymerase chain reaction (RT-PCR) in respiratory samples are the gold standard for COVID-19 diagnosis (6) . This diagnostic strategy has been limited by significant logistics and capacity constraints, ranging from the short time of high viral excretion in respiratory secretions, availability of well-equipped laboratories, trained personnel, reagents, swabs used for the collection of nasopharyngeal specimens and personal protective equipment for health care providers collecting samples. Thus, RT-PCR is particularly challenging in resourcelimited settings. Additionally, RT-PCR execution is relatively time consuming and highly dependent on the pre-analytical phase.

In this context, numerous immunological tests, based on antigens or antibodies detection and including point-of-care or conventional platforms, have recently become available and approved for use worldwide. The tests developed to detect SARS-CoV-2 antibodies are typically based on lateral flow immune assays (LFIAs), enzyme-linked immunosorbent assays (ELISAs) or chemiluminescent immunoassays (CLIAs). Unlike tests based on viral detection, whose diagnostic window is short and related to the period of viral excretion, serological tests would have the advantage of being longer-lasting markers of infection, which has classically been used as a tool in assessing the population dissemination of infections.

Currently, the available tests predominantly target antibodies to the main surface proteins of the novel coronavirus (7) . In theory, the serological strategy, and a point-ofcare approach based on rapid tests, would have the potential to significantly improve the current testing capacity for COVID-19. Serology is easier to perform, requiring less technical expertise and equipment and has a much lower unitary cost than RT-PCR J o u r n a l P r e -p r o o f assays. The samples are blood collected in tubes or taken from digital pulp, which pose a lower potential risk to the health care staff. Serology can be performed in a basic clinical laboratory and community settings, thereby reaching a wider application. These potential advantages have been sufficient to encourage government from several countries, especially those with limited resources, and employers in the private sector, to acquire and use serological tests on a large scale during the COVID-19 pandemic, both as a diagnostic tool and as a marker of previous infection and guarantee of immunity. However, the serodiagnostic power of the specific IgM, IgA and IgG antibodies against SARS-CoV-2 remain largely uncertain, such as the relationship between the presence of antibodies and presence of immunity against re-infection (8).

Although serologic tests contribute little to urgent decisions on social withdrawal and quarantine, from a public health perspective, serological analysis could be useful to estimate epidemiological variables, such as the attack rate and case fatality rate, which are necessary to assess the virus community transmission and its burden (9) .

Regardless of the intended use, the first stage of any decision on the implementation of serological tests in the context of the COVID-19 pandemic is the careful analysis of their performances over the various stages of the infection, in different specimens and their specificity, challenged in the face of other clinical conditions. Here, we aimed to describe the accuracy of serological assays for COVID-19 registered in Brazil up to May 2020, the comparative performance of rapid tests performed in digital whole blood and serum and between patients with severe and mild clinical manifestations, in addition to the positivity of the different antibodies among patients with less than seven days, between 7 and 14 days and more than 14 days after the onset of symptoms.

The panel-based study comprised 289 serum samples from 173 symptomatic patients with confirmed SARS-CoV-2 infection and 116 negative controls. All the cases (SARS-CoV-2 positive) were confirmed by RT-PCR testing of nasopharyngeal or oropharyngeal swabs and had their clinical condition and demography data compiled.

The RT-PCR tests used for case confirmation were performed according to the protocols proposed by Center for Diseases Control/USA (10) or Charité Hospital/Germany (11) , both accepted by World Health Organization (12) . The negative control sera were all obtained before January 2020, the milestone of the introduction of the new coronavirus in Brazil, from patients with serological markers for other infectious or no infectious diseases. Only one sample per individual was included in this panel.

Sera from confirmed cases of SARS-CoV-2 infection were provided by the Minas Gerais State Department of Health, which is responsible for collecting and storing a biorepository since the beginning of the COVID-19 pandemic. Additionally, the performance of LFIAs performed in blood and serum was compared in a group of 32 patients who consented to be double tested. The SARS-CoV-2 RT-PCR confirmed cases included hospital patients and outpatients. For each case, information about the presence of acute respiratory distress syndrome (ARDS) presence, according to the definition adopted in Brazil ( ≥ 30 breaths/min or an oxygen saturation ≤93% at rest), was registered. Negative control sera were collected before the emergence of the novel coronavirus in previous studies and were kindly provided by their legal guardian, with authorization from the Ethical Board of Instituto René Rachou, according to the J o u r n a l P r e -p r o o f Brazilian legislation for research with humans. Only serum samples were available for the control group, for this reason, no control was submitted to SARS-CoV-2 RT-PCR.

Due to the small volume of serum available from patients with malaria, samples from ten patients infected by Plasmodium vivax and nine patients with malaria by 

The minimum sample required for this validation was estimated through one-sample proportion test, using Statistics/Data Analysis software (Stata), version 11.0. As premises were considered a power of 80% and significance of 5% to reproduce the sensitivity and specificity of tests with an expected binomial exact 95% confidence limits, based on the lowest performance rates reported by the manufacturers to ANVISA up to May 18, 2020 (86% for sensitivity and 98% for specificity). Based on that, it was defined as the minimum of 149 cases with seven or more days of symptoms and 116 controls. Additionally, the minimum number of 20 tests performed on digital blood and serum from SARS-CoV-2 confirmed patients was estimated as sufficient to identify a minimum difference of 20% between the sensitivity in the two clinical specimens. Two of the manufacturers involved in this validation provided fewer tests than requested, and J o u r n a l P r e -p r o o f the evaluation was carried out with a proportional sub-group of case samples and negative controls, chosen randomly.

A search for diagnostics for COVID-19 was carried out with records in force at the Brazilian National Health Surveillance Agency (ANVISA) through the Agency's website (https://consultas.anvisa.gov.br/#/saude/). The search strategy was based on the terms "COVID 19", "SARS", "nCOV", "COV" and "coronavirus" and was carried out on May 18, 2020. Sixty-seven serological tests registered in Brazil to diagnose SARS-CoV-2 were identified: 55 lateral flow immunoassays (LFIAs), six ELISA assays, four chemiluminescence and two immunofluorescence tests. Five manufacturers did not present any commercial contact information, thirty-eight manufacturers did not respond to contact, and other three refused to participate. All the companies responsible for producing the tests identified using the commercial contact available were invited to participate in this validation. By the end of June, nine companies had sent kits for validation and nine others were committed to donating the tests, if not yet received.

Thus, twelve registered and commercially available serological tests for SARS-CoV-2 diagnosis were included in this analysis and their main characteristics are shown in Table 1 . There were six LFIA and six ELISA tests.

Among the LFIA tests, only one exhibits a total antibody detection line, while in the other five the cassette display two test lines (M and G lines) and a quality control line (C line). All the kits used capture reaction to detect SARS-CoV-2 antibodies and were based on the colloidal gold-labeled immunochromatography principle and one-step method with results obtained within 10-30 minutes, using whole blood, serum or plasma samples. Briefly, the sample was absorbed by capillary action and mixed with the 

The serum samples were randomly coded and kept frozen at -70ºC until needed, and then was thawed for ten minutes at room temperature and homogenized before testing.

The tests were carried out following each manufacturer's instructions strictly. To avoid comparison between tests, all the samples were submitted to a test before moving on to the next test. The reproducibility of the LFIA kits was assessed using the interpretation of the results by three independent observers using the Kappa index and the final result defined was that indicated by at least two of the three readers. For one of the LFIA kits, the result was obtained using a micro reader provided by the manufacturer. For LFIA blood-serum comparison, approximately 10% of the SARS-CoV-2 confirmed patients represented in this panel was consecutively recruited to donate capillary blood for testing, until reaching the minimum of 20 tests performed of each brand test.

The performance parameters of interest were sensitivity, specificity and accuracy, 

The patients whose serum samples were tested in this study were diagnosed with Tables 2 and 3 summarize the sensitivity by serological test and immunoglobulin class detected. For LFIA tests, the sensitivity for IgM ranged from 13.3% to 72.3% and that for IgG, ranged from 51.4 to 65.9%. For all except one LFIA tests, the sensitivity of the IgG detection alone was numerically higher than that observed for IgM band and for all of them, the highest detection rates were observed by combining the IgM and IgG results, ranging from 52.6% to 75.1% (Table 1) . Among the ELISAs, a test based on the IgA/IgM detection exhibited the highest sensitivity (90.2%, 95%CI 84.9-93.8%). For IgG, the sensitivity for ELISAs ranged from 58.7% to 76.8% (Table 3) . Considering only patients with seven or more days of symptoms, the sensitivity ranged from 59.5% to 83.1% for LFIA and from 50.7% to 92.6% for ELISAs ( Table 4 ). As expected, the sensitivity for patients with less than seven days of symptoms was in general poor, up to 40% for all except one ELISA test. The sensitivity tends to increase with the number of days from the day of symptoms onset. However, there was substantial overlap between the sensitivity 95% confidence interval for the groups of confirmed cases with 7 to 14 days and more than 14 days of symptoms (Table 5) .

For all LFIA tests and three of six ELISAs, the sensitivity was higher among patients presenting with ARDS than among those presenting with mild symptoms (Table 6 ).

Except for one LFIA, which presents an exceptional low specificity (81%, 95%CI 72.9%-87.1%), the specificity of all other LFIAs was in general high, varying between 97.4% and 100% (Table 4) . Overall, sensitivity for ELISA assays was higher than for LFIA. Excluding one IgG based ELISA test, which presented a sensitivity rate of 58.7%, the rates varied from 66.9% to 92.6%. By contrast, the specificity for ELISA tests was in general lower than that for LFIA, except for the same test presenting the J o u r n a l P r e -p r o o f lowest sensitivity referred above, which exhibited the highest specificity among the ELISA tests (95.8%). Agreement between the results of LFIA performed in digital blood and serum varied markedly among different commercial kits, from perfect to only slight agreement (Table 7) . Among the 116 control sera, only 21 did not show reactivity to any of the 12 tests evaluated, 53 were positive in one test, 27 in two tests, 10 reacted positively in three tests and five controls showed positive reaction in four different tests.

The patients of control group whose sample reacted falsely in more than three different SARS-Cov-2 serological tests had serum markers to HIV, dengue, zica, Chagas disease, syphilis, toxoplasmosis or parasitological confirmation of visceral leishmaniasis or malaria.

This is the first study to assess comparatively the clinical performance of the serological tests available to diagnose SARS-CoV-2 in Brazil. Although some systematic reviews have already been published on the subject (14) (15), none has included data from Brazil, the current epicenter of the COVID-19 pandemic in Latin America. Local accuracy data based on real scenarios are essential considering the marked regional differences in the performance of the tests. In the case of SARS-CoV-2 serological tests, this information is especially relevant to the current reality of Brazil, a developing country that faces serious budgetary constraints and that has been performing suboptimally in relation to its mass testing capacity. By contrast, several successful strategies implemented worldwide, such as aggressive testing and isolation, have promoted transmission control (16) (17) . In this sense, the inverse association between testing capacity and mortality from COVID-19 has been consolidated as evidence of the impact of the isolation of those infected and the tracing and quarantining of their presented here, the highest sensitivity rate observed after14 days of symptoms reaches only a moderate level, just over 90% for a few tests in the best scenario. Thus, in regions with SARS-CoV-2 prevalence below 10%, a reality in many regions of the world, the positive predictive value (PPV) of these tests remain below 80%, that is, these tests will produce around 20% of false-positive results if they have a very high specificity. Additionally, for tests with specificities lower than 95%, this moderate sensitivity will generate an even lower PPV, even with a disease prevalence above 20%.

However, considering that a set of clinical manifestations could be used as disease suspicion criteria increasing the pre-test probability, in theory, immunoassays could play a complementary function to RT-PCR, enhancing COVID-19 detection sensitivity and accuracy (21), at rates that need to be established. An important observation presented here is a test performance in general lower than that described by others (24, 25) . The main difference between this and those studies lies on the studied population.

Here, approximately 40% of the cases did not meet the Covid-19 severity criterion adopted in Brazil while in most of the first studies only hospitalized patients were tested. Furthermore, the test's performance in the two groups stratified according to the clinical criterium was significantly different, reinforcing the link between clinical J o u r n a l P r e -p r o o f severity and positivity in serological tests. Other factors as sample size issues and genetic specificities could also justify that difference. On the other hand, as a common result among validation studies, immunoassays still will produce delayed information, if the critical period of viral transmission is considered (26) . For a more accurate performance, serological tests should be used after two weeks after symptoms onset in a context of high probability. Unlike PCR-based tests, serology cannot be used to confirm the presence of the SARS-CoV-2 virus, making its use limited for clinical decision-making or as a reinforcement in the recommendation for social isolation.

There is even less evidence to support serological testing of asymptomatic individuals, as proposed for the screening of contacts of COVID-19 confirmed cases or in the supposed assessment of protective immunity. The antibody presence and circulating titers may exhibit behavior different from that observed in symptomatic infection.

considering the performance reported here and the still low SARS-CoV-2 prevalence in general, except in few hotspots, we can expect many more false-positive than true positive results.

In relation to the choice between LFIA and ELISA tests, in addition to performance, logistics issues and total cost involved should be considered. To assist in this decision, cost-effectiveness analysis needs to be conducted and should guide more accurate decisions applied to different scenarios where pre-test probability or disease prevalence are estimated.

The inadequacy of the use of IgA and IgM antibodies as markers of contagiousness need to be highlighted. Our results revealed that both increase directly with the number of days of symptoms, reaching the highest rate among samples from patients with more J o u r n a l P r e -p r o o f than 14 days of symptoms, a period in which the SARS-CoV-2 infectivity is considered low (27) .

Specificity was homogeneously high for all LFIA tests, except for one test, assembled and packaged in Brazil but imported from a manufacturer based in the United States, where the kit had its FDA (Food and Drug Administration) license revoked in June 2020, in addition to more than 70 other tests, because of a poor performance detected by independent analyses (28).

Although low specificity was not a problem for most LFIAs, data for ELISAs differ significantly from that described in the package inserts of the tests and in relation to other accuracy studies carried out in China and in countries in the northern hemisphere.

Infectious diseases prevalent in the tropics, such as leishmaniasis, arboviruses and malaria, were for the first time described as cause of false-positivity in tests for SARS-CoV-2, which raises concern about its use in these regions. On the other hand, there were few samples from patients with acute respiratory symptoms enrolled in the control group, which would be the real control for this validation. Even so, other studies contemplating samples from acute respiratory patients, including other endemic coronavirus confirmed cases, have revealed similar specificity rates (14, 29, 30) .

Additionally, another group of patients possibly prone to cross-reactivity are those with chronic autoimmune diseases, as indicated by some previous observations with SARS-CoV, an association still not confirmed for SARS-CoV-2 (31).

High specificity, in turn, is one of the most important properties required for a test to be used in epidemiological surveys. Estimation of the extent of the population that has already been infected in the community is essential to understand the spread of the epidemic and the main characteristics of the virus, its attack rate, its lethality and the impact of the various prevention and control interventions. These parameters would also J o u r n a l P r e -p r o o f be useful to monitor the resumption of social and economic activities. A limiting factor, however, is the lack of knowledge of the longevity of these antibodies, an issue that will require studies with longer observation periods. In this panel, only 19 patients were between 31 and 60 days and another 13 had more than 60 days since the onset of symptoms. This small sample of evaluated patients does not allow us to confirm the lack of differences in the positivity rates of the tests over time.

Another limiting factor for the use of LFIA as a point-of-care test is the heterogeneity observed between the results of tests performed using fingerstick whole blood and serum, with the kappa varying from 0.2 to 1.0. This observation cautions the possibility of lower performance using blood, which is variable among kits, and an undeniable commitment to the more striking potential advantages of LFIA: agility and decentralization in mass testing. Few studies until now have addressed this issue because most have presented the results of tests performed solely on serum. In addition, the sample sizes evaluated were generally small and the results conflicting (30, 32) .

Finally, the most controversial point regarding the use of immunoassays use is the lack of robust evidence of a correlation between circulating antibodies and acquired immunity (33) , that is, whether the antibodies detected are protective for a significant The accumulated experience with SARS-CoV, another coronavirus with strong genetic similarity to the current SARS-CoV-2, and involved in an outbreak in 2002, provides some insights on immunity. Specifically, the presence of antibodies has been extended for at least three years, being more intense among patients with the most severe forms, with progressive and significant reduction in the neutralizing antibody over time (35) .

Thus, at this point, any use of serology as a marker of immunity and criteria to allow or prevent the resumption of social life is only speculation and should not be recommended.

In summary, our observations revealed marked differences among the serological tests registered in Brazil. Generally, the sensitivity was only moderate, with insufficient performance for use before seven days of symptoms, as expected for a method based on the search for antibodies. The sensitivity rates reach around 80% to 90% for LFIA and ELISA after 14 days from the onset of symptoms, respectively, confirming that immunoassays are not suitable tools for screening SARS-CoV-2 virus infection in the general population, except for regions presenting high prevalence rates, over 20%. Of the three uses recommended so far, as a diagnostic method for acute cases, as a marker of immunity to allow the resumption of social life and as an instrument for measuring viral dissemination in epidemiological studies, only the last one seems to be justified. Seroprevalence can play an important role in the understanding of COVID-19 spread, however, to estimate the extent of the population that has already been infected in the community, the estimated prevalence rates need to be adjusted by test sensitivity and specificity. As an epidemiological tool, seroprevalence could still be helpful to assess the impact of different collective interventions on different demographics retrospectively. As a diagnostic tool for symptomatic patients, serology represents delayed information, greatly limiting its role in decision making. The inaccuracy of detecting IgM or IgA, as markers of active infectivity, was also confirmed by our results. Further data need to be gathered correlating antibody detection and protective immunity, besides the duration of protection. 

Coronavirus disease (COVID-2019) situation reports

Published online at OurWorldInData.org. 2020

Asymptomatic carriage and transmission of SARS-CoV-2: What do we know?

Report of the WHO-China Joint Mission on

COVID-19). World Health Organisation

Laboratory Diagnosis of COVID-19: Current Issues and Challenges

Genomic characterisation and epidemiology of 2019 novel coronavirus: implications for virus origins and receptor binding

Virological assessment of hospitalized patients with COVID-2019

Centers for Disease Control and Prevention. A CDC 2019-Novel Coronavirus (2019-nCoV) Real-Time RT-PCR Diagnostic Panel

Detection of 2019 novel coronavirus (2019-nCoV) by real-time RT-PCR

Laboratory testing for coronavirus disease

COVID-19) in suspected human cases: interim guidance

An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers

Antibody tests for identification of current and past infection with SARS-CoV-2

Diagnostic accuracy of serological tests for covid-19: systematic review and metaanalysis

Interrupting transmission of COVID-19: lessons from containment efforts in Singapore

Covid-19 in South Korea -Challenges of Subclinical Manifestations

Temporal profiles of viral load in posterior oropharyngeal saliva samples and serum antibody responses during infection by SARS-CoV-2: an observational cohort study

Variation in False-Negative Rate of Reverse Transcriptase Polymerase Chain Reaction-Based SARS-CoV-2 Tests by Time Since Exposure

Department of Health & Social Care. Coronavirus (COVID-19) scaling up our testing programmes

Analysis of adjunctive serological detection to nucleic acid test for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection diagnosis

Assessment of SARS-CoV-2 serological tests for the diagnosis of COVID-19 through the evaluation of three immunoassays: Two automated immunoassays (Euroimmun and Abbott) and one rapid lateral flow immunoassay (NG Biotech)

Clinical performance of different SARS-CoV-2 IgG antibody tests

Dynamic profile of RT-PCR findings from 301 COVID-19 patients in Wuhan, China: A descriptive study

Predicting infectious SARS-CoV-2 from diagnostic samples. Clin Infect Dis. 2020. 28. Food and Drug Administration (FDA News). FDA Nixes Chembio's EUA for COVID-19 Antibody Test

The receptor binding domain of the viral spike protein is an immunodominant and highly specific target of antibodies in SARS-CoV-2 patients

Test performance evaluation of SARS-CoV-2 serological assays. medRxiv

Cross-reaction of SARS-CoV antigen with autoantibodies in autoimmune diseases

Development and clinical application of a rapid IgM-IgG combined antibody test for SARS-CoV-2 infection diagnosis

An evaluation of COVID-19 serological assays informs future diagnostics and exposure assessment

Retest positive for SARS-CoV-2 RNA of "recovered" patients with COVID-19: Persistence, sampling issues, or re-infection?

Protective Adaptive Immunity Against Severe Acute Respiratory Syndrome Coronaviruses 2 (SARS-CoV-2) and Implications for Vaccines

COVID-19 IgG IgM

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.