key: cord-0953137-coxoqp2t
authors: Bermejo-Pelaez, D.; Marcos-Mencia, D.; Alamo, E.; Perez-Panizo, S.; Mousa, A.; Dacal, E.; Lin, L.; Vladimirov, A.; Cuadrado, D.; Mateos-Nozal, J.; Galan, S.; Romero-Hernandez, S.; Canton, S.; Luengo-Oroz, M.; Rodriguez-Dominguez, S.
title: Performance of an artificial intelligence-based smartphone app for guided reading of SARS-CoV-2 lateral-flow immunoassays
date: 2022-02-19
journal: nan
DOI: 10.1101/2022.02.16.22271042
sha: 6281e8d60f39334690b16f4ff411f85396518eb5
doc_id: 953137
cord_uid: coxoqp2t

Objectives: To evaluate an artificial intelligence-based smartphone application to automatically and objectively read rapid diagnostic test (RDT) results and assess its impact on COVID-19 pandemic management. Methods: Overall, 252 human sera from individuals with PCR-positive SARS-CoV-2 infection were used to inoculate a total of 1165 RDTs for training and validation purposes. We then conducted two field studies to assess the performance on real-world scenarios by testing 172 antibody RDTs at two nursing homes and 92 antigen RDTs at one hospital emergency department. Results: Field studies demonstrated high levels of sensitivity (100%) and specificity (94.4%, CI 92.8-96.1%) for reading IgG band of COVID-19 antibodies RDTs compared to visual readings from health workers. Sensitivity of detecting IgM test bands was 100% and specificity was 95.8%, CI 94.3-97.3%. All COVID-19 antigen RDTs were correctly read by the app. Conclusions: The proposed reading system is automatic, reducing variability and uncertainty associated with RDTs interpretation and can be used to read different RDTs brands. The platform can serve as a real time epidemiological tracking tool and facilitate reporting of positive RDTs to relevant health authorities.

To control COVID-19 pandemic, timely and accurate early-detection strategies of 2 SARS-CoV-2 infections have been critical to slow down the spread of the virus. The 3 use of rapid diagnostic tests (RDTs), both for detection of antibodies and antigens, has 4 contributed to improve COVID-19 testing capacity reducing costs of diagnosis and 5 allowing for fastest results (1) . First, COVID-19 RDTs were intended to be used just by 6 professional healthworkers, who have extensive experience in the use of this tool for the performance of the AI-based system for reading both COVID-19 antibodies and 1 antigen RDTs in real-world scenarios. Ethics approval for the study was obtained from 2 Rapid Test Device (Abbott) (6+,6-) were also included to train the algorithm to read not 1 6

only three-band tests (such as the COVID-19 antibody tests used in this study) but also 1 7

two-band RDTs such as COVID-19 antigen tests. The entire training dataset consisted 1 8

of 3614 images. 1 9

For collecting the independent validation dataset, 240 human sera samples 2 0 independent from the ones used for training were used to inoculate 720 COVID-Ab 2 1

RDTs (each serum was tested in triplicate using the aforementioned brands). The . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) Each RDT was visually read by multiple observers (from 3 to 5) and the ground truth 1 was established as the majority result from the total of analyzers. Each inoculated RDT 2 was digitized by using the TiraSpot mobile app (Spotlab, Madrid, Spain) for guided and 3 standardized acquisition ensuring correct positioning of RDTs in the image and using a 4 total of 9 smartphone models. Results data was uploaded to a web platform. Field validation studies 7

The workflow for the field studies was as follows: a health professional digitized the 8 RDTs by using the app, was asked for recording the visual interpretation of the test 9 result, images were uploaded to the telemedicine platform and processed by the AI 1 0

algorithm, and discrepancies between the interpretation made by the health professional 1 1

and that obtained by the algorithm were subsequently reviewed by an external health 1 2

professional through the platform. 1 3

The first field study used the system as part of a seroprevalence study conducted in 1 4

two nursing homes in Madrid (Spain). A total of 172 vaccinated health care personnel 1 5

were included in this study from which a finger-prick blood sample was taken and 1 6

inoculated into SARS-CoV-2 Rapid Antibody Test (Roche). A trained nurse digitized 1 7

the RDTs and record their result using the application. 1 8

The second field validation field study tested the system to read also COVID-19 1 9

antigen tests composed of two bands (Panbio COVID-19 Ag Rapid Test Device, 2 0

Abbot). This study was carried out at the Emergency Department of the Ramón y Cajal 2 1

Hospital (Madrid, Spain), where 92 individuals' nasal swabs were inoculated in antigen 2 2 tests, and digitized by experienced health professionals using the app.

All images were acquired in very diverse real-world conditions (including different 2 4

environmental illuminations and shades). 2 5

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted February 19, 2022. ;

1 Results 2

All images acquired with the app were uploaded to a cloud platform where the AI 4 algorithm processed the photographs to predict its result interpretation. As shown in 5 Table 1A , when comparing the visual interpretations (used as ground truth) against the 6 AI algorithm, the performance was high for all brands of RDTs tested, obtaining a mean 7 sensitivity and specificity of 98% and 100% respectively for detecting the IgG band and 8 a mean sensitivity and specificity of 80% and 89% for the detection of the IgM band. negative), we only found 9 discrepancies between test result interpretation made by the 1 3

health professional and the AI algorithm. From these 9 cases, two of them were 1 4

incorrectly classified by the algorithm due to an incorrect image acquisition with the 1 5

app. The remaining discrepant cases were further reviewed by a second professional, 1 6 and the AI-based system allowed to detect and modify the result with respect to the 1 7

initial healthworker interpretation in 4 cases by confirming the result predicted by the 1 8

algorithm.

The overall performance of the algorithm with respect to the ground truth is shown in 2 0 Table 1B . It should be noted that the performance of the system is high even when used 2 1

with a RDT different from those used for training the algorithm, suggesting its potential 2 2

use with any RDT on the market. The slight disparity in the performance of IgM band 2 3

identification in antibody RDTs between the validation set and this field study may be 2 4

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted February 19, 2022. explained by the presence of very faint signals which were almost invisible in the 1

Regarding the second field study for reading COVID-19 antigen RDTs, we found 3 that all tests used and digitized using the TiraSpot app (58 negative, 30 positive) were 4 correctly interpreted by the proposed system (Table 1C) , demonstrating that the system 5 can also be applied for reading two-band (control and test) as well as three-band tests 6 (IgG, IgM and control). We describe the usefulness of an app for reading and result interpretation of lateral-flow 1 0

RDTs for SARS-CoV-2 testing. Additionally, results are sent to a telemedicine platform 1 1

which allows for case identification and confirmation, quality control and real-time 1 2

Our AI algorithm demonstrates excellent performance, especially in prospective 1 4

validation in real-life scenarios and for both antibodies and antigen detection tests. The 1 5

algorithm performed as well in RDTs brands which were not used at all for training 1 6

purposes, making the solution suitable for other RDTs, including other diseases.

Compared with previous studies (6, 7), our system is able to identify individual bands of 1 8

the RDTs allowing complex results reading and sending them in real-time to a cloud 1 9

platform. A requirement and limitation of the proposed system is the correct acquisition 2 0 of the image (acquisition error in the field studies <0.8%).

In conclusion, the use of TiraSpot (Figure 1) is a useful tool for reporting, real-time 2 2

monitoring and quality control, as the results can be reviewed by specialists when 2 3

needed. This is especially important in contexts where massive testing is to be done and 2 4

the likelihood of subjectivity and errors in the interpretation of the result is higher. It is 2 5

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) also important in the validation of self-diagnostic tests performed by untrained users, as 1

it avoids the loss of information in case it is not notified by the user and provides an 2 efficient system to confirm and report data, which has been a key challenge during the 3

Omicron wave (4). 4

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) 

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

The copyright holder for this preprint this version posted February 19, 2022. ; Figure 1 . Tiraspot system is composed of (1) a mobile app for test digitization and result recording, (2) an AI model for RDT result interpretation and (3) a web platform where all collected data can be visualized and allows for result corrections in the cases in which a discrepancy exists between AI and user interpretation.

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

The copyright holder for this preprint this version posted February 19, 2022. ; https://doi.org/10.1101/2022.02.16.22271042 doi: medRxiv preprint

European Centre for Disease Prevention and Control. Options for the use of rapid antigen detection testsfor COVID-19 in the EU/EEA -first update

Use of CLIA-waived point-of-care tests for infectious diseases in community pharmacies in the United States

Online HIV Self-Testing (HIVST) Dissemination by an Australian Community Peer HIV Organisation: A Scalable Way to Increase Access to Testing, Particularly for Suboptimal Testers

Detection of a SARS-CoV-2 variant of concern in South Africa

Target Product Profile for a mobile app to read rapid diagnostic tests to strengthen infectious disease surveillance

Deep learning of HIV field-based rapid tests

CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity