key: cord-1008532-2082cgp5 authors: Kapoor, A.; Mahajan, G. title: COMPARISON OF ARTIFICIAL INTELLIGENCE ENABLED METHODS IN THE COMPUTED TOMOGRAPHIC ASSESSMENT OF COVID-19 DISEASE. date: 2020-09-03 journal: nan DOI: 10.1101/2020.09.02.20186650 sha: 64d9352ff0b93af4bba1cc38baeaee5198abd96f doc_id: 1008532 cord_uid: 2082cgp5 Objectives: Comparison of three different Artificial intelligence (AI) methods of assessment for patients undergoing Computed tomography (CT) for suspected Covid-19 disease. Parameters studied were probability of diagnosis, quantification of disease severity and the time to reach the diagnosis . Methods: 107 consecutive patients of suspected Covid-19 patients were evaluated using the three AI methods labeled as AI-I,II, III alongwith visual analysis labeled as VT for predicting probability of Covid-19, determining CT severity score (CTSS) and index (CTSI) , percentage opacification (PO) and high opacification (POHO). Sensitivity, specificity along with area under curves were estimated for each method and the CTSS and CTSI correlated using Friedman test. Results: Out of 107 patients 71 patients were Covid-19 positive and 20 negative by RT-PCR while 16 did not get RT-PCR done. AI-III method showed higher sensitivity and specificity of 93% and 88% respectively to predict probability of Covid 19. It had 2 false positive patients of interstitial lung disease. AI-II method had sensitivity and specificity of 66% and 83% respectively while visual (VT) analysis showed sensitivity and specificity of 59.7% and 62% respectively. Statistically significant differences were also seen in CTSI and PO estimation between AI-I and III methods (p<0.0001) with AI-III showing fastest time to calculate results. Conclusions: AI-III method gave better results to make an accurate and quick diagnosis of the Covid-19 with AUC of 0.85 to predict probability of Covid-19 alongwith quantification of Covid-19 lesions in the form of PO, POHO as compared to other AI methods and also by visual analysis. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted September 3, 2020. . https://doi.org/10.1101/2020.09.02.20186650 doi: medRxiv preprint • AI-III method had a higher sensitivity and specificity of 93% and 88% compared to other methods in predicting probability of Covid-19. • Significant inter method variations were seen in quantifying Covid-19 opacities as CTSS,CTSI, PO and POHO variables (p<0.0001). AI-III method showed no statistical difference with VT method for PO variable (p=0.24) and was the only method which depicted all the variables.. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted September 3, 2020. . https://doi.org/10.1101/2020.09.02.20186650 doi: medRxiv preprint Covid-19 is a highly infectious disease affecting more than 3 million people worldwide as on this date. Due to its high rate of infectiousness rapid tools are needed for early and accurate diagnosis and also for monitoring the progress of disease (1) . So far RT-PCR test using throat and nasal swabs form the backbone of diagnosis and is considered to be gold standard but is marred by reduced sensitivity. It has high false negative rates i.e.30-35%, is more time consuming and also has false positives (2) . The epidemic is thus driving researchers to think of more efficient ways to cope with the huge demand for diagnosis. Imaging has been used in China on a large scale to tackle the endemic and there have been numerous reports about the experiences using CT scans. Chinese national guidelines have recommended CT scan as a key method to diagnose Covid-19 (3) . Typical reported features on CT of Covid-19 include multifocal ground glass opacities and consolidations with peripheral and basal predilection (4, 5) .Based on these Radiological society of North America has proposed CORADS classification system on a scale of 1-6(6,7). However American college of Radiology in June 2020 issued guidelines mitigating its use for diagnosis of COVID-19 mainly due to fear of contaminating radiology facilities and also due to its lack of specificity (8) . So what should be the role of radiology in current pandemic is a question under debate (9, 10) . Many researchers have come up with Artificial intelligence(AI) based prototypes to automate the diagnosis of Covid-19 disease which will expedite early and accurate diagnosis. This should help physicians triage patients into Covid-19 designated units for treatment and also help contain further spread of disease(11). We evaluated the results of three such AI based methods using Computed tomography images of the chest done on patients suspected to be having Covid-19 disease with following objectives a) To estimate the ability of AI methods to diagnose Covid-19 disease. b) Compare the results of different AI methods and with visual analysis in determining the severity of disease. c) To determine the time taken for making diagnosis using AI. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted September 3, 2020. All the results were statistically analysed using Analyse-IT software (Leeds, UK) to determine the parametric of mean, standard deviation, correlation was done using Friedman test for pair groups between all four methods and Sensitivity, specificity along with Area under curves calculated. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted September 3, 2020. VT method showed a sensitivity and specificity of 59.7% and 62% to diagnose Covid-19 with false positive and false negative of 37% and 40% respectively with positive predictive value of 83% and negative predictive value of 37%( table2). The area under curve estimation showed for AI-II,III and VT methods as 0.68, 0.85and 0.62 respectively with differences between AI-III and AI-II and VT method being statistically significant( p-0.0001).(Figure4). AI,III methods could calculate CTSS and CTSI. AI-I method determined the CTSS in all patients with median CTSS of 12 (10-13 96.7%) and CTSI of 48% ( median 40-52% 96.7% CI). AI-III method and VT methods also determined the CTSS and CTSI with mean scores of 7 ( 6-9 96.7% CI), 35% ( 30-45 97.6% CI)for AI-III method and 6.50(6-9 96.7%CI) 32.5 (30-45 96.7%CI) respectively. Friedman test was done between three methods of analysis . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted September 3, 2020. . https://doi.org/10.1101/2020.09.02.20186650 doi: medRxiv preprint and showed statistically significant differences between AI-III method and AI-I and VT methodsp<0.0001(Table3), (Figures 5A-D) . . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted September 3, 2020. CT imaging has a important undeniable role in the evaluation of patients with lower respiratory tract disease including suspected COVID-19. Evaluation of a suspected Covid-19 patient can be done by chest CT to answer several questions in a clinical setting i.e, from diagnosis, to determining the disease severity, progression and treatment response. In a pandemic situation timely diagnosis by the use of imaging can be the essence of management (12) . Our study shows that use of AI with CT does improve the sensitivity and specificity of diagnosis compared with visual analysis method which was the primary objective of the study . It is critically important to have probability algorithm in the AI methods to be able to answer the question of whether Covid-19 or not . Our study shows that at this point of time not all AI methods have this ability which was absent with in AI-I method and variations in predicting the probability were also seen in the results between AI-II, III methods . AI-III showed the highest sensitivity and specificity of 93% and 88% respectively suggesting it is almost ready for clinical use with results being superior to visual analysis alone . The probable reason for this is the machine learning method used in this technique. The developers of this AI method used a supervised deep learning method based on 3D neural network in which a two channel 3D tensor was used where the first channel contained CT Hounsfield units masked by lung segmentation while the second channel had probability maps of opacity classifiers of metric data (13, 14) . Chiganti etal (13) showed improved performance with AUC of 0.90(0.85-0.94 95%CI with sensitivity and specificity of 86% and 81% respectively which was similar to the results seen in the present study. The earlier AI methods used metric based AI classifiers based on opacity metrics with generation of heat maps and regression algorithms with reduced sensitivity specificity of 74% as was with AI-II method in this study. Similar results have also been shown by Li etal (15) who showed reduced false positive and negatives in making diagnosis with the use of 3D machine learning methods. CTSS has recently been proposed as a reliable measure to demonstrate correlation of the severity of disease with clinical condition of patient (16) . It has been used to assess the progression of disease by many studies (17) . In the present study accurate CTSS scoring was possible with both AI-I and AI-III methods but only AI-III method shows no statistical difference with the VT method (p=0.24). AI-II method could not compute a severity score. In our study lung and lobar and opacity segmentations were also possible with the use of all AI . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted September 3, 2020. . https://doi.org/10.1101/2020.09.02.20186650 doi: medRxiv preprint based methods. However AI-III method used deep image to image network machine learning algorithms which were more robust and depicted both PO and POHO estimates which was not possible by other AI methods nor by visual method. These quantitative analysis of the severity of lung involvement help physicians to triage and monitor Covid -19 patients and are essential in all AI methods of analysis (18, 19) .Findings of the present study suggest that standardized objective measures of disease evaluation were missing in some of the methods evaluated in this study therefore improved training data sets and algorithms are required in AI-I,II methods to achieve accurate segmentation before all AI based methods can be applied in routine clinical practice (20). This would mean that some more time is required before AI comes into clinical use. The third objective of the study i.e determining time to diagnosis was also achieved in the study with AI-III method which had a mean processing time of 2 minutes which was quick enough to make the diagnosis of Covid-19. To deal with highly infectious disease like Covid-19 early diagnosis is the essence and onsite processing algorithms like used in AI-III should be preferred than cloud based evaluation techniques. To conclude addition of AI along with CT chest evaluation looks attractive and its use achieves the objectives set for evaluation in the present study especially with the use of AI-III method. It has the potential to improve the accuracy and time to make the diagnosis. The study also highlights limitations in various AI methods tested along with inter method variations of results like estimation of CTSI, percentage opacification and some more time may be required before it comes into clinical use. Out of the AI techniques compared AI-III method appears to be more advantageous and accurate compared with other AI methods including the visual method alone. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted September 3, 2020. with CTSS and CTSI of 9 and 45% with Covid probability 1.0. D) VT method correlates CTSS 10 and CTSI of 50%. 6. Table 1 : Showing patient demographics. 7. Table2: Showing qualitative parameters of diagnostic evaluation of AI-I,III and VT methods. 8. Table 3 : Friedman test of CTSI of AI-I,III and VT method. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted September 3, 2020. is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted September 3, 2020. . https://doi.org/10.1101/2020.09.02.20186650 doi: medRxiv preprint . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted September 3, 2020. . https://doi.org/10.1101/2020.09.02.20186650 doi: medRxiv preprint Sensitivity of chest CT for COVID-19: comparison to RT-PCR Essentials for radiologists on COVID-19: an update-radiology scientific expert panel Radiological diagnosis of new coronavirus infected pneumonitis: expert recommendation from the Chinese Society of Radiology Chin Sensitivity of chest CT for COVID-19: comparison to RT-PCR Chest CT findings in coronavirus disease-19 (COVID-19): relationship to duration of infection Radiological Society of North America Expert Consensus Statement on Reporting Chest CT Findings Related to COVID-19. Endorsed by the Society of Thoracic Radiology, the American College of Radiology, and RSNA.RadiolCardiothorac Imaging CO-RADS-A categorical CT assessment scheme for patients with suspected COVID-19: definition and evaluation ACR recommendations for the use of chest radiography and computed tomography(CT) for suspected COVID-19 infection Outbreak of novel coronavirus (COVID-19): What is the role of radiologists Performance of radiologists in differentiating COVID-19 from viral pneumonia on chest CT Artificial intelligence(AI) applications for COVID-19 pandemic. Diabetes and metabolic syndrome: clinical research and reviews Essentials for radiologists on COVID-19: an update-radiology scientific expert panel Quantification of tomographic patterns associated with COVID-19 from chest CT. arXivPrepr arXiv200401279 Modern hierarchical, agglomerative clustering algorithms. arXivPrepr arXiv11092378 Artificial intelligence distinguishes COVID-19 from community acquired pneumonia on chest CT Relation Between Chest CT Findings and Clinical Conditions of Coronavirus Disease (COVID-19) Pneumonia: A Multicenter Study Chest CT Severity Score: An Imaging Tool for Assessing Severe COVID-19 Relation Between Chest CT Findings and Clinical Conditions of Coronavirus Disease (COVID-19) Pneumonia: A Multicenter Study Chest CT Severity Score: An Imaging Tool for Assessing Severe COVID-19. RadiolCardiothorac Imaging International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity