key: cord-0812922-h71x297l authors: Suleiman, M.E.; Rickard, M.; Brennan, P.C. title: Perfecting detection through education date: 2020-07-19 journal: Radiography (Lond) DOI: 10.1016/j.radi.2020.06.006 sha: 70b0b6f3b02e794b55eb604b550cb24528c83711 doc_id: 812922 cord_uid: h71x297l INTRODUCTION: Radiologists’ image reading skills vary, such variations in image interpretations can influence the effectiveness of the early treatment of disease and may have important clinical and economic implications. In screening mammography, clinical audits are used to assess radiologists’ performance annually, however, the nature of these audits prevent robust data analysis due to the low prevalence of breast cancer and the long waiting periods for the audit results. Research-based evidence revealed a need for changes in the methods utilised to optimise the assessment of the efficacy of radiologists’ interpretations. METHODS: A cloud-based platform was developed to assess and enhance radiologists’ performance help reduce variability in medical image interpretations in a research environment; however, to address a number of limitations, the platform was commercialised to make it available worldwide. RESULTS: DetectED-X’s team have been able to make their cloud-based platform available worldwide, tailored to the needs of radiologists and accredited for continuing medical/professional education; thus, changing the continuous professional development practice globally. CONCLUSION: DetectED-X’s Rivelato, was developed to address a need for effective, available and affordable educational solutions for clinicians and health care workers wherever they are located. A true fusion of industry, academia, clinics and consumer to adapt to the growing needs of clinicians’ around the world, the latest being COVID-19 global pandemic. DetectED-X repurposed its platform to educate physicians around the world on the appearances of COVID-19 on Lung Computed Tomography scans, introducing CovED to clinicians worldwide free of charge as a multi-national consortium of collaboration to help fight COVID-19, showing how research-based evidence can create effective and scalable change globally. Medical imaging is a vast science, dominated by diagnostic radiology. With a billion radiological imaging exams performed annually, 1 radiology constitutes an essential component of the information a physician utilizes to make diagnostic and treatment decisions. However, as technology advances, physicians are reading medical images from an increasing variety of modalities, and in increasing numbers. Varying levels of skill and experience contribute to errors in image interpretation. 1 Such errors and variations in image interpretations can influence the effectiveness of the treatment of disease and may have important clinical and economic implications. In this article we discuss the use of research-based evidence to provide insights on radiologist performance and the implementation of change in methodological approaches to optimise the assessment of the efficacy of radiologists' interpretations. Mammography interpretation in breast screening services has an ultimate aim of reducing breast cancer mortality by detecting cancer at an early stage. Accurately identifying cancers is critically important, however, it is also important to minimize unnecessary follow-up diagnostic procedures, costs, and patient anxiety associated with false-positive decisions. The early treatment of breast cancer relies on the accurate interpretation of mammograms, and radiologists' variability in interpretation has the potential to compromise the effectiveness of breast screening outcomes. 2e6 Radiologists involved in breast cancer screening encounter relatively low numbers of cancer cases, 7, 8 and receive delayed identification of missed cancers. They therefore have limited opportunities for timely feedback on the efficacy of interpretations. The main way to assess radiologists' performance is clinical audits, however these are associated with slow feedback periods and low cancer numbers yielding less than robust data for analyses. The Australian Institute of Health and Welfare (AIHW) regularly report on breast cancer screening outcomes in Australia. In the period 2006e2008, the BreastScreen Australia services met most performance objectives, 9 the program sensitivity for the target population (50e69years) was shown to range from 89.4 to 91.8 for 0e12 months post first screening rounds and from 83.2 to 86.8 for 0e12 months post subsequent screening rounds. While the cancer detection rates showed steady improvement over the years since the beginning over the program, there was and still is no national breast screening training program for radiologists. To help address BreastScreen reader training and quality issues, a novel system, the BreastScreen Reader Assessment Strategy (BREAST), was created by Brennan et al. 10 utilising seeded test sets of carefully selected mammograms. BREAST was implemented in Australia in 2011 with clinical expertise led by Professor Warwick Lee (BreastScreen NSW State Radiologist at the time) and financial support from the Australian Department of Health and Ageing and the Royal Australian and New Zealand College of Radiologists (RANZCR). The strategy secured additional funding from the National Breast Cancer Foundation and the Cancer Institute NSW to further develop the infrastructure in Australia. Subsequently, BREAST has been implemented in most BreastScreen Australia programs and has held workshops in all RANZCR Annual Scientific Meetings and Breast Interest Group meetings since 2011. The BREAST research and development team is comprised of post-doctoral research fellows with specialties in medical imaging, physics, biomedical engineering, information technology, and epidemiology. This system has proven to be an effective supplement to clinical audits for assessing radiologists' performance and identifying potential areas for improvement. Findings from our previous work showed that reading performance with test sets correlated well with performance in clinical practice. 11, 12 BREAST has been available to Australian and New Zealand radiologists since 2011. Approximately 80% of mammography radiologists have used these test sets, and a 34% improvement in performance has been shown with repeated use. 13 Nonetheless, BREAST has some limitations: The original DICOM images have to be uploaded manually to a local PACS in order to be available for radiologists to diagnose, limiting access. Also, BREAST images are not available for worldwide use due to ethical considerations around image access, and furthermore the system's funding is reliant on grants as it is considered a research platform to understand the underlying causes of radiologists' performance variations. To address these limitations, in 2019, three medical imaging academics, Professor Patrick Brennan, Professor Mary Rickard and Dr Mo'ayyad Suleiman, created a University of Sydney spinoff company, DetectED-X, to achieve the ideal of cloud based medical imaging test sets using different modalities, and available to radiologists worldwide. The main goals of DetectED-X team were: Provide test sets via an online platform available 24/7, enabling radiologic self-assessment with immediate personalised feedback wherever a reader is located; Define standards for performance and support quality assurance-improvement programs; Give each radiologist confidential ranking on their performance compared with radiologists in their location or anywhere in the world. The team started to look into ways to gauge the interest of international radiologists, using international conferences such as the Radiological Society of North America (RSNA), The Royal Australian and New Zealand College of Radiologists (RANZCR), conferences in South East Asia, Europe and the Middle East. It was evident that international radiologists welcomed the idea of making the platform available worldwide and saw the benefits behind it. Trying to commercialise the platform, the founders quickly realised that although they were experts in academia, they lacked the skills in business to create a successful commercial entity. Therefore, they found support through Incubate, a University of Sydney start-up accelerator program, which over a four-month period assisted the team to navigate the difficulties of launching a start-up, provided key connections, and mentored the co-founders. Also, Jobs NSW saw the benefits that DetectED-X could provide to the Australian jobs market and granted them the Minimum Viable Product (MVP) grant, which helped build the new online platform. This culminated in DetectED-X winning the Australian Start-Up of the Year 2019 for Community and Social Good award for its breast and lung diagnosis tool e Rivelato. Rivelato is an educational, cloud-based platform focussing on breast and lung radiologic test sets and aiming to improve radiologic detection of cancer. The software and image test sets allow radiologists and medical trainee "readers" to diagnose sets of images wherever in the world they are located and provide the readers with immediate individual feedback. Rivelato's clinically relevant test cases allow readers to look at real de-identified images (mammography for breast cancer and pulmonary CT for lung cancer) in the same way as they would in the clinic. A reader is given a unique login to the Rivelato platform and independently judges each image in a test set, marking any site where he/she perceives a lesion, and giving the lesion a score: 2 ¼ benign finding, 3 ¼ indeterminate/equivocal finding, 4 ¼ suspicious finding of malignancy, and 5 ¼ malignant finding. If the reader decides that the case has no significant abnormality, he/ she clicks on the NEXT icon and the next case is displayed. Readers can go back to any image or case and correct a previous decision prior to submitting their answers (Fig. 1) . The test set cases are rigorously validated as being cancerous or cancer-free, so accurate feedback can be given to each reader on their performance and any diagnostic errors. Once the test set is completed, the system instantly and intelligently analyses the data and presents a range of performance metrics including sensitivity, specificity, true positive, true negative, false positive and false negative scores, and anonymised comparison charts that allow readers to compare their results against others who completed the same test set (Fig. 2) . In addition, reader-specific image files are generated so that correct and incorrect decisions can be examined (Fig. 3) . It is intended that education test sets can be built and delivered to individual radiologists based on performance errors. The result is a better way for radiologists to effectively improve their cancer detection skills. Finally, readers can download a certificate of completion, which is recognised as a continuing medical education (CME) activity by the Royal Australian and New Zealand College of Radiologists and by professional bodies in other countries. The success of DetectED-X's platform (Rivelato) opened the way to adapt this platform to other areas of disease diagnosis. In particular, the team was approached to develop a similar platform to help increase radiologist performance reading chest CT scans for the diagnosis of dust disease. Pneumoconiosis resulting from inhalation and retention of dust can result in debilitating respiratory conditions and death, and its early diagnosis facilitates the introduction of strategies to effectively manage disease progression as well as allowing the introduction of more effective treatments. 14 Its diagnosis relies on accurate detection and interpretation of changes on chest radiographs or CT lung scans. However, identifying suspicious lung features particular those representing early stage disease is considered a substantial challenge for clinicians resulting in important numbers of diagnostic errors DetectED-X is developing CHEST (Chest diagnosis: a High level Education STrategy) based on Rivelato, allowing radiologists to diagnose chest CT images with known truth and instantly receive feedback on performance levels and any errors made. Steps are being taken to finalise the CHEST platform and implement it within the Australian arm of the International Lung Screening Trial which will promote involvement of relevant clinicians across Australia in both urban and rural areas, and award CME points to those who engage. The World Health Organisation called for solidarity in the global response to the COVID-19 pandemic and recognised that the number of patients suffering from this life-threatening illness was fast outpacing the number of skilled staff required for appropriate care. Accurate detection and assessment are critically important, enabling prompt treatment, effective isolation and rapid implementation of public health strategies. It was evident that accurate clinical diagnosis was a limiting factor in the face of the looming pandemic, and that scalable and accurate imaging diagnosis would be vital to effective triage. DetectED-X team adapted the Rivelato and CHEST platforms to help educate clinicians to recognise the CT appearances of COVID-19. The CovED platform was launched free to clinicians worldwide on March 30th, 2020 (Fig. 4) . It was delivered through a consortium involving DetectED-X -a University of Sydney Start Up, the University of Sydney, clinical experts and the leading medical imaging companies e GE Healthcare and Volpara who ensured global adoption of CovED. Using CovED, individuals can assess their performance on clinically relevant images in their own reading environment, and receive immediate feedback using well-known performance scores. Image files personalised for each clinician are instantly provided showing responses and any errors, and providing an expert opinion. It is planned to augment the educational impact of CovED, by introducing the "AI Companion" which will provide automated segmentation and annotation with detailed descriptions of groundglass opacities, consolidations etc and their distribution -focal, multifocal, or diffuse. 15 This categorisation aims to improve standardised reporting and facilitate improvement of COVID-19 specific diagnosis. Our group has recognised the need for effective and affordable educational solutions for image interpretation, available to clinicians and health care workers wherever they are located. Its deliverables continue to adapt to clinical needs and rely on regular partnerships between industry, academia, clinicians and consumers. Without these partnerships and recognition that constant methodological change is required to reflect clinicians needs as well as shifting professional and regulatory requirements, the project would not work. With enterprises like this, the willingness to work hard, the need to adapt to rapidly changing clinical needs and the value of surrounding oneself by clever people with complementary skills should not be underestimated. Error and discrepancy in radiology: inevitable or avoidable? Reader variability in mammography and its implications for expected utility over the population of readers and cases Variability in the interpretation of screening mammograms by US radiologists: findings from a national sample Inter-observer and intra-observer variability of mammogram interpretation: a field study Screening mammograms by community radiologists: variability in false-positive rates Variability in radiologists' interpretations of mammograms A portrait of breast imaging specialists and of the interpretation of mammography in the United States Physician workload in mammography Australian Institute of Health and Welfare. BreastScreen Australia monitoring report BREAST: a novel method to improve the diagnostic efficacy of mammography. SPIE Screening mammography: test set data can reasonably describe actual clinical reporting Improvement of cancer detection on mammograms via BREAST test sets Impact of Breast Reader Assessment Strategy on mammographic radiologists' test reading performance Role of chest computed tomography in prevention of occupational respiratory disease: review of recent literature Correlation of chest CT and RT-PCR testing in coronavirus disease 2019 (COVID-19) in China: a report of 1014 cases The Authors declare that they are the co-founders and Directors of DetectED-X.