key: cord-0024154-w8suqme5 authors: Kim, Sungchul; Cho, Sungman; Cho, Kyungjin; Seo, Jiyeon; Nam, Yujin; Park, Jooyoung; Kim, Kyuri; Kim, Daeun; Hwang, Jeongeun; Yun, Jihye; Jang, Miso; Lee, Hyunna; Kim, Namkug title: An Open Medical Platform to Share Source Code and Various Pre-Trained Weights for Models to Use in Deep Learning Research date: 2021-10-26 journal: Korean J Radiol DOI: 10.3348/kjr.2021.0170 sha: a36aab8170cad914c9495033a5ae958d4deab6f6 doc_id: 24154 cord_uid: w8suqme5 Deep learning-based applications have great potential to enhance the quality of medical services. The power of deep learning depends on open databases and innovation. Radiologists can act as important mediators between deep learning and medicine by simultaneously playing pioneering and gatekeeping roles. The application of deep learning technology in medicine is sometimes restricted by ethical or legal issues, including patient privacy and confidentiality, data ownership, and limitations in patient agreement. In this paper, we present an open platform, MI2RLNet, for sharing source code and various pre-trained weights for models to use in downstream tasks, including education, application, and transfer learning, to encourage deep learning research in radiology. In addition, we describe how to use this open platform in the GitHub environment. Our source code and models may contribute to further deep learning research in radiology, which may facilitate applications in medicine and healthcare, especially in medical imaging, in the near future. All code is available at https://github.com/mi2rl/MI2RLNet. are trained and another dataset with which they are tested or validated are required. However, thus far, they have not been easily accessible to AI developers because they often contain sensitive patient information that would compromise patient confidentiality and privacy [7] . Prior to its expansion into the medical field, deep learning has been revolutionized through the use of open databases and innovation. Image databases, such as MNIST [8] , ImageNet [9] , COCO [10] , and CelebFaces [11] , have been used not only as datasets for the development of deep learning models but also as sources of open code, networks, and pre-trained weights. This open innovation approach made it easy for research groups around the world to share their experiences, leading to the increased popularity of deep learning approaches. Although it is well known that transfer learning from a pre-trained model that was trained using ImageNet could be helpful when researchers use a small dataset in radiology [12] kjronline.org medical big data is difficult for local hospitals. To tackle this problem, in this study, we released various types of pre-trained weights for various models through an open platform, MI2RLNet, for downstream tasks. We hope that our released model will have a positive effect on individual transfer learning. Figure 1 shows the overall architecture of MI2RLNet. We provide detailed examples for classification, detection, and segmentation tasks, and summarize how to use the open platform in the GitHub environment. In this paper, we introduce MI2RLNet, which consists of nine models that derive results for a specific purpose using medical images of various body parts obtained using different modalities. To train the MI2RLNet models, most data were obtained from open datasets, and some anonymized data were obtained from the Asan Medical Center (AMC). All AMC datasets were approved by the Institutional Review Board of AMC. The imaging data were anonymized in accordance with the Health Insurance Portability and Accountability Act Privacy Rule. Detailed information regarding our data is presented in Tables 1 and 2. In this paper, we briefly introduce our representative five models that may be widely used; the other four models not specifically addressed in this article include view classification in chest X-ray, liver segmentation in abdominal CT, polyp detection in endoscopy, and blackblood segmentation on MRI. To confirm the rationality of our implementation, we used network architectures that are commonly used in the computer vision field, such as U-Net [13] , three-dimensional (3D) U-Net [14] , ResNet [15] , EfficientDet [16] , and EfficientNet [17] . The performance is listed in Table 1 , and the details of each task, such as hyperparameters and training environments, are summarized in the Supplement. In actual clinical settings, there can be many errors in the digital imaging and communications in medicine (DICOM) header, especially when the images are collected from multiple centers, including incorrect recording of enhanced and non-enhanced CT imaging acquisition, which could strongly affect the accuracy of deep learning models. Therefore, we developed a classifier that consists of ResNet-50 [15] as a backbone network to classify enhanced and non-enhanced chest CT images. As a result, the model showed a 0.99 accuracy with the test set. An left/right (L/R) mark, typically seen as a strong white letter object, is located on every X-ray image, and it is not related to the status of the patient. However, the training of neural networks is usually affected substantially by pixels that have a strong intensity on images. If the L/ R mark is erased, the network will be able to learn better representative features. Therefore, we developed an L/R mark detection model, which consists of EfficientDet [16] with EfficientNet-B0 [17] as a backbone network, to detect and erase the mark. As a result, the model showed a 0.99 mean average precision with the test set. Figure 2 shows the results of the L/R mark-detection model. The aim of chest radiography is to identify abnormalities in the lung region; therefore, obtaining a lung mask for chest X-ray images is very important. In addition, this procedure can be used as a pre-processing step for the registration, focusing, or reduction of false positives in irrelevant regions. In particular, registration of the field of views of given images based on the lung region on chest X-ray images can be used to overcome differences in the pose and breath-hold level of each lung. Therefore, we developed a lung segmentation model, which consists of U-Net [13] , to obtain lung masks. As a result, the model Figure 3 shows the results of the lung segmentation model. Obtaining kidney and tumor masks is very important because of the wide variety in their morphology and how their morphology relates to surgical planning and outcomes. Therefore, we developed a kidney and tumor segmentation model, which consists of a cascaded 3D U-Net [14] , to obtain kidney and tumor masks. Figure 4 shows the cascaded 3D U-Net architecture. As a result, the model recorded the 33rd place in KiTS19 [13] challenge with a total DSC of 0.83 (0.96 for the kidney and 0.70 for the tumor) with the test set. Figure 5 shows the results of the kidney and tumor segmentation models. Brain extraction is a fundamental step in brain analyses. To perform an effective analysis, brain skull stripping is usually performed, but manual skull stripping is timeconsuming. Therefore, we developed a fully automated segmentation model, which consists of U-Net [13] , to automatically extract the brain area on T1-weighted MRI. As a result, the model showed a DSC of 0.95 with the test set. Figure 6 shows the results of the brain extraction model. In this section, we briefly describe how to use MI2RLNet. One module, liver segmentation, is presented with pseudocode as an example; other specific instructions can be found in the README in the GitHub repository. We also describe how MI2RLNet can be fine-tuned to fit custom datasets. The performance of the modules of MI2RLNet is excellent; however, for new data, the performance may be lower than expected. In this case, MI2RLNet can be fine-tuned for the new dataset to achieve robust performance for extended tasks. For more detailed information on each model, please refer to the Supplement, which includes typical models for classification, detection, and segmentation in chest X-ray, CT, and MRI, respectively. This paper can be cited as a reference when using models in future research. MI2RLNet is implemented so that users can use it easily. Furthermore, each module is well-modularized; thus, the desired results can be obtained with only a few lines of code. To use this concise library, only three steps need to be taken: initialize the module, set the model with pre- kjronline.org trained weights, and run. The "Checker" object is also provided, which can check the data type for each module and set up resources, such as Graphical Processing Units. Although the modules in MI2RLNet are implemented in different deep learning frameworks, such as TensorFlow [21] and PyTorch [22] , we provide an integrated environment independent of these frameworks as required. The guideline for this simple inference is introduced in Algorithm 1, with liver segmentation as an example. Detailed instructions on how to use other modules can be found in the README of each module in the GitHub repository. # 1. Initialize # Import the module you want to use from medimodule.Liver import LiverSegmentation # 2. Set the model with weight model = LiverSegmentation("path/of/weight") # 3. Run # Get a result of the image # 'save_path' must be set to save the result in the module image, mask = model.predict("path/of/image", save_ path="path/for/save") Although we trained our model using big datasets for robustness, the accuracy may be affected by the CT, X-ray, and MR devices used. Different standards between hospitals can also affect the model accuracy. Fine-tuning the released model weights can stabilize the training of the model from the beginning, and the accuracy can be increased for existing models. Detailed instructions on how to transfer this model into a more fine-tuned model can be found in the README of each module in the GitHub repository. Algorithm 2 shows an example of the fine-tuning of the liver segmentation model. Deep learning is becoming increasingly important in the medical field and may enable accurate predictions from the analysis of complex medical data. Deep learning requires sufficient training datasets and algorithms to optimize the performance on the training dataset before testing. Large datasets are required to achieve sufficient accuracy, and the data must be appropriately labeled. The difficulties in collecting data from medical records are as follows: access to relevant medical records may be difficult, particularly if treatment is spread over multiple different providers; timeconsuming administrative process to acquire access to medical records; heavy burden of filtering the data to meet research goals; and the accuracy of some data collection systems may be questionable. Owing to these difficulties, deep learning-based studies have mainly been conducted in large hospitals. Deep learning technologies related to medical data organization should be widely shared for the scalability of deep learning-based research in the medical field. We released pre-trained deep learning models that can be applied in various medical deep learning studies. This could help researchers to start their own deep learning research with fewer obstacles. For example, chest radiographs may have multiple views, such as posterior-anterior and lateral views, consequently increasing the difficulty of training the deep learning model. One of the models of MI2RLNet that we shared here could potentially be useful after fine-tuning in other hospitals. To develop a deep learning solution that guarantees generalizability, medical images should be processed considering the characteristics of the modality. We trained models for various tasks with various modalities through the classification, detection, and segmentation of X-ray, CT, and MR images. The pre-processing code we introduced could help colleagues start new tasks, and if they perform transfer learning using the trained models, they could start with smaller datasets or train their model faster. As we trained the models using public datasets and datasets from AMC (non-public datasets), our trained models may not be able to cover all possible sources of variability. In the future, we will keep our open platform sustainable by expanding its application to new tasks, covering more datasets, and considering the feedback of colleagues worldwide. The Supplement is available with this article at https://doi.org/10.3348/kjr.2021.0170. Data sharing does not apply to this article as no datasets were generated or analyzed during the current study. Resource use data by patient report or hospital records: do they agree? BMC Artificial intelligence in healthcare: past, present and future Deep learning in medical image analysis Deep learning techniques for medical image segmentation: achievements and challenges Deep learning for chest radiograph diagnosis in the emergency department Prediction of patient management in COVID-19 using deep learning-based fully automated extraction of cardiothoracic CT metrics and laboratory findings Big healthcare data: preserving security and privacy Gradient-based learning applied to document recognition ImageNet: a large-scale hierarchical image database Microsoft coco: common objects in context Deep learning face attributes in the wild CheXtransfer: performance and parameter efficiency of ImageNet models for chest X-Ray interpretation U-Net: convolutional networks for biomedical image segmentation 3D U-Net: learning dense volumetric segmentation from sparse annotation Deep residual learning for image recognition Efficientdet: scalable and efficient MI2RLNet for Deep Learning Research Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Efficientnet: rethinking model scaling for convolutional neural networks Development of a digital image database for chest radiographs with and without a lung nodule: receiver operating characteristic analysis of radiologists' detection of pulmonary nodules The KiTS19 challenge data: 300 kidney tumor cases with clinical context, CT semantic segmentations, and surgical outcomes Open access series of imaging studies: longitudinal MRI data in nondemented and demented older adults Tensorflow: large-scale machine learning on heterogeneous distributed systems Pytorch: an imperative style, high-performance deep learning library Namkug Kim who is on the editorial board of the Korean Journal of Radiology was not involved in the editorial evaluation or decision to publish this article. All remaining authors have declared no conflicts of interest.