key: cord-0797510-e65x5cb6
authors: Prokop, Mathias; van Everdingen, Wouter; van Rees Vellinga, Tjalco; Quarles van Ufford, Jet; Stöger, Lauran; Beenen, Ludo; Geurts, Bram; Gietema, Hester; Krdzalic, Jasenko; Schaefer-Prokop, Cornelia; van Ginneken, Bram; Brink, Monique
title: CO-RADS – A categorical CT assessment scheme for patients with suspected COVID-19: definition and evaluation
date: 2020-04-27
journal: Radiology
DOI: 10.1148/radiol.2020201473
sha: 8caf74ee62d941294f8cf3e18f9b4d6036f5792b
doc_id: 797510
cord_uid: e65x5cb6

PURPOSE: To introduce the COVID-19 Reporting and Data System (CO-RADS) for standardized assessment of pulmonary involvement of COVID-19 on non-enhanced chest CT and report its initial interobserver agreement and performance. METHODS: The Dutch Radiological Society (NVvR) developed CO-RADS based on other efforts for standardization, such as Lung-RADS or BI-RADS. CO-RADS assesses the suspicion for pulmonary involvement of COVID-19 on a scale from 1 (very low) to 5 (very high). The system is meant to be used in patients presenting with moderate to severe symptoms of COVID-19. The system was evaluated using 105 chest CTs of patients admitted to the hospital with clinical suspicion of COVID-19 in whom RT-PCR was performed (62 +/- 16 years, 61 men, 53 with positive RT-PCR). Eight observers assessed the scans using CO-RADS. Fleiss’ kappa was calculated, and scores of individual observers were compared to the median of the remaining seven observers. The resulting area under the receiver operating characteristics curve (AUC) was compared to results from RT-PCR and clinical diagnosis of COVID-19. RESULTS: There was absolute agreement among observers in 573 (68.2%) of 840 observations. Fleiss’ kappa was 0.47 (95% confidence interval (CI) 0.45-0.47), with the highest kappa for CO-RADS categories 1 (0.58, 95% CI 0.54-0.62) and 5 (0.68, 95% CI 0.65-0.72). The average AUC was 0.91 (95% CI 0.85-0.97) for predicting RT-PCR outcome and 0.95 (95% CI 0.91-0.99) for clinical diagnosis. The false negative rate for CO-RADS 1 was 9/161 (5.6%, 95% CI 1.0-10%), and the false positive rate for CO-RADS 5 was 1/286 (0.3%, 95% CI 0-1.0%). CONCLUSIONS: CO-RADS is a categorical assessment scheme for pulmonary involvement of COVID-19 on non-enhanced chest CT providing very good performance for predicting COVID-19 in patients with moderate to severe symptoms and has a substantial interobserver agreement, especially for categories 1 and 5.

technically insufficient examination (CO-RADS 0) and RT-PCR-proven SARS-CoV-2 infection at the time of examination (CO- RADS 6) .

It should be noted that CO-RADS is a CT-based system that assesses the suspicion of pulmonary involvement in COVID- 19 . The actual interpretation of whether a patient suffers from COVID-19 needs to include other data, such as laboratory tests, clinical findings, and type and duration of symptoms. At present, the reference standard for diagnosing COVID-19 remains a positive RT-PCR. In clinical practice, however, this may require repeated testing including deep bronchial and fecal samples and may be hampered by scarcity of tests in high-prevalence areas.

An overview of CO-RADS is given in Table 1 , and a pictorial overview is presented in Supplement 1.

CO-RADS 0 is chosen if none of the five categories can be assigned because of scans that are incomplete or of insufficient quality, for example because of severe artifacts due to coughing or breathing.

based on either a normal CT or CT findings of unequivocal non-infectious etiology. This was modelled on Lung-RADS, where cases with no nodules or with nodules with definitely benign features are reported together. As opposed to BI-RADS, where category 1 refers to normal only, we consider this approach more suitable for potential COVID-19 patients; concomitant findings are frequent in the lung and there is considerable interobserver variability regarding which findings are considered normal or not. Using our definition, mild or severe emphysema, perifissural nodules, lung tumors, or fibrosis are classified as CO-RADS 1. The category is identical to the "negative for pneumonia" category of the RSNA consensus statement (13) .

CO-RADS 2 implies a low level of suspicion for pulmonary involvement by COVID-19 based on CT findings in the lungs that are typical of infectious etiology that are considered not compatible with COVID-19. Examples are bronchitis, infectious bronchiolitis, bronchopneumonia, lobar pneumonia, and pulmonary abscess. Features include tree-in-bud sign, a centrilobular nodular pattern, lobar or segmental consolidation, and lung cavitation.

These features are similar to the ones in the "atypical appearance" category of the RSNA consensus statement (13) . Cases with smooth interlobular septal thickening with pleural effusion, which is also part of this RSNA category, are assigned CO-RADS 1 if considered typical for interstitial pulmonary edema, or CO-RADS 3 if ground-glass opacities that may mimic pulmonary involvement by COVID-19 are also present. This choice was made because CO-RADS describes the pulmonary and not the cardiac involvement of COVID-19.

CO-RADS 3 implies equivocal findings for pulmonary involvement of COVID-19 based on CT features that can also be found in other viral pneumonias or non-infectious etiologies.

Findings include perihilar ground-glass, homogenous extensive ground glass with or without sparing of some secondary pulmonary lobules, or ground glass together with smooth interlobular septal thickening with or without pleural effusion in absence of other typical CT findings. CO-RADS 3 also includes small ground glass opacities that are not centrilobular (otherwise CO-RADS 2) or not located close to the visceral pleura (otherwise CO-RADS 4). In addition, it contains patterns of consolidation compatible with organizing pneumonia without other typical findings of COVID-19. This category partially overlaps with the "indeterminate appearance" category of the RSNA consensus statement but includes those cases with lower likelihood for COVID-19 (13) .

CO-RADS 4 implies a high level of suspicion for pulmonary involvement by COVID-19 based on CT findings that are typical for COVID-19 but showing some overlap with other (viral) pneumonias. Findings are similar to CO-RADS 5 but are not located in contact with the visceral pleura or are located strictly unilaterally, are in a predominant peribronchovascular distribution, or are superimposed on severe diffuse pre-existing pulmonary abnormalities.

CO-RADS 4 comprises the features of the "indeterminate appearance" category of the RSNA consensus statement that are associated with a higher likelihood of COVID-19 (13).

CO-RADS 5 implies a very high level of suspicion for pulmonary involvement by COVID-19 based on typical CT findings (Table 2) . Mandatory features are ground-glass opacities, with or without consolidations, in lung regions close to visceral pleural surfaces, including the fissures, and a multifocal bilateral distribution. Other classifications only describe a peripheral location, but we found that the vicinity to the minor or major fissure is also typical. Subpleural sparing is allowed to be present. We found that the previously described lower lobe predominance is frequently not present in otherwise typical RT-PCR-positive cases and therefore lower lobe predominance was excluded as a required feature.

CO-RADS 5 requires the presence of at least one confirmatory pattern, which aligns with the temporal evolution of the disease (15) . The pattern that has been described early in the course of COVID-19 is dominated by multiple ground-glass areas, which often show (half) rounded and unsharp demarcation, but can be accompanied by sharply delineated ground-glass areas that outline the shape of multiple adjacent secondary pulmonary lobules. The crazy paving pattern, which has been described to appear later in the course of the disease, shows visible intralobular lines. As the disease progresses, increasing consolidations occur within the ground-glass areas. Finally, opacities occur that resemble organizing pneumonia, such as reverse halo signs or ground glass with extensive subpleural consolidations and air bronchograms. Subpleural curvilinear bands or bands of ground glass with or without consolidation in a tethered, arching pattern with small connections to the pleura are also considered typical. Thickened vessels within lung abnormalities are typical and frequently found in all other confirmatory patterns. CO-RADS 5 is largely identical to the "typical appearance" of the RSNA consensus statement (13).

CO-RADS 6, similar to BI-RADS 6, was introduced to indicate proven COVID-19 as signified by a positive RT-PCR test for virus-specific nucleic acid.

An observer study was performed on a set of 105 randomly selected chest CT scans obtained in a group of consecutive patients presenting to the emergency ward between March 14th 2020 and March 25th 2020 with suspected SARS-CoV-2 infection, in whom RT-PCR was performed. Patient inclusion, CT protocol, and radiation parameters are described in 

CT images were extracted from the picture archive and communication system, anonymized, and imported in a browser-based dedicated viewing system for CT scans (CIRRUS Core available at https://grand-challenge.org/reader-studies/). The software facilitated reading and scoring of anonymized CT images in the three orthogonal views, providing reading tools such as average or maximum intensity projections, window width-window level adaptation, panning and zooming.

Eight observers from seven hospitals in the Netherlands participated in the study (B.G., J.K., L.F.B., M.P., H.A.G., J.L.S., C.M.S-P., T.R.V.). Four observers had <5 years of experience in reading chest CTs, while the others had 5 to 27 years. All observers were familiar with the CO-RADS score from clinical experience interpreting at least 30 CT scans. Observers scored CO-RADS using a scoring software with drop-down lists, blinded for RT-PCR results and patient information except for age and gender. In addition, they were blinded for the prevalence of COVID-19 in the selected cohort, medical history, and clinical follow-up.

Statistical analysis was performed using SPSS statistics version 25 (IBM, Armonk, New York, USA). Data is presented as mean ± standard deviation or median and interquartile range (IQR), based on normality of data. A 5x5 confusion matrix was made separately per observer, in which the CO-RADS score of the observer was compared to the median CO-RADS score of the remaining seven observers. A similar matrix was calculated using the sum of all individual 5x5 tables.

For each observer, a receiver operating characteristics (ROC) curve was calculated, and the area under the ROC curve (AUC) was used to assess the performance of CO-RADS relative to two reference standards for the diagnosis of COVID-19: a positive RT-PCR test (PCR+) and a reference that combined the results of the RT-PCR test with a clinical COVID-19 diagnosis (PCR+ and PCR-/clinical+). Mean AUC across observers and 95% confidence intervals (95% CI) were calculated. In addition, the average percentage of cases assigned to each CO-RADS category, including 95% CI, were determined for PCR+, PCR-/clinical+, and PCR-/clinicalcases.

To quantify interobserver agreement, the Fleiss' kappa was determined across observers.

Kappa values were obtained by comparing the CO-RADS scores of each observer to the median of the remaining seven observers. Interobserver agreement was considered slight for a kappa value of 0.01-0.20, fair for 0.21-0.40, moderate for 0.41-0.60, substantial for 0.61-0.80, and almost perfect for 0.81-1.00 (16) . Table 3 depicts baseline characteristics of the 105 included patients (62 +/-16 years, 61 men, 53 with positive RT-PCR). In 21 patients at least one repeated RT-PCR was performed because of high clinical suspicion for COVID-19 but with a negative initial RT-PCR. An additional five patients had a clinical diagnosis of COVID-19 despite between one (n=2) and five RT-PCR negative tests (PCR-/Clinical+).

There was absolute agreement in assigned CO-RADS category in 573 of all 840 (68.2%) observations. A discrepancy by a single CO-RADS category was seen in 235/840 (28.0%) observations, of which pairs of CO-RADS 4 and 5 and CO-RADS 1 and 2 occurred in 128/840 (15.2%) observations. A difference of two CO-RADS categories was found in 31 (3.7%) and of three categories in 1 (0.1%) of the 840 observations. The resulting 5x5 table is given in Table 4 . Agreements of individual observers with the median of the remaining observers were either substantial (n=4) or moderate (n=4). In our setting with a high pre-test probability of disease in the acute phase of the pandemic, the performance of CO-RADS was very good, with an average AUC of 0.91, when compared to RT-PCR, and an AUC of 0.95, when compared to a combined RT-PCR and clinical reference standard. However, our results also indicate that the diagnosis of COVID-19 on CT remains difficult in a subset of patients, which underlines the importance of a reporting tool that includes diagnostic confidence. Bernheim et al. described that CT can be negative at the early stages of COVID-19 (19) , which might be the case for some of the 13/58 COVID-19 patients whose CTs were rated CO-RADS 1 or 2 by at least one observer. Therefore, a CO-RADS score of 1 and 2 should be interpreted with caution within the first days of disease presence. CO-RADS 3 encompasses a category in which CT alone offers little for the diagnosis of COVID-19. Presumably, knowledge of the prevalence of disease within the patient population, prior imaging studies, or a higher level of experience may decrease the number of equivocal calls. While CT findings are not specific for COVID-19 (12), they appear highly suggestive, which is underlined by only 1 false positive rating out of 286 CO-RADS 5 ratings.

Several caveats concern the performance of CO-RADS, mainly because the system was developed in the acute stage of the COVID-19 pandemic with rapidly rising case numbers and a parallel restriction in resources. Whether its accuracy remains high in other settings may depend on the prevalence of the disease, the duration of the pandemic, and the prevalence of other diseases with overlapping CT morphology. CO-RADS was developed in a high-prevalence setting, which implies that the positive predictive value is much higher than in a low prevalence situation. Also, no patients with residual abnormalities, such as subpleural banding after previous COVID infection, existed yet at the beginning of the pandemic. At the time that SARS-CoV-2 spread throughout Europe, the "influenza season" was coming to an end, reducing the number of overlapping patterns due to other viruses.

Finally, whether this system suffices for patients with mild or no symptoms, has not been validated.

Our observer study has limitations. First, the study group is relatively small. Second, it is representative of a population presenting to the emergency ward in the acute phase of the SARS-CoV-2 outbreak and requiring hospital admission for clinical reasons. This increases the disease prevalence substantially over a population with fewer symptoms. Third, observers had limited experience compared to areas with a larger outbreak of SARS-CoV-2. Finally, the diagnosis of COVID-19 was based on clinical decision despite negative RT-PCR results, but this occurred in a small subset of patients (n=5). In this multidisciplinary decision, the report of the CT scan was known, introducing an affirmation bias. Nevertheless, we included those patients in the study because it reflects current clinical practice.

The COVID-19 Reporting and Data System CO-RADS, developed by the Dutch Radiological Society, provides a framework that builds on other reporting schemes for COVID-19 but expands the concept in a way similar to systems like Lung-RADS. Its categories 1 to 5 provide increasing suspicion for pulmonary involvement of COVID-19 on non-contrast chest CT, thus allowing for task-specific cutoff points for clinical decision making. It provides very good performance for predicting COVID-19 in patients with moderate to severe symptoms and has substantial interobserver agreement, especially for categories 1 and 5. Therefore, the system fulfills the need for a structured and fast reporting system that decreases ambiguity in the communication with referring physicians and facilitates collection of CT performance data for further research of this worldwide healthcare problem. CO-RADS 6 proven RT-PCR positive for SARS-CoV-2 Table 2 . Features -typical for COVID-19

Obligatory features:

• ground-glass opacities, with or without consolidations, in lung regions close to visceral pleural surfaces, including the fissures (subpleural sparing is allowed) AND

• multifocal bilateral distribution Confirmatory patterns:

• ground-glass regions -unsharp demarcation, (half) rounded shape -sharp demarcation, outlining the shape of multiple adjacent secondary pulmonary lobules

• crazy paving

• patterns compatible with organizing pneumonia

• thickened vessels within parenchymal abnormalities found in all confirmatory patterns For each observer the CO-RADS score was compared to the median of the CO-RADS scores of the other seven observers. The sum of all CO-RADS scores of single observers compared to the other observers is shown.

Stability issues of RT-PCR testing of SARS-CoV-2 for hospitalized patients clinically diagnosed with COVID-19

Correlation of Chest CT and RT-PCR Testing in Coronavirus Disease 2019 (COVID-19) in China: A Report of 1014 Cases

Detection of SARS-CoV-2 in Different Types of Clinical Specimens

SARS-CoV-2 Viral Load in Upper Respiratory Specimens of Infected Patients

Detection of 2019 novel coronavirus (2019-nCoV) by real-time RT-PCR

Sensitivity of Chest CT for COVID-19: Comparison to RT-PCR

Diagnosis of the Coronavirus disease (COVID-19): rRT-PCR or CT?

Novel Coronavirus Pneumonia Diagnosis and Treatment Plan

A role for CT in COVID-19? What data really tell us so far

ACR recommendations for the use of chest radiography and computed tomography (CT) for suspected COVID-19 infection

Statements/Recommendations-for-Chest-Radiography-and-CT-for-Suspected-COVID19-Infection

The Role of Chest Imaging in Patient Management during the COVID-19 Pandemic: A Multinational Consensus Statement from the Fleischner Society

Performance of radiologists in differentiating COVID-19 from viral pneumonia on chest CT

Radiological Society of North America Expert Consensus Statement on Reporting Chest CT Findings Related to COVID-19. Endorsed by the Society of Thoracic Radiology, the American College of Radiology, and RSNA

Temporal Changes of CT Findings in 90 Patients with COVID-19 Pneumonia: A Longitudinal Study

The Equivalence of Weighted Kappa and the Intraclass Correlation Coefficient as Measures of Reliability

Intra-and interreader reproducibility of PI-RADSv2: A multireader study

Observer variability for Lung-RADS categorisation of lung cancer screening CTs: impact on patient management

Chest CT Findings in Coronavirus Disease-19 (COVID-19): Relationship to Duration of Infection

The authors would like to acknowledge Pieternel van der Tol and Willem Jan van der Woude for their help with the chest CT protocol and collecting radiation dose parameters. Ioannis Sechopoulos is acknowledged for proofreading the manuscript. Barbara Janssen and KarlijnGroenen are acknowledged for their work to obtain ethics board approval.