key: cord-0985492-06wonfza
authors: Yoshioka, Takashi; Funada, Satoshi; Luo, Yan
title: The Inherent Problems With the Generalizability of the CALL Score: Towards Reliable Clinical Prediction Models for Coronavirus Disease 2019 (COVID-19)
date: 2020-10-16
journal: Clin Infect Dis
DOI: 10.1093/cid/ciaa1564
sha: 6f89d9bdeb401008896677343995011bc31de4a5
doc_id: 985492
cord_uid: 06wonfza

nan

To the Editor-As the coronavirus disease 2019 (COVID-19) pandemic has evolved and caused a large number of infections and deaths, clinical prediction models that either aim to identify subjects at risk in the general population or to predict disease progression and mortality are becoming increasingly important. A recently published clinical prediction model, the CALL score, developed by Ji et al, showed good performance in predicting COVID-19 disease progression [1] . To develop a robust and user-friendly model, the authors performed internal validation, selected only 4 predictors, and developed a simple prediction model. As a result, the developed model was expected to be relatively robust and easy to use for clinicians. Although the model showed a high predictive ability, we have 2 concerns regarding its generalizability.

Our first concern is the variable selection. In that study, the authors defined 4 variables (CALL; comorbidity, age, lymphocyte, and lactate dehydrogenase [LDH]) as candidate predictors based on the P value-based criteria (ie, P < .05), in the multivariable Cox regression model. Although these predictors were clinically important in the study, it is unknown whether they would be equally important in other clinical settings. In other words, the CALL model may not have included other clinically important variables. For example, many studies suggested that obesity, defined as body mass index ≥30 kg/ m 2 , is an important risk factor for severe illness from COVID-19 [2] ; however, the CALL model did not contain obesity as a predictor. Lack of such an important predictor may lead to a lower predictive ability of the model after external validation. One simulation study suggested that 5 of the 6 variables selected via the P value-based criterion (P < .05) showed more than 50% biased estimation of β coefficients in the multivariable logistic regression model [3] . To make the prediction model more generalizable, variable selection based on both clinical reasoning and external literature should be considered [3, 4] .

Our second concern is regarding missing values. The authors did not show the number of study participants with missing data for the predictors. When missing data exists, the authors should consider the mechanism of missingness (missing completely at random, missing at random, or missing not at random), and they should deal with missing data using an appropriate method, such as multiple imputation [5] . If the authors used complete-case analysis despite the presence of missing data, they would have overestimated the results due to overfitting [6] . The discriminative ability of the CALL score is 0.91, which was too high to predict the outcome using only 4 variables; hence, overfitting due to missing data was suspected. To confirm transparency and generalizability, presenting and dealing with missing values are needed.

In summary, the authors may have shown an overestimated model performance due to inappropriate variable selection and missing data. Many COVID-19 prediction models are known to have a high risk of bias, and such unreliable models may result in more harm than good in clinical decision making [7] . To develop prediction models with high predictive abilities, using robust methodologies, such as the TRIPOD guidelines [5] , is essential.

Author contributions. All authors confirm they have contributed to the intellectual content of this paper and have met the following 4 requirements: (a) significant contributions to the conception and design, acquisition of data, or analysis and interpretation of data; (b) drafting or revising the article for intellectual content; (c) final approval of the published article; and (d) agreement to be accountable for all aspects of the article thus ensuring that questions related to the accuracy or integrity of any part of the article are appropriately investigated and resolved.

Potential conflicts of interest. 

Prediction for progression risk in patients with COVID-19 pneumonia: the CALL score

Effect of obesity and body mass index on coronavirus disease 2019 severity: A systematic review and meta-analysis

Poor performance of clinical prediction models: the harm of commonly applied methods

Variable selection strategies and its importance in clinical prediction modelling

Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement

A comparison of imputation techniques for handling missing predictor values in a risk model with a binary outcome

Prediction models for diagnosis and prognosis of Covid-19: systematic review and critical appraisal