Mortality Prediction Models Perform Poorly in Pneumonic COPD Exacerbations

Common models used to predict mortality among hospitalized patients with pneumonic chronic obstructive pulmonary disease (COPD) exacerbations are largely ineffective and better models are warranted, according to a new report.

The study, published in ERJ Open Research, found the CURB-65 and BAP-65 models had low predictive value, but so did a new model developed by investigators that used the extreme gradient boosting algorithm (XGBoost).

Corresponding author Akihiro Shiroshita, MD, MPH, of Ichinomiya Nishi Hospital in Japan, and colleagues explained that some research has suggested pneumonic and non-pneumonic COPD exacerbation have different inflammation profiles. However, existing studies that have looked at predictive modeling do not make it possible to thoroughly examine the performance of models in the specific subset of patients with pneumonic COPD exacerbations.

The investigators wanted to know how the models performed in this subgroup, so they tracked 1190 patients who were hospitalized with pneumonic COPD exacerbations at multiple health care centers.

The models used in the study were CURB-65, which bases predictions on confusion, blood urea nitrogen, respiratory rate, blood pressure, and age; and BAP-65, which uses blood urea nitrogen, altered mental status, heart rate, and age. Shirashita and colleagues then used machine learning and the XGBoost algorithm to develop their own prediction model.

Overall, 88 patients (7%) in the study died in hospital. The authors then attempted to validate the predictive models using the metric area under the receiver operating characteristic curve (AUROC).

All 3 had low predictive value, the authors reported. The BAP-65 model had an AUROC of 0.69 (95% CI, 0.66-0.72); the CURB-65 model had the same AUROC. Even the model based on machine learning, XGBoost, had an AUROC of just 0.71 (95%CI, 0.62-0.81). The differences in performance among the 3 were not statistically significant, the authors said.

Shiroshita and colleagues noted that previous research similarly showed that CURB-65 had low predictive value in pneumonic COPD exacerbations, although it performed better in non-pneumonic exacerbations.

“The disease spectrum of pneumonic COPD exacerbation, which fulfils the diagnostic criteria for both pneumonia and COPD exacerbation, may differ from that of COPD exacerbation and pneumonia,” the authors wrote. “A specific clinical prediction model for pneumonic COPD exacerbation is warranted.”

Shiroshita and colleagues wrote that in theory, the XGBoost model should have been able to overcome the limitations of the 2 more simplistic models. They said the new model’s inability to outperform the simple models was likely related to the size of the data set used to validate the model.

“The number of events required for model development is at least 10 events per variable,” the investigators noted. “Our input data included 10 variables, and at least 100 events were required; however, there were only about 60 events in our training dataset.”

The authors also noted that their study population was entirely Japanese, so the results may not be generalizable to other populations. Furthermore, since the scope of the study was limited to in-hospital mortality, the results do not capture longer-term outcomes.

The authors concluded by saying that a better predictive model will need to be developed using a larger data set and incorporating more variables into the analysis.

Reference:

Shiroshita A, Kimura Y, Shiba H, et al. Predicting in-hospital death in pneumonic COPD exacerbation via BAP-65, CURB-65 and machine learning. ERJ Open Res. Published online January 24, 2022. doi:10.1183/23120541.00452-2021