Evaluating the Clinical Utility of a New Risk Prediction Model in Cystic Fibrosis

Abstract

Risk prediction models (RPMs), which estimate the probability of some future event, can inform clinical decisions about appropriate testing and treatment. Novel, machine learning (ML)-based RPMs have demonstrated superior performance for predicting events in numerous clinical applications, but the utility of such models for decision-making in real-world clinical practice remains unclear and uptake has been limited. We explored the development and evaluation of a novel RPM in Cystic Fibrosis (CF), where predictions of short-term mortality can inform decisions about when to refer patients for lung transplantation (LTx). In the first aim, we used real-world data (RWD) and ML approaches to develop a novel RPM for predicting 2-year mortality among adults with CF. We compared the discrimination accuracy and calibration of 8 potential ML models to the biomarker forced expiratory volume in 1 second (FEV1) alone. Super learner, an ensemble approach, had the highest discrimination accuracy at baseline, with an area under the receiver operating curve (AUC) at baseline of 0.914 (95% CI: 0.898, 0.929), compared to 0.876 (0.858, 0.895) for FEV1. In the second aim, we considered the potential impact of using the novel ML model for clinical decision-making. Using health outcomes modelling with RWD, we predicted the clinical decisions and downstream health outcomes of three alternative policies for LTx referral: (1) ML-based decisions, (2) FEV1-based decisions, and (3) usual care (UC) decisions identified in RWD. ML-based referral resulted in more patients referred for LTx (20.4% of patients (19.1%, 21.6)), compared to FEV1 (19.2% (18.0%, 20.4%)) and UC (12.4% (11.4%, 13.4%)). Of patients who died without referral under usual care, 40% would have been referred under ML and 31% would have been referred under FEV1. However, given a fixed supply of organs available for transplantation, higher referral rates did not lead to differences in the number of transplants or pre-transplant deaths. We found no significant difference in 5-year post-transplant or overall 5-year survival among policies. Our work demonstrates the value of using health outcomes modelling with RWD to evaluate the potential real-world clinical utility of novel RPMs.