An Evaluation of Adaptive Clinical Trial Designs with Pre-specified Rules for Modifying the Sampling Plan

Abstract

Adaptive clinical trial design has been proposed as a promising new approach that may improve the drug discovery process. A comprehensive evaluation of adaptation should balance potential flexibility and efficiency gains against interpretability, logistical, and ethical concerns. In this research, we develop and rigorously evaluate a class of adaptive designs with pre-specified rules for modifying the sampling plan. We demonstrate that optimal pre-specified adaptive designs provide only very small efficiency gains over group sequential designs with the same number of analyses. Our findings provide insight into what are good and bad choices of adaptive sampling plans and suggest that adaptive designs proposed in the literature often include inefficient sample size modification rules. We also evaluate the reliability and precision of different inferential procedures. We extend group sequential orderings of the outcome space based on the stage at stopping, likelihood ratio test statistic, and sample mean to the adaptive setting in order to compute point estimates, confidence intervals, and P-values. The likelihood ratio ordering is found to average shorter confidence intervals and produce higher probabilities of P-values below important thresholds than alternative approaches. The bias adjusted mean demonstrates the lowest mean squared error among candidate point estimates. A conditional error-based approach in the literature has the benefit of being the only method that accommodates unplanned adaptations. We compare the performance of this and other methods in settings where adaptations could realistically be pre-specified at the design stage in order to quantify the cost of failing to plan ahead. We find the cost to be meaningful for all designs and treatment effects considered, and to be substantial for designs frequently proposed in the literature. Finally, we demonstrate that the behavior of adaptive designs relative to group sequential designs may suffer when considering both statistical and clinical significance, as well as in settings where the treatment effect varies over time. We also address the merit of weighting trial participants differently, the added complexity of protocol development, and the possibility that certain adaptation rules may unblind trial investigators and compromise trial integrity.