Sequential safety monitoring using observational data: A comparison of methods appropriate for newly-licensed vaccines in children

Abstract

Sequential safety monitoring of newly-licensed vaccines is a national health priority and is routinely conducted using observational data. However, applying sequential methods in observational settings where the adverse events (AEs) being monitored are rare is relatively new and much is unknown about method performance. In this thesis we use simulations and an example application to compare four existing group sequential (GS) methods: two that use regression to control for confounding (GS Generalized Estimating Equations and GS Lan-DeMets-Regression) and two that use one-to-one matching to control for confounding (GS Likelihood Ratio Test and GS Lan-DeMets-Matching). We simulated five sites and varied the amount of confounding by site, sample sizes of the sites, and prevalence of the AE under surveillance. We also applied the methods to data from a recent Vaccine Safety Datalink study. In the simulations, the matched methods were less powerful, were slower to detect true safety signals, and experienced more implementation difficulties with rare AEs compared to the regression methods. Across different confounding by site scenarios, differences in power to detect a safety signal depend on how evenly the AEs were distributed across sites, the amount of statistical information at each site, and the direction of the relationships between site and exposure or between site and AEs. In the data application, both regression methods successfully detected a safety signal under a variety of testing frequencies, while neither matched method detected a safety signal for any of the testing frequency options we used. The differences in power and time-to-surveillance-end between the matched and regression methods are largely explained by the reduction in data contributing to the test statistic for the matched methods (i.e., reduction to discordant matched pairs). Additionally, lack of newly accrued statistical information between analyses required us to skip some analysis times for the matched methods. Our results indicate that the choice of sequential method, particularly the choice of strategy for confounder control, is critical in rare AE observational safety monitoring settings, and further study is needed to better understand and optimize method performance.