How to Politely Re-dip Your Chip! On the Use of Data-Based Informative Priors in Linear Mixed Models

Abstract

The use of Bayesian analyses in social science research has been on the rise, yet the issue of prior specification still poses theoretical controversies and practical challenges. In educational psychology, the prevalence Bayesian analysis and choice of priors is currently unknown, and the impact of using sample-model-based informative priors for multilevel models has yet to be evaluated. The current study therefore investigates: 1) the use of Bayesian analyses and prior specification choices in recent applied educational psychology research, and 2) the consequences of using increasingly informative sample-model-based priors (“double-dipping”) on fixed effect coefficient parameter recovery for 2-level hierarchical linear models. Our results show that, first, applied researchers tend to rely on software default priors (i.e., noninformative or weakly informative priors), and on rare occasions where informative priors are used, about one-third rely on sample-related values. Second, our simulation results show that posterior standard errors are progressively underestimated (leading to over-credibility) as fixed effect coefficient sample-model-based prior informativeness increases, particularly for conditions involving a larger number of clusters (L2 sample size). Third, the best approach for obtaining unbiased fixed effect coefficient credible intervals is to use weakly informative priors; the next-best alternative is to use a cross-validation method whereby a random half of the data is modeled using uninformative priors to obtain sample-model-based priors for a subsequent model for the other half of the data. These findings are consistent with previous methodological work warning that data-based priors require careful implementation. Limitations and future directions are discussed.