Estimating Individual-Level Interaction Effects in Multilevel Models: A Monte Carlo Simulation Study with Application

Abstract

Moderated multiple regression (MMR) provides a useful framework for understanding moderator variables. When the relationship between an independent variable (predictor) and a dependent variable (criterion) varies as a function of a third variable, that third variable is considered a moderator of the predictor-criterion relationship. Moderated relationships can also be examined within datasets that include nesting, for example individuals might be nested within groups. However, the literature is not clear on the best way to assess data for significant moderating effects, particularly within a multilevel modeling framework. This study explores potential ways to test moderation at the individual level (level 1) within a 2-level multilevel modeling framework, with varying effect sizes, cluster sizes, and numbers of clusters that represent realistic conditions in applied educational research. The study examines five potential methods for testing interaction effects: the Wald test, F-test, likelihood ratio test, BIC, and AIC. For each method, the simulation study examines how Type I error rates vary as a function of number of clusters and cluster size and how power varies as a function of number of clusters, cluster size, and interaction effect size. Following the simulation study, an applied study uses real data to assess interaction effects using the same five methods. Results indicate that the Wald test, F-test, and likelihood ratio test all perform similarly in terms of Type I error rates and power. Type I error rates for the AIC are more liberal, and for the BIC typically more conservative. A four-step procedure for applied researchers interested in examining interaction effects in multi-level models is provided.