Uncertainty in Estimating Between-Teacher Variation for Value-Added Modeling: A Bayesian Perspective

Abstract

The growing use of value-added modeling (VAM) in high-stakes personnel decisions implies that VAM teacher effect estimates can accurately differentiate higher- from lower-performing teachers. In fact, statistical power for these comparisons depends on the proportion of overall test score variation that is attributed to between-teacher differences - the intra-class correlation (ICC). This dissertation demonstrates innovative approaches to realistic treatment of ICCs as quantities estimated with uncertainty. Design priors, representing Bayesian prior beliefs about between-teacher ICC, were generated from thirty-one estimates from eight recently published VAM studies. The estimates from math test scores tended to be larger than estimates from reading test scores, and were also more variable, so separate priors were generated for math and reading estimates. This study introduces to the educational literature the use of fully Bayesian design priors to represent empirical evidence about ICC values. The fully Bayesian priors were derived from two different distributional assumptions for the likelihood - Swiger's and Fisher's distributions - and compared to empirical Bayes (kernel density) priors. Analysis using simulated data sets supports the Fisher likelihood as a reasonable description of the distribution of the published estimates. Power analysis indicates that for either math or reading, empirically supported variation in the between-teacher ICC values has little effect on power to detect teachers who differ from average. However, somewhat better power can be expected for math teacher comparisons than for reading teacher comparisons. This study strengthens the evidence that the small contribution of between-teacher variance to total variance limits the utility of VAM for teacher comparisons. Acceptable (80%) power can be achieved for one year of data only for math scores, and only for a large threshold difference of a teacher from the average: more than a third of an average annual gain. The utility of Bayesian design priors for representing uncertainty about ICC extends to other prospective analyses for VAM, as well as to hierarchical analyses more generally.