Assessing the Accuracy of Provider Profiling Methods for Classification

Abstract

Provider profiling as a means to describe and compare performance of health care professionals has gained great momentum in the past decade. The implications of profiling, which can drive provider incentives and guide health policy, call for precise and accurate statistical methods. We used a simulation study to compare the performance of three commonly used methods for estimating provider performance (ranking) and for identifying high performing providers (classifying). We evaluated classification performance based on sensitivity and specificity and ranking performance based on mean squared error. We found that when between-provider variability in performance was low, all three methods performed poorly, with low accuracy for identifying top performers and high mean squared error for ranking. We then demonstrated the performance of these methods in an application to data on satisfaction with mental health care providers. Based on these findings, we caution against the use of any classification method in the setting of low between-provider variability and recommend the use of risk-adjusted methods, which take into account variation in characteristics of providers' patients, when the ratio of between-provider variability to within-provider variability is high.