Causality, Fairness, and Information in Peer Review

Abstract

In this dissertation, I study peer review---the process by which scientists evaluate one another's work for publication or funding---through three distinct but related lenses. I focus on multi-step grant proposal peer review processes, in which reviewers score a research grant proposal on a set of criteria (via criterion scores) as well as overall. The National Institutes of Health (NIH) and American Institute of Biological Sciences (AIBS) both use this type of peer review. We begin by analyzing racial disparities in NIH peer review scores, determining that the criterion scores explain racial disparities in overall scores. We also find---perhaps surprisingly---negligible racial differences in commensuration, the way in which criterion scores are weighed when determining the overall score and a potential vector for racial disparities. Our analysis uses hierarchical mixed-effects models to account for the intricate dependencies in the NIH's peer review structure and matching to nonparametrically adjust for covariates. Additionally, I discuss the conditions under which estimates from these models carry causal interpretations, and investigate the robustness of our estimates to deviations from these assumptions. Outstanding questions from the NIH study motivate the subsequent chapters of the dissertation. The unmeasurability of a grant proposal's underlying quality---a sure mediator of the relationship between demographics and peer review scores---leads us to explore a related question: how informative are peer review scores? We leverage the decimal AIBS scoring scale and the proven tendency of raters to round to define and study refinement, which characterizes the informativeness of a set of peer review scores. We find evidence that overall scores are more informative than criterion scores at AIBS. Finally, the experimentally unverifiable causal structure underlying our NIH study's racial disparity models motivates us to adapt causal discovery techniques for use in peer review. We quantify uncertainty in discovery with a fully Bayesian approach---Bayesian Causal Discovery---that enables researchers to establish confidence in the causal structures that underpin future analyses.