Understanding and Addressing Uncertainty of the Crowd

Abstract

Uncertainty is crucial to understanding human judgments. Whether it's annotators producing data used to train and evaluate machine learning systems, teaching staff assigning grades to open-ended student responses, or online communities adjudicating the moderation action to apply to a piece of content, many groups and individuals need to account for and address the uncertainty that comes along with making judgments. As the application of computing technology expands to more areas of society, groups and individuals are faced with the need to make judgments on increasingly complex, subjective, and nuanced tasks at scale. This dissertation presents a set of novel tools and processes that improve upon how we capture, distinguish, and address various sources of uncertainty present in individual and collective human judgments. I will start by introducing, Goldilocks, a tool for conducting scalar rating annotations that enables different sources of uncertainty to be distinguished while also improving consistency. Then I will introduce case law crowdsourcing as a process that enables capturing similar insights about uncertainty on complex categorical classification judgment tasks by utilizing prior decisions in the form of precedent cases. Following this, I will present Cicero, a tool that addresses one specific source of uncertainty -- disagreement -- through multi-turn, contextual deliberation. Finally, I will tie together individual tools for understanding and addressing uncertainty through a dynamic workflow that applies a targeted intervention on a per-instance scale to reduce uncertainty using measurements that allow us to distinguish the source of uncertainty. I conclude by discussing the limitations of current tools and give some insights for future work in designing new tools and processes that natively support judgment under uncertainty.