Experimental Design with Scientific Applications

Abstract

Experimentation is a powerful tool to understand and optimize the world around us. The scientific method, with its emphasis on experimentation, has become the de facto means to generate knowledge across the physical, biological, and increasingly the social sciences. Experimentation is used to determine the functions of genes, design chemical substances with desired properties, evaluate the performance of new medicines, optimize products on the internet, and forecast the potential impacts of economic and social policies. Every experiment includes an experimental design, which specifies the data to be collected. A thoughtful experimental design collects the data that is most informative to the scientific question at hand, allocating the data collection budget to answer the scientific question accurately and efficiently. The importance of careful experimental design is self-evident in long, expensive experiments such as human clinical trials, in which every human subject must be justified from a cost and an ethics perspective. Even in experiments with lower marginal cost, such as high-throughput biological screening or internet A/B testing, choosing the appropriate experimental design can be the difference between making a scientific discovery or losing that signal in the experimental noise. This thesis addresses several aspects of experimental design. In Chapter 2 we ask how precise our measurements should be when that precision comes at a cost, with applications to the design and analysis of pilot experiments in the setting of high-throughput screening. In Chapters 3 and 4 we ask which of many units to measure if the units all have observed features, with applications to the optimization of antibiotic combinations and estimation in global health. In Chapter 5 we ask how to experiment on a collection of units when experimenting on one unit affects the outcomes of the other units, with applications to A/B testing in online marketplaces.