Towards More Flexible Models in High Dimensions

Abstract

Recently, technological advances have allowed us to gather large and high-dimensional data. In high-dimensional data, the number of variables measured on each subject is quite large, often larger than the number of subjects. Consequently, there is growing need for improved supervised learning methods. We consider the setting in which we have an outcome variable and p covariates measured for n subjects; our goal is to estimate the conditional relationship between covariates and outcome. Fitting linear models to high-dimensional data has been extensively studied in the past two decades, and numerous methods have been proposed for this task, such as the lasso. On the other hand, more flexible or nonparametric modeling of high-dimensional data is relatively less studied. Desirable flexible models should be interpretable, computationally-efficient, and have theoretical guarantees. Existing literature fails to achieve these three properties simultaneously. In this dissertation, we extend the existing literature and address gaps within the literature. In Chapter 2, we present a general framework for fitting sparse interaction models. Our framework not only generalizes many existing methods, but allows us to build new estimators; we present two such novel estimators in Chapter 2. In Chapter 3, we develop a general framework for fitting sparse additive models; this framework encompasses state-of-the-art techniques for additive models. We develop an efficient algorithm for computation, and establish theoretical guarantees for our general framework. In Chapters 4 and 5, we develop two novel estimators for nonparametric regression and extend them to sparse additive models. The main appeal of these estimators is that the fitted models have a parsimonious representation; this facilitates interpretation of models. Using the general framework of Chapter 3, we derive efficient algorithms for the estimators of Chapters 4 and 5, and establish theoretical convergence rates.