Some Problems in Conic, Nonsmooth, and Online Optimization

Abstract

In this thesis, we study algorithms with provable guarantees for structured optimization problemsarising in machine learning and theoretical computer science. One of the threads of this thesis is semidefinite programs (SDPs), a problem class with a variety of uses in engineering, computational mathematics, and computer science. Concretely, we study approximately solving the MaxCUT SDP. Aside from its significance as the SDP relaxation of an NP-hard problem, it has found use in matrix completion algorithms. Our algorithm for this problem combines ideas from the multiplicative weights framework and variance-reduced estimators. We adopt this idea of robust updates to give a faster high-accuracy algorithm to solve general SDPs via interior-point methods. Conceptually, our result for this general problem resolves the prior paradox of cutting-plane methods being faster at solving SDPs than interior-point methods, despite the former tracking far less structural information about the iterates. A common structural assumption on real-world datasets is that of sparsity or low rank. This structure is mathematically captured by convex non-smooth functions, thus making convex non- smooth optimization a cornerstone of signal processing and machine learning (e.g., in compressed sensing and low-rank matrix problems). Non-smooth optimization has risen in prominence on the non-convex front as well in the context of deep learning (e.g., in deep neural networks). In this thesis, we focus on two problems under the umbrella of nonsmooth optimization: In the2 convex setting, we study minimizing finite sum problems with each function depending only on a subset of the coordinates of the problem variable, and our proposed scheme develops a generalized cutting-plane framework; in the nonconvex setting, we focus on the problem of finding a Goldstein stationary point, and our solution combines randomization with geometric insights into prior work along with a novel application of cutting-plane methods. Optimization techniques have been used with great success to further progress in foundational questions in applied linear algebra. We explore this interplay in two questions. We first study least-squares regression with non-negative data and problem variables. This structure appears in several real-world datasets (e.g., in astronomy, text mining, and image processing) but has generally not been leveraged by standard least-squares algorithms (including ones in commercial software); in contrast, we utilize this structure, yielding improvements in the runtime (in both theory and experiments). We further study the computation of ℓ p Lewis weights. These are generalized importance scores of a given matrix used to sample a small number of key rows in tall data matrices and thus a crucial primitive in modern machine learning pipelines. We offer a fresh perspective to this problem, departing from the prior approach of using a fixed-point iteration. We also apply optimization theory in the context of market economics. Specifically, we study budget-constrained online advertising, an important problem for many technological companies, and develop an optimal-regret bidding algorithm under the “return-on-spend” constraint. Our main insight combines a novel white-box analysis of first-order methods for packing LPs with problem-specific structure.