Variational Problems in Feature Selection and Data Compression

Abstract

Feature selection and data compression are fundamental problems in dimension reduction and representation learning. This dissertation develops a common variational framework for these tasks, posing them as quadratic optimization over spaces of probability distributions. The first substantive chapter introduces the local expected gradient outer product (local EGOP) motivated by high-dimensional nonparametric regression. It is utilized in a recursive feature-learning algorithm which yields an intrinsic learning rate for signals parameterized by low-dimensional manifolds. The second substantive chapter develops CO2 for selecting convexly weighted coresets to approximate measures with respect to smooth divergences. Second-order Hadamard differentiability turns a divergence's Hessian into a loss-adapted kernel, reducing local compression to maximum mean discrepancy minimization and kernel quadrature. For the Sinkhorn divergence, this geometry is equivalent to a scaled Gaussian reproducing kernel Hilbert space and supports coresets with poly-logarithmically many observations. The final substantive chapter analyzes entropic self-transport as the regularization parameter $\varepsilon$ tends to zero. A second-order expansion of the symmetric Schr\"odinger potential makes explicit its relationship to Gaussian-kernel estimators of density, score, and diffusion.