Advancing Differentially Private Optimization: Efficiency, Utility, and Applications

Abstract

With the rapid development and widespread application of modern machine learning and artificial intelligence—particularly following the emergence of large language models—privacy has become a critical concern.Differential privacy, a rigorous mathematical framework for defining privacy, has emerged as the de facto standard. This thesis addresses two fundamental problems in privacy-preserving machine learning: differentially private empirical risk minimization (DP-ERM) and differentially private stochastic (convex) optimization (DP-SCO). Our goal is to design more efficient algorithms for these problems while achieving better and optimal privacy-utility trade-offs. The thesis is structured into five parts: (1) Part I focuses on improving and achieving near-optimal gradient or function value computation complexity. (2) Part II extends the analysis under alternative geometries and norms beyond the classic Euclidean spaces. (3) Part III investigates non-convex functions, which are increasingly common in practice and gaining significant attention. (4) Part IV examines the user-level differential privacy setting, a practical scenario where users contribute multiple items, as opposed to the classical item-level DP assumption of a single item per user. (5) Part V explores additional settings, including online optimization, heavy-tailed distributions, and low-rank structures. This work comprehensively explores these challenges, proposing innovative methods to enhance algorithmic efficiency and optimize the privacy-utility trade-off.