Nonconvex and Nonsmooth Inverse Problems

Abstract

Optimization approaches to inverse problems and parameter estimation have wide-ranging applications, from classical physics and biology to recently developed topics in statistical computing. Here, we focus on solving nonsmooth and nonconvex inverse problems.These properties are increasingly prevalent in modern applications yet typically approximated in many settings to simplify analysis. Such difficulties preclude common algorithms, giving rise to approaches that are highly problem specific and in many cases intractable at scale. Nonsmooth, nonconvex inverse problems arise in a wide range of fields, from PDE-constrained optimization to machine learning applications. These objectives often have composite structure; an objective which minimizes data misfit, and a regularizer that controls model complexity. These regularizers are often nonsmooth or discontinuous, while expensive cost functions must be evaluated inexactly for numerical efficiency. We develop and analyze efficient relaxation algorithms that take advantage of this composite structure, and illustrate their performance on seismic interpolation, denoising, and data-fitting problems. We deploy algorithms that solve these problems in as general a manner as possible, while allowing us to leverage problem structure. The main route of study is the creation of fast, first order splitting algorithms for approximate subproblems.In particular, we develop a family of splitting methods that first relaxes key parts of the inverse problem, and then solves an augmented problem with improved numerical properties and easier analyzation. Our key application is seismic inversion, with more additional applications to data interpolation and denoising. We extend this framework to the trust-region setting for nonlinear objectives. This requires new results that align convergence analysis from splitting methods for nonconvex and nonsmooth models with classic trust region convergence analysis. The practical implementation allows us to use derivative information from smooth problem components, and atomic operators for nonsmooth and nonconvex components, all within the context of general trust region methods for unconstrained and constrained problems. Along with theoretical results, we illustrate the efficacy of the proposed method numerically. Finally, we address convergence for a large class of splitting methods. These work for a variety of nonconvex and even nonsmooth problems, but a-priori convergence knowledge is limited and in particular requires linear constraints. We attempt to solve both of these issues by guiding convergence with augmented Lagrangian filter methods, and solve a highly nonlinear nonnegative matrix factorizaton problem with applications to chemical spectra determination. We conclude by proposing new directions that enable large-scale implementation of these algorithms, such as leveraging inexact evaluations of gradients and operators, as well as mixed precision arithmetic in next generation hardware. We also propose extensions to nonlinear least squares algorithms, implicit sampling techniques, and a new way of looking at splitting methods for PDE inverse problems.