Optimization methods for parameter identifications in settings with only partial knowledge

Abstract

This work summarizes two projects focused on incorporating prior knowledge into machine learning models. In the first project, a universal feature selection method for linear mixed-effect models is developed. Namely, Sparse Relaxed Regularized Regression (SR3) is extended to the case of Linear Mixed-Effect (LME) likelihoods, and we prove that one can minimize such likelihoods with proximal gradient descent. Theoretical underpinnings of the proposed extension are also presented, including consistency results, variational properties, implementability of optimization methods, and convergence results. In particular, convergence analyses are provided for a basic implementation of SR3 for LME and an accelerated hybrid algorithm. Numerical results show the utility and speed of these algorithms on real and simulated datasets. Finally, both algorithms are implemented in an open-source python package pysr3. Conveniently, this package offers complete compatibility with scikit-learn, so all pysr3 models can be used in a pipeline with classic modeling blocks such as data pre-processors, randomized grid search, cross-validation, and quality metrics. The second line of work develops a framework for training Reduced-Order Models (ROMs) with Physics-Informed Neural Ordinary Differential Equations (PINODE). In particular, a classic technique of collocation points is adapted to transfer knowledge from a known equation to a model that approximates solutions of that equation. The addition of a physics-informed loss allows for exceptional data supply strategies that improve the performance of ROMs in data-scarce settings, where training high-quality data-driven models is impossible. The resulting ROMs extrapolate forward in time more accurately, perform better for unseen initial conditions, and exhibit less sensitivity to noise. Finally, I show how such ROMs can be used as strong regularizers in single-pixel imaging (SPI), enabling the reduction of samples-per-frame rate by an order of magnitude relative to other state-of-the-art algorithms.