Automating Stochastic Optimal Control

Abstract

Stochastic Optimal Control is an elegant and general framework for specifying and solving control problems. However, a number of issues have impeded its adoption in practical situations. In this thesis, we describe algorithmic and theoretical developments that address some of these issues. In the first part of the thesis, we address the problem of designing cost functions for control tasks. For many tasks, the appropriate cost functions are difficult to specify and high-level cost functions may not be amenable to numerical optimization. We adopt a data-driven approach to solving this problem and develop a convex optimization based algorithm for learning costs given demonstrations of desirable behavior. The next problem we tackle is modelling risk-aversion. We develop a general theory of linearly solvable optimal control capable of modelling all these preferences in a computationally tractable manner. We then study the problem of optimizing parameterized control policies. The study presents the first convex formulation of control policy optimization for arbitrary dynamical systems. Using algorithms for stochastic convex optimization, this approach leads to algorithms that are guaranteed to find the optimal policy efficiently. We describe applications of these ideas to multiple problems arising in energy systems. Finally, we outline some future possibilities for combining policy optimization and cost-learning into an integrated data-driven cost shaping framework.