When Models Meet Data: Pragmatic Robot Learning with Model-based Optimization

Abstract

Autonomous robots operating in complex and dynamic real-world scenarios must exhibit fast and reactive behaviors to adapt to environment changes, and learn to improve their performance over time. While recent advances in reinforcement learning (RL) have shown remarkable progress in learning complex decision-making directly from experience, the sample inefficiency of existing methods limits their application to robotics domains where data collection can be dangerous, time-consuming and expensive. In contrast, traditional model-based optimization approaches for robot control can leverage prior knowledge of physics and task structure to efficiently plan a robot’s motion, and have a rich history of successful application in safety-critical systems. However, the complexity of real-world environments can be hard to model precisely, and assumptions made by practical approaches can significantly limit performance. In this thesis, we propose principled methods to combine data-driven approaches with model-based optimization that enable efficient, adaptive and self-improving robots. We address key challenges in designing such methods with varying degrees of prior model knowledge. First, we show that even with access to perfect models, the real-time performance of existing motion planning algorithms can be sensitive to environment changes. We propose adaptive planning frameworks that leverage past experience to directly optimize the performance of motion planners across the distribution of environments the robot encounters. Second, for tasks where our prior models are approximate, long horizon planning can be error prone. We show how model-predictive control (MPC), offers an efficient solution for generating adaptive behaviors via finite horizon optimization. We present a GPU accelerated MPC framework for real-world reactive manipulation that can rapidly optimize complex task objectives while ensuring qualitative behavior requirements. Further, we present a general framework that improves over MPC using model-free RL to overcome the effects of model-bias over time. Finally, we consider situations without access to prior models, and explore policy learning from static datasets of interactions. We show how learning predictive models from such datasets in an adversarial manner can enable learning policies that can improve over arbitrary reference policies regardless of data coverage, with formal guarantees. This thesis contributes general algorithmic frameworks that are broadly applicable across robotics domains, as well as efficient open-source implementations of practical systems that can be leveraged by the wider community. The methods we present are supported by strong theoretical guarantees and empirical performance across a wide variety of benchmark tasks, and real-world manipulator control in dynamic environments.