Agile Legged Robots through Reinforcement Learning and Optimal Control

Abstract

This thesis addresses the challenge of developing agile legged robot controllers capable of high-speed, precise, and rapidly adaptive behaviors in real-world scenarios. While recent advancements have demonstrated impressive hardware capabilities of legged robots in controlled environments, deploying these controllers in complex, real-world environments remains a significant challenge. Traditional optimal control methods, which rely on predefined physics models to optimize motor commands, can precisely track desired motions but cannot plan for complex, long-horizon trajectories due to computational constraints. On the other hand, reinforcement learning frameworks offer the potential to learn versatile, perception-integrated motion policies end-to-end. However, they often lack the precision and robustness of optimal control methods and require extensive tuning in reward shaping and sim-to-real, for effective real-world application. In this thesis, we propose a hierarchical framework that merges the versatility of reinforcement learning with the precision of optimal control for enhanced agility of legged robots. We explore key challenges in training and deploying this framework and suggest methods to extend it to novel environments and tasks. We introduce the proposed hierarchical learning-control framework in three stages. First, we develop an early version of this framework for learning energy-efficient gait transitions in high-speed locomotion, with a high-level gait policy and a low-level centroidal controller. We then expand the interface between the high-level policy and the low-level controller for advanced control of continuous jumping motions, and restructure the low-level optimal control problem for GPU-accelerated training. Lastly, to achieve real-world terrain-aware jumping, we integrate perception into the framework and redesign critical components for robust real-world performance. With this final version of our framework, we achieve high-speed, animal-like jumping on challenging terrains such as stairs and stepping stones. Next, we extend the proposed hierarchical learning framework to novel terrains and tasks. By incorporating semantic information into the perception pipeline, we enable the legged robot to navigate quickly and safely through complex offroad terrains such as rocks, mud, or vegetation. Furthermore, we increased the capabilities of quadrupedal robots from basic locomotion to versatile loco-manipulation, with a novel lightweight gripper design and a restructured hierarchical framework optimized for teleoperation. This thesis delivers general-purpose algorithmic frameworks for perception-integrated, highly dynamic motion control of legged robots, as well as open-source implementations that can be readily deployed or further developed by the broader robotics community. To demonstrate their robustness and applicability, the methods proposed have been rigorously tested in standard real-robot benchmark tests and in diverse, complex real-world scenarios.