Domain Invariant and Semantic Aware Visual Servoing

Abstract

Robots should understand both semantics and physics in order to be able to make meaningful interactions in the real world. Vision is one of the primary modalities for us humans to learn and reason about our world. Equipping robots with semantic visual understanding can increase their versatility and realizes adaptable interaction using visual feedback. In this thesis, we investigate how to teach robots learn generalizable skills for diverse real world vision-based tasks that incorporate semantics while human effort and intervention is minimized. To this end, we address major challenging problems brought by real-world constraints for teaching highly generalizable, versatile and semantic aware vision-based robot policies in low cost and safe manner. We propose domain invariant visual servoing for both manipulation and navigation that enables seamless and direct transfer of vision-based robot policies from simulation to the real world. By introducing simulation randomization (domain randomization) we make it possible to collect large volumes of robot data in a low cost and safe fashion. Additionally, we devise techniques that incorporate spatio-temporal semantic visual reasoning on RGB images to train highly generalizable and versatile vision-based robot policies in 3D simulation. More concretely, we investigate several research questions in this thesis: (1) How can learning algorithms be used to enable machines gain visual semantic understanding? (2) How can we learn robotic skills safely and how we can equip robots with visual semantics to make them capable of doing diverse tasks while requiring small amount of robot data? (3) How can we enable our robots/machines to adapt to new unknown environments and situations? I will propose several first steps towards answering these questions in this thesis.