Learning Scene CAD Recomposition

Abstract

Humans perceive the world in three dimensions and can interpret the hidden part of objects and scenes despite partial observations and occlusions. This level of understanding comes from imagining the hidden surfaces based on the knowledge of object shapes, scene arrangement, and high-level inference based on common scene patterns. Understanding the 3D scene with the object-level composition of its components is essential for meaningful interaction with surrounding objects in the real world physical environments. In this thesis, first, we consider the problem of scene understanding where we show that if we can decompose a scene into its prominent objects, then we can start analyzing the scene. This enables us to infer high-level information about the scene structure, such as scene recognition and scene completion. Second, we introduce a novel perspective of recomposing the CAD model of the scene. We propose transforming raw visual sensory observations in order to re-create the scene with corresponding 3D object-level components. Towards this goal, we take advantage of large databases of object CAD models and leverage learning methods to solve this problem for real-world scenes at scale. Our scene CAD recomposition re-creates the scene by matching, placing, and aligning the objects from a database of thousands of CAD models. We also propose learning-based methods to automatically recompose the scene from a single-view RGB image as well as a sequence of RGB-D images for whole scene recomposition. Finally, we incorporate scene recomposition to solve a robotic object interaction problem. By means of technical analysis and experimental studies on real-world scenes, we validate that our novel object-level scene recomposition perspective provides a useful and yet concise representation that can facilitate accomplishing downstream tasks such as object manipulation in robotics. This thesis takes first steps towards developing a fully automatic system for recomposing scenes, and we hope our work inspires future research both on 3D learning and recomposition applications.