Realistically Editing Indoor Scenes

Abstract

Mixed reality is an exciting application of computer graphics that seamlessly combines the real and the virtual. Many of the most compelling mixed reality applications involve modifying the contents of the surrounding real-world scene, whether it is adding a virtual game character running around your room, replacing your existing couch with a new one in a furniture retail app, or changing the style of your clothing in a fashion app. For most applications, especially those for which the primary goal is aesthetic evaluation, these edits need to be done in a visually realistic way: something that looks like a bad Photoshop job is nowhere near as useful as something that looks like you had physically placed or moved something in your room. In this thesis, I describe methods for reconstructing models of an indoor scene’s lighting and materials in a way that enables realistic, physically consistent edits to the scene. These edits include not only inserting virtual objects into a room, but also removing existing objects, as well as relighting and retexturing. I first outline a method to scan scenes to acquire a 3D mesh textured with a high dynamic range representation of the scene’s appearance. I then show how to use this captured data to infer parametric models of the room’s base geometry (e.g walls, floor, doors, windows). I present an inverse rendering framework to use the high dynamic range data to simultaneously infer light intensity distributions and diffuse reflectances across an entire indoor scene that enables realistic visualizations of emptying, refurnishing, and relighting a room. I subsequently derive a method to solve for not just the intensities but also the number and locations of unobserved local light emitters in a scene. Finally, I describe a neural rendering method to reconcile the differences between a scene’s true appearance and its reconstructed scene model.