Interactive Playspaces for Object Assembly and Digital Storytelling

Abstract

Today we observe a consistent shift towards doing our tasks virtually through machines. This mode of work ensures that the users are not tied by lack of resources required for the task, and get additional advantages like ability to make quick corrections and share the result remotely. Researchers in the field of human computer interaction have constantly pushed towards tangible user interfaces which allow the users to get a sense of doing the task physically while it happens virtually. Designing interfaces for 3-dimensional tasks poses interesting challenges. The traditional desktop or tabletop setups do not work very well here because it is hard for the user to visualize the 3D virtual world using 2-dimensional displays and control them using un-intuitive devices like keyboard/mouse/joystick etc. Researchers and industry have explored augmented reality-style or immersive environments-based interfaces to let users interact with the virtual world. However, most of these interfaces are too specialized and hard to set up. In this thesis, I explore an easy to set up interactive environment, called a playspace, for a variety of 3D tasks. The user performs the task while a color+depth camera observes and understands the task in real-time. It then presents context-specific feedback and automatically reflects the inferred activity in a virtual world on a screen in front of the user. The playspace also integrates other input modalities such as gestures, voice commands and standard devices like keyboard and mouse. The modular nature of the framework allows different applications to plug into the playspace environment easily. Playspaces allow users to do a task physically while it is virtually replicated on the fly. The virtual result can then be post-processed or edited in real-time, again through physical props. The framework also opens the opportunities to assist users in real-time. I have developed and evaluated three applications in the playspace environment -- 1. Block model assembly - The system automatically learns and builds a virtual replica of a Duplo block model by observing the user build it. It also assists the user in creating a predefined model in a novel way while detecting any mistakes and assisting in making any corrections on the fly. I report on a user study that shows that the proposed guidance method is better than the traditional figure-based guidance method. 2. Digital storytelling - The system allows a user to act out a story using rigid puppets and automatically converts that into an animation. Further, it also allows the user to record multiple takes for the same story and merge them automatically after the user has roughly annotated them based on his liking. This is helpful when the user wants to try out different styles and later merge them. I report on a user study to test this utility. 3. Designing 3D environment prototypes - The system allows the user to easily manipulate virtual objects in a scene by “attaching” them to a physical object of user's choice. The user can add, move, scale, clone or delete objects from a database, thus creating simple 3D virtual environments. The user can also paint the terrain in the virtual world by using textures from his surroundings.