A framework for video annotation, visualization, and interaction

Abstract

Existing approaches to interaction with digital video are complex, and some operations lack the immediacy of interactive feedback. In this thesis, I present a framework for video annotation, visualization, and interaction that harnesses computer vision to aid users in understanding and commu nicating with digital video. I first review the literature concerning visual representations, navigation, and manipulation of video data, and explore the literature of professional film editing, summarizing some of the techniques applied by and operations performed by film editors. I describe a new approach for computing the motion of points and objects in a video clip, and I present my interactive system that utilizes this data to visually annotate independently moving objects in the video, including speech and thought balloons, video graffiti, hyperlinks, and path arrows. I also demonstrate an application of this interface to construct visualizations of a short video clip in a single static image, using the visual language of storyboards. The principal advantage of the storyboard representation over standard representations of video is that it requires only a moment to observe and comprehend but at the same time retains much of the detail of the source video. The layout of the storyboard can be optimized to place the elements in a configuration that maximizes the clarity of presentation. Finally, I also demonstrate two novel interaction techniques for random video frame access using either the natural spatial dimensions of a storyboard representation or an individual video frame.Throughout the thesis, I discuss how these approaches simplify and streamline the understanding and manipulation of video materials.