Perceptual Optimizations for Video Capture, Processing, and Storage Systems

Abstract

Visual media is the dominant form of content used in modern computing systems. Advances in machine learning, virtual reality, and display form factors drive demand for richer visual experiences, putting pressure on systems to efficiently use compute and storage infrastructure. At the same time, the rapid pace of performance and energy efficiency gains computer architects depended on to meet growing application requirements has slowed. Designing computer systems to meet the requirements of modern video-based applications requires specialization in compute design, using hardware-software codesign techniques to closely optimize computer system performance for specific visual computing workloads. This thesis uses perceptual information to optimize the design of video capture, processing and storage systems. I describe system optimizations using three classes of perceptual cues: structure (e.g., color, depth); semantics (e.g., faces, objects); and saliency (e.g., human visual saliency, neural network feature saliency). This thesis demonstrates how perceptual information can be used in hardware accelerator designs on ASICs and FPGAs, and in cloud video storage infrastructure.