Exploiting Image Resolution Holistically in Computer Vision Models

Abstract

Modern computer vision systems are built upon a complex stack of hardware and software, from general purpose processors to specialized accelerators, and low level operator libraries to expressive deep learning frameworks. However, from this complexity arises many opportunities for optimization across the stack. From the lens of image resolution, a fundamental hyperparameter of computer vision, we propose methods for optimizing models and characterize the space of choices as introduced by the hyperparameter of resolution. In the process, we cover related topics such as object scale (as introduced by data augmentations), image storage (including methods for efficient multi-resolution storage), and deep learning kernel tuning. An understanding of these topics allows us to consider resolution with respect to deep learning choices holistically, enabling efficient inference from the metrics of computational cost, latency, accuracy, and storage bandwidth use. We describe the mechanisms by which we enable efficient inference according to these metrics, spanning kernel tuning, image data layout, and model architecture pipelines.