Data-Constrained Model Compression

Abstract

In recent years, strong progress has been made in compressing compute-heavy machine learning models to enable them to execute in real-time on edge devices. Typically, model compression techniques require retraining a model on the original dataset of interest. This is problematic if the original dataset is unavailable due to privacy or legal concerns, or if the model to be compressed was obtained from a third party. We explore the challenges associated with compressing a model in three different data-constrained scenarios. In the first scenario, labels are unavailable. We approach this problem through knowledge distillation, training a smaller model using predictions made from a larger model on unlabeled data. In the second scenario, both data and labels are unavailable. We approach this problem by separately compressing every layer of a pretrained model to obtain a compressed approximation of the original model. Our method is computationally efficient, achieving strong compression rates while maintaining accuracy. In the third scenario, we explore the problem of dynamic, real-time compression after model deployment. We demonstrate a training technique in which we condition a model to achieve high accuracy across a variety of compression levels, allowing for efficient, real-time model selection along the efficiency-accuracy trade-off curve after model deployment. We present these works to elucidate the challenges associated with data-constrained model compression, and to provide solutions for compressing models in these challenging scenarios.