Scalable and Intelligent Learning Systems

Abstract

Data, models, and computing are the three pillars that enable machine learning to solve real-world problems at scale. Making progress on these three domains requires not only disruptive algorithmic advances but also systems innovations that can continue to squeeze more efficiency out of modern hardware. Learning systems are in the center of every intelligent application nowadays. This thesis discusses aspects of learning systems under the context of three real-world systems -- XGBoost, MXNet, and TVM. The first half of the thesis focuses on scalable learning systems that learn parameters for complex models using large-scale data. We introduce XGBoost, a scalable tree boosting system that scales to billions of examples in distributed or memory-limited settings. We then bring a systematic approach under the context of MXNet to reduce the memory consumption of training to scale up real-world deep learning workloads using a minimal amount of resources. The second half of the thesis brings intelligence to learning systems themselves. We introduce TVM, a system for deploying learning to diverse hardware platforms. TVM exposes graph-level and operator-level optimization knobs to provide performance portability to deep learning workloads across diverse hardware back-ends. We propose transfer learning methods to automate TVM and deliver performance competitive with state-of-the-art hand-tuned libraries for low-power CPU, mobile GPU, and server-class GPU.