Towards Adaptive Intelligence

Abstract

Living beings, including humans, are highly adaptive, especially in terms of context and compute (resources). While intelligent machine learning systems are ubiquitous today, their current rigid design hinders adaptation as they struggle with ever-changing data, use cases, and deployment settings, requiring dedicated efforts to function properly. In this thesis, I present my work towards enabling adaptive machine learning solutions for flexible and seamless deployment across widely changing scenarios. First, I present Matryoshka information packing for adaptive data representations to handle growing data size and task-specific usage seamlessly. Then, I build a web-scale search system, AdANNS, leveraging matryoshka representations to enable adaptive search across data. Next, I extend these principles to the neural networks, crafting MatFormer models. This next-generation Transformer architecture adapts its computational footprint based on input and device with minimal overhead during deployment. Along the way, I worked on the first end-to-end learnable sparsity solution to solve the problem of optimal compute allocation across layers of neural networks. Further, to address the inherent rigidity in the design of web-scale intelligent systems, I worked on differentiable search solutions, fundamentally rethinking how large-scale AI pipelines harness data for continuous improvement. Finally, I conclude with the impact these works had in real-world deployments and present future works directed towards adaptive contextual and continual intelligence across disciplines.