Beyond Scaling: Frontiers of Retrieval-Augmented LMs

Abstract

Language Models (LMs) have made significant progress by scaling training data and model sizes. However, they still face key limitations, including hallucinations and outdated knowledge, which undermine their reliability especially in expert domains like scientific research and software development. In this thesis, I argue that overcoming these challenges requires moving beyond monolithic LMs toward Augmented LMs: systems that are designed, trained, and deployed alongside complementary modules to improve reliability and efficiency. Specifically, my work has pioneered the field of Retrieval-Augmented LMs, which precisely locate relevant knowledge from large-scale text data and incorporate them at inference time. I begin by analyzing the limitations of current LMs and demonstrate how retrieval augmentation provides a more reliable, adaptable, and efficient path forward. I then introduce our work on establishing new foundations for Retrieval-Augmented LMs, moving beyond simple post-hoc combinations of off-the-shelf models to tackle challenges driven by broader adoption. Finally, I highlight the real-world impact of Retrieval-Augmented LMs through applications in domains such as scientific literature synthesis. Our fully open \osmodel system are now used by over 30,000 researchers. I conclude by outlining our vision for the future of Augmented LMs, including better handling of heterogeneous modalities, flexible integration with diverse components, and rigorous interdisciplinary evaluation.