Between Language and Models: Rethinking Algorithms for Encoding and Decoding Text

Abstract

Language models are statistical models that operate over real numbers, taking in a sequence of continuous vectors as input and producing a new continuous vector as output. However, users of language models need to interface with human-readable text. This is made possible by two processes: at the input level, tokenizers encode text as a sequence of discrete symbols which correspond to learned embedding representations, and at the output level, inference-time algorithms convert real-valued predictions back into generated text. In this dissertation, I present three key contributions towards better algorithms that govern this process of encoding and decoding text. First, to bridge the divergent approaches of human annotation and synthetic generation for data creation, I introduce a novel approach that combines the creativity of language model outputs with the reliability of human annotation in order to create data that is both diverse and high-quality. When applied to the task of natural language inference (NLI), we find that models trained on our dataset achieves significantly stronger generalization to out-of-domain test sets compared to existing NLI datasets. Next, I present a flexible and lightweight approach for modifying language model behavior at inference-time.The method works by intervening on the unnormalized scores predicted over the vocabulary, using small LMs that exemplify desirable or undesirable attributes in a product-of-experts fashion. I demonstrate that this can achieve the result of finetuning a language model, but by accessing only a language model's prediction over the vocabulary, not its parameters. This enables lightweight customization of API-based models, which is useful in a variety of practical settings and would not be possible with traditional finetuning. Finally, I turn to the tricky topic of tokenization, where de facto methods have not changed meaningfully in the last decade.In particular, in essentially all language models today, tokenization occurs at the level of \textit{subwords}, meaning that tokens are parts of words not bridging whitespace. Here, I introduce an algorithm for ``superword'' tokenization, extending traditional subword tokenization to include tokens spanning multiple words. In experiments, we demonstrate empirically that models pretrained from scratch with superword tokenization achieve stronger performance on downstream tasks while also being significantly more efficient at inference-time. Together, these works demonstrate the rich space of possibilities for improving how language models interface with natural language. To conclude, I outline directions for future work.