Crosslingual Sharing for Low-Resource Natural Language Processing

Abstract

Modern NLP systems have been highly successful at a wide variety of tasks, including language modeling and structured prediction problems such as syntactic and semantic parsing. This is due in large part to the use of supervised neural networks and more recently to unsupervised contextualized representations. However, these techniques rely on resources, such as extensive task-specific annotation and vast amounts of unlabeled text, which are not available in every language. Thus, most prior research has been focused on high-resource languages such as English. Crosslingual transfer from a high-resource source language, or sharing among many languages, has increasingly been used to achieve similar improvements in low-resource languages by exploiting underlying similarities between languages; however, many techniques for crosslingual transfer in turn require crosslingual resources such as parallel corpora, which again may not be available. All of these factors pose challenges to natural language processing in low-resource languages. This thesis argues that even with little, indirect or absent crosslingual supervision, sharing information between languages is a highly effective strategy for low-resource NLP, and quantifies the benefits in various low-resource settings and languages. We describe two lines of work addressing the problem of crosslingual transfer in such low-resource settings. In the first, we present language models and supervised structured prediction models which take a joint training approach, sharing parameters across several languages, to improve performance relative to monolingual training. We begin the second with GroC, a language model with compositional input and output representations which store linguistic information independently of any specific vocabulary, and show that GroC succeeds in low-resource language modeling and monolingual domain adaptation. Finally, we unite these two threads by using joint crosslingual training for compositional language models, including ones which use crosslingual lexicons not available to previous multilingual models. We show that this combined approach improves low-resource learning for a variety of target languages.