Knowledge-driven Natural Language Understanding

Abstract

Recent advances in natural language processing (NLP) mainly relied on pre-trained language models (PLMs) that are trained on vast amounts of data. Although these PLMs achieve remarkable success in improving NLP performance over conventional approaches, they still struggle to accurately understand the semantics of the text. Therefore, extra knowledge, especially the dynamically extracted one, is expected to be leveraged to improve the understanding of the models in the text. This thesis proposes a knowledge-driven approach for NLP that improves PLMs. By dynamically integrating external knowledge from multiple sources, the proposed approach enhances model generalization in different scenarios. Specifically, the thesis leverages three types of knowledge, namely, the lexicon knowledge (e.g., n-grams) extracted directly from raw data, syntax knowledge (e.g., dependency parse trees) obtained through existing toolkits, and the pattern knowledge (e.g., vectors) captured during the training process. Several novel architectures are proposed to leverage the knowledge, such as key-value memory networks for incorporating wordhood information, span attention mechanisms with categorical grouping for improved syntactic parsing, and graph convolutional networks to further enrich contextual representations. Extensive experiments on different NLP tasks at various levels demonstrate the effectiveness of the proposed approaches, which outperform strong baselines and existing studies. Overall, this dissertation not only broadens the definition and utilization of knowledge in natural language processing but also lays a solid foundation for future research in multi-modal, cross-domain, and low-resource environments.