The Suitability of Generative Adversarial Training for BERT Natural Language Generation

Abstract

This thesis presents a study that was designed to test the effect of generative adversarial network (GAN) training on the quality of natural language generation (NLG) using a pre-trained language model architecture: Bidirectional Encoder Representations from Transformers (BERT). Perplexity and BLEU scores were used as metrics for evaluation on 1000 samples of generated text. Results indicated that perplexity decreased and BLEU scores comparing the original data distributions increased; thus, there was evidence that quality of NLG was improved by the introduction of GAN training. This alternative training method may also be effective for other more state-of-the-art pre-trained architectures.