Building A User-Centric and Content-Driven Socialbot

Abstract

Researchers in artificial intelligence (AI) have long been interested in the challenge of developing a system that can have a coherent conversation with humans. The Loebner Prize is a form of Turing test that has challenged researchers since 1990. More recently, several commercial products of conversational AI have emerged in the market, such as Amazon Alexa, Microsoft Cortana, Google Assistant, and Apple Siri. Research in both task-oriented systems that aim at accomplishing a well-defined task and chatbots that engage users in chit-chat interactions have made considerable advancement. The recent Alexa Prize sets forth a new challenge: creating a socialbot that can hold a coherent and engaging conversation on current events and popular topics such as sports, politics, entertainment, fashion, and technology. This thesis was born out of the Alexa Prize. Our socialbot, Sounding Board, demonstrated that it is feasible to build a system that can engage in long conversations when backed by rich content crawled from the web and knowledge of the user obtained through interaction. This user-centric and content-driven design helped Sounding Board win the inaugural Alexa Prize with an average score of 3.17 on a 5-point scale and an average conversation duration of 10:22, evaluated by a panel of independent judges. To build Sounding Board, we develop a system architecture that is capable of accommodating dialog strategies that we designed for socialbot conversations. The architecture consists of a multi-dimensional language understanding module for analyzing user utterances, a hierarchical dialog management framework for dialog context tracking and complex dialog control, and a language generation process that realizes the response plan and makes adjustments for speech synthesis. Additionally, we construct a new knowledge base to power the socialbot by collecting social chat content from a variety of sources. An important contribution of the system is the synergy between the knowledge base and the dialog management, i.e., the use of a graph structure to organize the knowledge base that makes dialog control very efficient in bringing related content to the discussion. The Alexa Prize offers a new and unique platform for researchers to build and test socialbots by allowing the systems to interact with millions of real users through Alexa-enabled devices. Using the data collected from Sounding Board during the competition, we carry out in-depth analyses of socialbot conversations and user ratings which provide valuable insights in evaluation methods for socialbots. We additionally investigate a new approach for system evaluation and diagnosis that allows scoring individual dialog segments in the conversation. Finally, observing that socialbots suffer from the issue of shallow conversations about topics associated with unstructured data, we study the problem of enabling extended socialbot conversations grounded on a document. To bring together machine reading and dialog control techniques, a graph-based document representation is proposed, together with methods for automatically constructing the graph. Using the graph-based representation, dialog control can be carried out by retrieving nodes or moving along edges in the graph. To illustrate the usage, a mixed-initiative dialog strategy is designed for socialbot conversations on news articles. A new socialbot prototype is developed for user studies demonstrating the benefits of the proposed document representation and dialog strategies.