Towards More Intelligent, Intuitive, and Inclusive Communication with Computers in Text and Images

Abstract

Communication is fundamental to human experience and our interaction with computers. The efficiency of human communication relies largely on the quality of the medium. Modern computing devices offer mediums such as keyboards to interact with information, yet they are still primitive (e.g., one has to press every character of a phrase) and often fail to support different user needs (e.g., limited emoji support for visually-impaired users). Thus an important question is, how to make the keyboard understand our languages like a human being, so that communication with computers can be intelligent, intuitive and inclusive? In this dissertation, I demonstrate how to design, build and evaluate communication interactions using the power of artificial intelligence. My dissertation addresses the above question in three different strands of work: (1) Intelligent text entry and editing interactions that understand the user intention; (2) Assistive systems that help blind or low vision users to communicate with pictorial information such as emojis and GIFs; (3) Models and metrics that evaluate intelligent input systems on their performance and impact on human behaviors. Together, the work demonstrates the thesis statement: Artificial intelligence can enable and improve advanced text production and accessible interactions with pictures; in addition, new metrics for text entry enable the evaluation of advanced capabilities.