From Research to UX Practice: Evaluation Approaches and Tools for Realizing Human-Centered AI Goals

Abstract

The integration of AI models in various products and services poses unique challenges for design and UX practice. Unlike other technologies, AI poses distinct challenges due to its probabilistic nature, technical complexity, and lack of precedent. While collaboration with AI practitioners can help alleviate these challenges, there are significant communication, process, and knowledge barriers to overcome. In this dissertation, I present formative, design, and evaluative research to better support UX practice of AI and UX-AI collaborations, which in turn can support human-centered design of AI products. First, I conduct a qualitative study to understand key challenges that UX practitioners face in model comprehension and reinforcing a human-centered lens on model evaluation. Based on these findings, I design and test visualization-based methods that enable UX practitioners to infer insights about human-AI alignment. I also examine how practitioner tools are currently designed to support interdisciplinary collaborations, such as UX-AI collaborations, through a design space analysis. Based on these three studies, I derive implications for designing evaluation tools for AI/LLM applications that can better support UX practitioners’ needs and improve HCI/UX and AI collaborations.