Designing for Interactive Systems Powered by Generally Capable AI

Abstract

The key concerns of user experience within human-computer interaction have historically evolved alongside the technology it serves. This dissertation argues that generally capable AI---in particular, large language models and the agentic systems built on top of them---motivate the next such evolution. Specifically, I identify two new dimensions of user experience design for interactive systems powered by generally capable AI: model behavior, the behavioral guidelines and constraints that govern how a model responds to users; and AI autonomy, the extent to which an AI system operates without user involvement when tackling multi-step, long-horizon work. I demonstrate that these dimensions can be made tractable through new design practices, tools, and frameworks. For model behavior, I first introduce PolicyPad, a system that conceptualizes model behavior design as a prototyping practice. PolicyPad affords iterative and collaborative authoring of model behavioral policies in conjunction with rapid testing of model behavior on sample usage scenarios. I then introduce Canvil, a tool that extends the prototyping of model behavior to design canvases, allowing practitioners to translate their design artifacts into model behavioral specifications and vice versa. For AI autonomy, I introduce a five-level framework that frames AI agent autonomy as a deliberate design decision independent of system capability. I then introduce Cocoa, an AI-powered document editor that operationalizes agent notebooks, a new interface for human-agent collaboration. In concert, these contributions pave paths for expanding how designers, developers, and researchers craft user experiences in an era of generally capable AI.