Think Through Touch: Designing a Tangible Multimodal System for Embodied Authorship in Human-AI Co-Ideation

Abstract

This thesis investigates how a tangible, multimodal interface can restructure human-AI co-ideation to support embodied cognitive engagement and authorial agency. As generative AI becomes a partner in thinking, its interface often remains a chat-based exchange: a record organized by time and operated through typed language. This thesis argues that this structure sits in tension with how people think — spatially, gesturally, and through the body — and that the resulting translation cost becomes cognitive. Over a long session, the human can gradually settle into the role of reader and editor of generated text, while authorship of the thinking process drifts toward the system. In response, the thesis develops the Think-Through Framework, a conceptual model grounded in embodied and extended cognition and extended through Material Engagement Theory. The framework proposes that AI-generated dialogue, when rendered as spatially persistent and handleable digital artifacts, can begin to acquire the properties of cognitive material: something worked, not only read. It describes a loop of Externalize, Materialize, Handle, and Think-Through, in which bodily action and cognitive change continually inform one another, with the recovery of authorship as the condition the loop works toward. The framework is explored through a gesture-elicitation study with seven design-trained participants, who mapped instinctive gestures to twelve motion stimuli on a back-projected textile surface. The study suggests that motion can function as a grammar of interaction: its temporal quality shaped how participants distinguished momentary triggers from sustained control, while its spatial scope shaped how broadly the body was expected to engage. The study also suggests that the soft, resistant materiality of the surface contributed to the sense of handling digital content as if it had substance. These insights inform FIELD, the software application developed within the Tangible Multimodal System (TMS): a network graph canvas where ideas hold spatial position, relationships become touchable, and a held fingertip and spoken phrase together direct a thought to its place. The thesis contributes a conceptual proposition for materializing AI dialogue as handleable artifacts, a working design demonstration for embodied authorship in human-AI interaction, and formative insights for future tangible multimodal AI interfaces. It identifies comparative evaluation against conventional chat-based co-ideation as the central next step. Its proposition is simple to state and demanding to build: that a person might think not only with AI, but through touch.