Co-adaptive Human-Machine Interaction with Multimodal Inputs

Abstract

Human-machine interfaces record biosignals generated by humans — such as neural (brain), visual (gaze), or limb movement — and decode them into control commands for computational, robotic, and assistive systems. However, the widespread clinical and consumer adoption of biosignal-based human-machine interfaces remains limited. Conventional systems struggle to generalize across diverse user populations, leading to poor usability and high abandonment rates, particularly among individuals with motor impairments. There is a need to seamlessly tailor these interfaces to diverse users and contexts, providing an "out-of-the-box" solution that requires no expert setup. This problem has traditionally been challenging because biosignals have large variability, which leads to extensive interface calibration across people and within the same user over time. This dissertation builds on prior research to address this challenge, laying the groundwork for interfaces personalized to individual needs and abilities. Specifically, I combined theoretical frameworks from control theory with data-driven and neuroengineering methods to: (1) model humans as control systems interacting with machines to investigate user strategies and adaptations in multimodal interfaces, and use these insights to (2) design intelligent interfaces that co-adapt in real time — that is, adapt in response to users’ ongoing adaptation. This dissertation demonstrates how I integrated systems with multimodal inputs, including surface electromyography (EMG), eye tracking, and manual joystick, and used experimental data from human participants to model user behaviors. I then developed machine learning algorithms to engineer interfaces that automatically adapt to individual usage, allowing continuous calibration with multimodal inputs. This setup provided a new paradigm to investigate the emerging co-adaptation patterns between human and intelligent interfaces. In summary, this work provides new theoretical, computational, and experimental tools for personalizing adaptive, multimodal interfaces to diverse users and tasks. Moving forward, I plan to implement the frameworks in rehabilitation programs, such as robot-guided therapy, to deliver personalized interventions for people with motor impairments. Together, this work will enhance the usability, generalizability, and clinical translation of biosignal-based human-machine interfaces.