Human-AI Interaction in Vehicles During Safety-critical Events

Abstract

While autonomous vehicles (AVs) promise to facilitate the driving task and improve human safety, people often find it difficult to fully understand or trust AVs. One reason could be that, unlike traditional vehicles with human drivers, AVs are absent of this human presence, introducing a key interaction gap between the vehicle and passengers. This lack of interaction becomes particularly profound in safety-critical events, with passengers left without clear explanations or reassurance, which can create uncertainty and further degrade trust. Communication and human adaptive responses could be significant in shaping how passengers perceive risk and calibrate their trust towards AVs during unexpected events. With the development of GenAI, there is a potential to design human interaction with AVs that are context sensitive, risk-adaptive, and emotion aware. However, it remains unclear what information passengers need in safety-critical events, when it should be provided, and through what interaction mechanisms it must be provided to calibrate appropriate levels of trust towards AVs and enable the appropriate use of autonomous capabilities. This dissertation addresses these challenges by focusing on the lack of effective human–AI interactions, particularly during safety-critical AV situations, through six interconnected studies. First, through an initial interview study with partially automated vehicle owners, the information and controls that passengers expect during AV failure events were identified. Second, building on these findings, a driving simulator experiment was conducted to examine how AV failures and a fixed, single-shot system response from the AV affect human trust. The results highlight that beyond an errorless drive and static vehicle alerts, proactive human interactions with the AV are needed to support users’ trust calibration in safety-critical events. Thus, more engaging interaction methods are necessary for AVs. Third, in a subsequent survey study, the effects of different types of AV failures (e.g., security vs. mechanical) and scenarios (e.g., perception errors, planning errors, etc.) on trust and risk perception were examined. This work demonstrates the need for AV systems to provide adaptive responses calibrated both to the risk level of specific failures and to individual risk propensities of people. Fourth, by collecting and analyzing passenger-generated social media posts documenting their AV rides, key interaction issues in real-world deployments were identified. This study underscores the importance of AVs actively assuming the role of a \textit{human} driver by providing adequate responses in unsafe situations, particularly through verbal communication. A proof-of-concept model that enables AVs to selectively monitor passenger language and deliver adaptive, emotion-aware communication was also developed. Fifth, a co-design study was conducted to derive design guidelines for human-AI communication that effectively supports passengers during safety-critical events. Lastly, these designs were implemented and tested through a driving simulator experiment, in comparison to static system responses and standard AI responses. In such a way, this dissertation ultimately provides solutions to current interaction gaps in AVs and proposes designs for effective human–AI interactions in AVs where adequate passenger support is essential.