The Future of Cars with Human Driver Sequential Decision-Making: An Adaptive Approach

Abstract

Current transportation systems require human drivers' consistent interaction with vehicle controls. These interactions affect drivers' decision-making over time, influencing overall driving behavior. Many driving decisions can be influenced by non-driving-related tasks, such as interacting with passengers, and factors related to the dynamic traffic environment. These decisions can have short-term and long-term effects on driving, some of which can be detrimental to safety. Furthermore, understanding these driving decisions can accelerate and enhance the development of Advanced Driving Assistance Systems (ADAS) and facilitate the deployment of autonomous vehicles in the current traffic environment. Therefore, based on these motivations, naturalistic and driving simulator study data were used to address three research questions. The first research question pertains to understanding the factors influencing drivers' decision-making. Analysis reveals that distractions, influenced by variables like secondary tasks and head rotations, significantly impact driving decisions. Based on this, a driver distraction complexity score and a driver decision criticality score were developed. In the second research question, these scores are used to develop models of sequential decision-making. Two offline reinforcement learning models, Batch Constraint Q-learning (BCQ) and Conservative Q-learning (CQL), were used to learn sequential decision-making from the naturalistic data and compared. The hypothesis tested is that integrating these scores, rather than using numerous variables that could induce sparsity, can help better understand and interpret the drivers' decision-making process. This approach aims to enhance the understanding of the driver's cognitive load and the significance of driving maneuvers, thereby improving predictive capabilities in real-world scenarios. The BCQ model demonstrates robustness in environments defined by driver distraction complexity scores, showing efficiency in straightforward scenarios. However, it struggles with complex decision-making environments, indicating limitations in its adaptability to real-world scenarios.In contrast, CQL shows sensitivity to standard deviation adjustments, achieving higher rewards under these conditions. This adaptability allows CQL to perform well in varied and complex scenarios, though it introduces greater risk due to higher prediction variability. This disparity suggests BCQ's consistency and CQL's potential for higher gains but increased risk in diverse driving conditions. In the third research question, an attempt is made to improve the models of sequential decision-making. The goal is to enhance learning by refining the reward function to make it more applicable in the real world and implementing state augmentation to consider steps from previous times. The reward function is updated to incorporate driver distraction and decision criticality scores, applying non-linear penalties to capture these factors' impact better. The BCQ model demonstrates robustness under the new reward function but struggles with complex decision-making scenarios. In contrast, the CQL model performs better in managing distraction and criticality, though it exhibits greater variability. Both models adapt to increased levels of distraction and criticality, with CQL showing a slight edge in overall performance. The state augmentation improves the models' ability to account for temporal dependencies, with BCQ generally having slightly lower mean rewards than CQL. However, both models exhibit substantial variability, highlighting their adaptability to diverse driving conditions. This variability underscores the need for tailored approaches depending on the complexity of the driving environment.