Factors influencing Audiovisual Speech Integration

Abstract

Audiovisual (AV) integration, identified as a pivotal factor in comprehending speechin noisy environments, is a complex phenomenon. Understanding speech perception, even within a single modality, presents various nuances due to language specificity. When consolidating information from multiple modalities, it is imperative to understand how the listener processes the speech signals in each modality, and how this information is successfully integrated to benefit our speech understanding. Prelinguistic integration mechanisms, such as synchronous temporal information from both modalities, have a significant role in identifying AV events. Yet, linguistic integration mechanisms, like phoneme-viseme connections of the AV speech signal or individual linguistic knowledge, significantly influence speech intelligibility. This dissertation contains two sections. First, we outline three remote AV speech perception tasks across developmental stages, and in developmental disorders such as developmental dyslexia (ages 4-15; n = 261). Second, we present a series of four remote AV psychophysical tasks in adults, ages 21-40 (n = 46), to elucidate the role of prelinguistic and linguistic features pertinent to AV speech integration. For the developmental work we find that weighting assigned to the auditory modality in the AV speech signal serves as a better explanation for individual variability across development than age itself. Moreover, atypical weighting of auditory modality explains differences between children with and without developmental dyslexia on a group level. In adults, our findings suggest that how well temporal asynchrony between the auditory and visual signal is tolerated depends not only on the linguistic complexity of the stimulus, but also on the individual. Prelinguistic information like temporal synchrony perception has an important role in AV speech perception as it endows a 3 dB increase in perceived loudness perception of the target speaker, but this increment interacts with linguistic complexity and temporal asynchrony. Together, these results offer novel insights into different factors influencing AV speech integration.