Comprehensive assessment and quantification of incoherent speech using natural language processing

Abstract

Coherence is a linguistic feature that is defined as the orderly and interconnected flow of ideas. The disruption of coherence is a linguistic anomaly that is commonly observed in a group of psychiatric disorders known as schizophrenia spectrum disorders (SSD), where disorganized thoughts manifest as incoherent speech. While early detection of symptoms can potentially lead to better outcomes, manual assessment of symptom severity can be time-consuming and require specialized expertise. Therefore, symptom evaluation through automated coherence assessment methods is desired. However, gaps remain in prior research on this area, namely 1) most prior work focuses on the estimation of local coherence (coherent transitions between adjacent semantic units) via computation of cosine values between vector representations of sequential semantic units. The estimation of global coherence (sustaining a theme or topic throughout a narrative) has received much less attention; 2) the impact of automated speech recognition (ASR) errors receives little attention. Prior work mainly focused on using manual transcript data; 3) there is limited exploration on using language model perplexity to assess coherence, especially given the recent advancement of large language models (LLM). This work bridges the gaps through the following contributions: 1) Two new global coherence assessment methods were developed based on centroids of embeddings (vector representation of semantic space). We found that the global coherence methods align better with human judgment than local coherence methods. 2) A time-series feature extraction pipeline is used to replace the aggregation step in coherence assessment pipelines. We found that by using this method, coherence evaluation process is resistant to the impact of ASR errors in the text input. 3) Two sentence-level perplexity-based coherence methods were developed, and we revealed that combining perplexity features with traditional coherence scores (proximity features because they are based on cosine similarity) resulting in better prediction models than using proximity or perplexity features alone. 4) The innovations and classical approaches were combined into the Comprehensive Coherence Calculator (CCC), a software package that can perform comprehensive coherence analysis with a myriad of configurations. With these contributions, fully automated coherence assessment pipeline can be established to offer patients easy monitoring at home, clinicians necessary information to provide better care and researchers an objective quantitative basis for the study of semantic coherence.