Towards Autonomous Histopathological Diagnosis: An End-to-End Multi-Agent AI Framework for Diagnostic Decision-Making and Interpretation

Abstract

The rising global incidence of cancer cases necessitates the development of AI-assisted diagnostic systems to support pathologists in analyzing Whole Slide Images (WSIs). While artificial intelligence (AI) holds promise in transforming medical imaging diagnostics, current methods often face challenges with the gigapixel scale of WSIs and struggle to provide holistic, interpretable explanations for their decisions. This dissertation introduces multiple interconnected contributions aimed at developing an end-to-end AI system for interpretable histopathological diagnosis. First, we present Quilt-1M, the largest multi-modal histopathology dataset to date, comprising over one million image-text pairs curated from open-source educational videos. Quilt-1M addresses the critical challenge of data scarcity in histopathology and enables the development of QuiltNet, a vision-language model that achieves state-of-the-art performance in zero-shot classification and cross-modal retrieval tasks. Building on this foundation, we developed Quilt-Instruct to create Quilt-LLaVA, a large language and vision assistant specifically tailored for histopathology. Quilt-LLaVA can analyze WSI patches in detail, spatially localize medical concepts, and perform reasoning that extends beyond individual patches. Next, we introduce PathFinder, a multi-modal, multi-agent framework that mimics the diagnostic workflow of expert pathologists. By orchestrating specialized AI agents for triage, navigation, description, and diagnosis, PathFinder delivers an interpretable diagnostic process for entire WSIs. Pathfinder establishes a new state-of-the-art in melanoma classification, even surpassing the average performance of human pathologists by 9%. Finally, through MedicalNarratives, we demonstrate the potential to expand this approach beyond histopathology into broader medical domains by leveraging the abundance of educational content available. This dissertation advances the field of AI-assisted pathology by creating a complete pipeline—from large-scale dataset curation to sophisticated multi-modal models to coordinated AI agents—resulting in diagnostic systems that collaborate effectively with pathologists while providing human readable predictions throughout the decision-making process.