Neural Mechanisms of Trial and Error Learning: a Study from Bird Song

Abstract

In this dissertation I examine how variation is generated, shaped and controlled in the brain during trial-and-error learning. The first part of this dissertation concerns the mathematical modeling of networks of neurons that constitute part of the neural architecture of song bird learning. Chapter 1 reviews the theories of learning in neuroscience that shape our subsequent modelling and analysis results as well as background on the song bird neural system and behavior. Chapter 2 examines the modulation of a microcircuit within the basal ganglia and proposes a mechanistic way in which dopamine could modulate the signaling properties of the nucleus to influence behavioral changes in song variability. Chapter 3 addresses the ability of a motor system to maintain stereotyped behavior over long periods of time, through adaptation to changes such as injury or aging. In the context of a model of bird song learning, ongoing instability in neural representation of stable behavior allows a system to more readily adapt and maintain performance with minimal cost. In this framework, behaviors are made more robust to environmental change by continually seeking new ways of performing the same task. Chapter 4 examines the way that exploration in trial and error learning is shaped by network properties. A reinforcement learning system, inspired by bird song architecture, is able to successfully learn when exploration is driven by variable network dynamics. Further, learning is made more successful when the exploratory dynamics from which variations are selected partially align with the elementary components of the desired behavior. The second part of this dissertation develops a method of analysis to compare song behavior to patterns of neural activity and suggests an interpretation of the covarying neural-behavioral activity that operates within the theoretical framework of reinforcement learning. Chapter 5 presents an analysis of the covariations between song and neural activity in the ventral tegmental area (abr. VTA) of the zebra finch. This analysis provides evidence that dopamine neurons in VTA encode representations of song error during natural behavior. Additionally, diverse components useful for error calculation exist locally within VTA, which could contribute to the final error signal. Lastly, novel, long-timescale state variations in song and cell activity are present in a subpopulation of VTA; I propose an interpretation in which the error calculation incorporates a normalizing operation relative to internal states.