Learning to Predict in Networks with Heterogeneous and Dynamic Synapses

Abstract

A salient difference between artificial and biological neural networks is the complexity and diversity of individual units in the latter (Tasic et al., 2018). This remarkable diversity is present in the cellular and synaptic dynamics. In this study we focus on the role in learning of one such dynamical mechanism missing from most artificial neural network models, short-term synaptic plasticity (STSP). Biological synapses have dynamics over at least two time scales: a long time scale, which maps well to synaptic changes in artificial neural networks during learning, and the short time scale of STSP, which is typically ignored. Recent studies have shown the utility of such short-term dynamics in a variety of tasks (Masse et al., 2019; Perez-Nieves et al., 2021), and networks trained with such synapses have been shown to better match recorded neuronal activity and animal behavior (Hu et al., 2020). Here, we allow the timescale of STSP in individual neurons to be learned, simultaneously with standard learning of overall synaptic weights. We study learning performance on two predictive tasks, a simple dynamical system and a more complex MNIST pixel sequence. When the number of computational units is similar to the task dimensionality, RNNs with STSP outperform standard RNN and LSTM models. A potential explanation for this improvement is the encoding of activity history in the short-term synaptic dynamics, a biological form of long short-term memory. Beyond a role for synaptic dynamics themselves, we find a reason and a role for their diversity: learned synaptic time constants become heterogeneous across training and contribute to improved prediction performance in feedforward architectures. These results demonstrate how biologically motivated neural dynamics improve performance on the fundamental task of predicting future inputs with limited computational resources, and how learning such predictions drives neural dynamics towards the diversity found in biological brains.