Brains in the Wild: Machine learning for naturalistic, long-term neural and video recordings

Abstract

Developing useful interfaces between brains and machines is a grand challenge of neuroengineering. An effective interface has the capacity to not only interpret neural signals, but it needs to be practical, robust and work without delay in real-world applications. These applications can range from controlling cursors computers to robotic arms for paralyzed patients, all with just a thought. A real world brain computer interface (BCI) needs to be effective outside of well-controlled laboratory experiments. In this thesis. I detail our novel approach to studying long-term naturalistic electrocorticography(ECoG) for behaviour decoding and prediction, as opposed to short experimental data. We develop new machine learning techniques in order to both automatically learn behavioural annotations from the simultaneously recorded video as well as to analyze the neural correlates of behaviour from the ECoG data itself. Firstly, I outline our unsupervised approach to decode high level categories of natural behaviour and functionally map associated areas in each individual brain. My next project centers around applying deep learning in a multimodal fashion to predict spontaneous human arm movements in the future. Finally, I solve the subject transfer learning problem in ECoG in order to combine data from multiple subjects despite heterogeneity in the electrode input space and apply trained models on unseen patients. Altogether, I show through these projects that a multimodal and multi-subject approach to neural decoding "in the wild'' is critical in advancing bioelectronic technologies and human neuroscience.