Virtual Reality for Intracranial Brain Computer Interface Design and Human Neural Engineering

Abstract

Implanted neural devices allow direct recording and stimulation of neural tissue. Immersive virtual reality (VR) offers a powerful new experimental venue, granting researchers unprecedented control over the visual and aural landscape to develop complex tasks with nuanced controls and otherwise impossible global variable access. Modern VR hardware also guarantees precise, continuous tracking in six dimensions (three linear and three rotational) of real-world objects, namely VR controllers, headset, and trackers. In my PhD research, I sought to leverage the advantages of VR to advance neuroscience research in a unique and valuable patient population. To do this, I built the first platform integrating immersive VR with awake, in-patient, invasive neural recording, aligning highly resolved neural signals to ongoing VR task variables and human behavior.1 I then expanded this initial platform to enable direct electrical stimulation (DES) of the brain in response to behaviors and events tracked in the virtual environment. This capability allowed me to explore novel approaches to neurohaptic feedback and demonstrate the first bidirectional brain computer interface in virtual reality (VR-BCI).2 By applying DES to the primary somatosensory cortex (S1), I enabled two human subjects to reach out, grasp, and feel a purely virtual object, and then to discriminate between visually identical virtual objects based only on their neurohaptic S1-DES profile. This represented the first time that humans used a neurohaptic cortical interface that was responsive to their volitional, active engagement to explore the feel of an immersive virtual environment. Then, incorporating online digital signals processing, I demonstrated a real-time bidirectional VR-BCI that enabled the first human “cerebronaut” (so coined by our research subject) to use gaze to select an object, a neural trigger to initiate animated grasp of that object,3 and S1-DES to feel the contact between animated virtual hand and virtual object.