Optimizing Markerless Motion Tracking for the Study of Hand Kinematics

Abstract

To facilitate studies investigating motor movements and learning mechanisms involved in complex kinematic tasks, there is need for effective real time 3D motion tracking. Current methodologies for motion tracking are insufficient for 24/7 data acquisition and can interfere with experimental conditions. Developments in deep learning have created tools capable of implementing pose estimation for neuroscience. While these tools are powerful and have promising experimental usability, they struggle to achieve stable, effective tracking on complex hand postures with significant occlusions or unusual angles. We have established these shortcomings in DeepLabCut, the deep learning pose estimation model we use in building our real time motion tracking system. To address the limitations in deep learning-based automatic pose recognition, we implement an iterative training paradigm which augments the training data of the neural networks based on performance of test data. Specifically, we aim to improve generalization and reduce training time – which are necessities for effective continuous 3D tracking - by optimizing the training data with the targeted addition of data. Using the iterative training framework, evaluated networks have their training data augmented based on the network’s performance before being retrained and reevaluated. Our initial testing shows significant performance increases for the networks using this new training framework. Additionally, the networks had better tuning and generalization to complex behaviors. With the successful performance of the iterative training framework, we hope to further optimize the system by implementing the framework as a semi supervised approach.