Vision-based Surgical Instrument Segmentation and Endoscopic Sinus Surgery Skill Assessment

Abstract

In robot‐assisted surgery, engineering technologies are applied to augment surgeons' ability to conduct safer surgeries and achieve better treatment outcomes. To provide appropriate assistance to the surgeons, the ability to understand surgical phases, and predict risks and remaining procedures is an enabling technology for next-generation autonomous surgical robots. Vision-based surgical instrument segmentation, which aims to identify instrument regions in surgical images, is one important task that can provide instrument location information to robotic systems. The potential uses of instrument segmentation results include surgical workflow analysis, risk supervision and prediction, and surgical skill assessment. Despite the wide range of potential applications of instrument segmentation, existing technologies are still not robust enough and lack generalizability when applied to highly dynamic surgical environments. In this dissertation, we develop a convolutional neural network that can aggregate video frame features temporally in a recurrent mode to achieve robust segmentation. Moreover, we explore transfer learning technologies that can improve segmentation performance on the target domains by leveraging knowledge learned from labeled source domains. Finally, we conduct several pilot experiments on vision-based, automatic and objective surgical skill assessment.