Automatic Summarization of Endoscopic Surgical Videos

Abstract

Endoscopic endonasal surgery is a medical procedure that utilizes an endoscopic video camera to view and manipulate a surgical site accessed through the nose. Despite these surgeries being video recorded, these videos are seldom reviewed or even saved in patient files due to the size and length of the video file. Editing to a manageable size may necessitate viewing 3 hours or more of surgical video and manually splicing together the desired segments. We suggest a novel multi-stage video summarization procedure utilizing deep semantic features, tool detections, and video frame temporal correspondences to create a representative summarization. Summarization by our method resulted in a 98.2\% reduction in overall video length while preserving 84% of key medical scenes on our data set. Furthermore, resulting summaries contained only ~1% of scenes with irrelevant detail such as endoscope lens cleaning, blurry frames, or frames external to the patient. A commercial summarization tool not designed for surgery only preserved 57% of key medical scenes in similar length summaries, and included 36% of scenes containing irrelevant detail.