Tissue Characterization with Surgical Smart Grasper and Hybrid CNN-GRU Model

Abstract

Minimally Invasive Surgery (MIS) is now standard in modern medicine and involves operating through small incisions using laparoscopic instruments (graspers) for manipulation and endoscopic cameras for visual feedback. Advantages of MIS include faster recovery, less blood loss, and lower risk of complications. Alongside its benefits, MIS brings new challenges, such as the lack of tactile feedback for surgeons because, in open surgery, surgeons can palpate the tissue to gain information about its non-visible structure and abnormalities, such as tumors, blood vessels, and foreign bodies (lost surgical instruments or shrapnel). Adding miniaturizing sensors to laparoscopic graspers can solve the problem of palpating and sensing for surgeons. Such devices paired with artificial intelligence (AI) algorithms for noise removal and data processing can provide valuable new information to surgeons and help with early diagnosis and treatment. The thesis aims to extend the work done by Philip Roan on the motorized Smart Grasper Robot. The thesis involves re-designing all electronics involved in Roan's design to accommodate integrating a new sensor - a 1D A-mode ultrasound sensor- into one of the grasper's tips. Ultrasound transducers were not used in bidirectional mode on surgical graspers before, which gives a set of problems solved in the thesis. One issue involves finding the methods for time series ultrasound signals to clear them from the ringing artifact and the noise that interferes with the echo. Another problem is estimating the Time of Flight on filtered ultrasound signals.Another problem the thesis aims to solve is evaluating and developing a multimodal AI algorithm that can process all sensor modalities simultaneously. Multimodal AI algorithms with physiological data are rarely used in research nowadays, but having the data from different sensor modalities that can measure and process simultaneously a lot of vital parameters from the patient could give valuable information for surgeons of minimal changes in the patient's vitals during the surgery. As a result of the thesis, we collected in-vitro and in-vivo multimodal time series data from animals and released it openly to use it for the community to train more sophisticated Deep Learning models.