AI-Enhanced Control for Surgical Robots - Calibration, Tracking, and Haptics

Abstract

With worldwide implementation, millions of surgeries are assisted by surgical robots. The cable-drive mechanism on many surgical robots allows flexible, light, and compact arms and tools. However, the slack and stretch of the cables and the backlash of the gears introduce inevitable errors from motor poses to joint poses, which affect the accuracy of the pose and orientation of the end-effector. Since the reported end effector position of surgical robots like Raven-II is directly calculated using the motor encoder measurements and forward kinematics, it may contain relatively large errors up to 10 mm. On the other hand, wiring complexity and sterilization also prevent the implementation of sensors on the end-effector, which have caused a lack of haptic feedback in many surgical robots ever since the beginning. When manually teleoperated, the system is closed-loop by well-trained operators, and the cable-driven inaccuracy and lack of haptic feedback are less of a concern. However, as semi-autonomous functions are introduced into abdominal surgeries, such as debridement and suturing, better closed-loop performance with better positional accuracy and force estimation is desired. However, obstacles of no direct measurements on joints and end-effectors still exist. Indirect measurements on motors rely on accurate modeling of the robot dynamics and kinematics, which can also be difficult due to cable-driven mechanisms. In contrast, machine learning methods show great potential to utilize indirect features, finding nonlinear relations among multiple factors. Thus, it can be desirable to develop a learning-based calibration that can take the original inaccurate robot states as input, and output corrected robot positional states and estimation of the contact force on the end-effector. This dissertation first develops an efficient data-driven calibration framework for the joint positions of the Raven-II surgical robot. External joint encoders are used to collect ground-truth data during calibration, while the trained models only require the original robot states during operation. The effects of calibration trajectory design, model structure, input-output selection, and long-term operation are evaluated to improve both accuracy and practical efficiency. Second, this dissertation studies how different robot-state features contribute to learning-based calibration. Through feature ablation studies, joint positions and motor torques are identified as important inputs. By selecting useful features and excluding unnecessary ones, the learning models have improved accuracy and reduced training and inference costs. Third, this dissertation investigates surgical instrument tracking through endoscopic instrument segmentation, using active learning and copy-and-paste synthetic image generation to reduce manual annotation effort. Finally, a motor-cable force actuation system and preliminary distal force sensing framework are developed for Raven-II, enabling controlled external force application and data collection for distal force estimation and haptic feedback.