AI-driven vision-based navigation for robotic minimally invasive surgery
File(s)
Author(s)
Xu, Haozheng
Type
Thesis
Abstract
Minimally invasive surgery (MIS) has revolutionised patient care by reducing tissue trauma, recovery time, and post-operative complications. However, the constrained field of view, reliance on external tracking hardware, and dynamic surgical environments present significant challenges for surgical navigation and robotic assistance. Current approaches—such as electromagnetic trackers, optical markers, and kinematic encoders—either disrupt clinical workflows or lack robustness to occlusion, tissue deformation, and instrument motion.
This thesis develops novel AI-driven, vision-only methods for surgical navigation, enabling accurate and robust perception without reliance on external markers or hardware. The work makes five key contributions. First, new markerless algorithms for surgical instrument pose estimation are introduced, including a graph-based framework for 5-Degree of Freedom (DoF) tracking of textureless tools and an occlusion-robust 6-DoF estimator validated on the da Vinci Research Kit (dVRK). Second, a diffusion-based stereo depth estimation framework, StereoDiffusion, is proposed to achieve temporally consistent disparity maps under the challenging visual conditions of MIS. Third, the thesis presents SurgRIPE, a large-scale dataset and benchmarking challenge for 6-DoF surgical instrument pose estimation, establishing the first public standard for this task. Fourth, an AI-assisted robotic control framework is developed, fusing vision-based estimates with robot kinematics via an Extended Kalman Filter (EKF) to enable real-time closed-loop instrument manipulation. Finally, novel methods for dynamic 3D reconstruction of deformable tissues are proposed, providing surgeons with reliable intra-operative visualisation that accounts for motion and deformation.
Collectively, these contributions advance the state of the art in computer vision for surgical navigation. They demonstrate the feasibility of robust, real-time, markerless perception and control in MIS, and provide datasets and frameworks that support community-wide benchmarking and clinical translation. The results highlight the potential of AI-assisted navigation to enhance surgical precision, improve patient safety, and accelerate the adoption of intelligent robotic systems in clinical practice.
This thesis develops novel AI-driven, vision-only methods for surgical navigation, enabling accurate and robust perception without reliance on external markers or hardware. The work makes five key contributions. First, new markerless algorithms for surgical instrument pose estimation are introduced, including a graph-based framework for 5-Degree of Freedom (DoF) tracking of textureless tools and an occlusion-robust 6-DoF estimator validated on the da Vinci Research Kit (dVRK). Second, a diffusion-based stereo depth estimation framework, StereoDiffusion, is proposed to achieve temporally consistent disparity maps under the challenging visual conditions of MIS. Third, the thesis presents SurgRIPE, a large-scale dataset and benchmarking challenge for 6-DoF surgical instrument pose estimation, establishing the first public standard for this task. Fourth, an AI-assisted robotic control framework is developed, fusing vision-based estimates with robot kinematics via an Extended Kalman Filter (EKF) to enable real-time closed-loop instrument manipulation. Finally, novel methods for dynamic 3D reconstruction of deformable tissues are proposed, providing surgeons with reliable intra-operative visualisation that accounts for motion and deformation.
Collectively, these contributions advance the state of the art in computer vision for surgical navigation. They demonstrate the feasibility of robust, real-time, markerless perception and control in MIS, and provide datasets and frameworks that support community-wide benchmarking and clinical translation. The results highlight the potential of AI-assisted navigation to enhance surgical precision, improve patient safety, and accelerate the adoption of intelligent robotic systems in clinical practice.
Version
Open Access
Date Issued
2025-10-01
Date Awarded
2026-03-01
Copyright Statement
Attribution-NonCommercial 4.0 International Licence (CC BY-NC)
License URL
Advisor
Giannarou, Stamatia
Takats, Zoltan
Publisher Department
Department of Surgery & Cancer
Publisher Institution
Imperial College London
Qualification Level
Doctoral
Qualification Name
Doctor of Philosophy (PhD)