Hybrid tissue surface shape measurement and hyperspectral imaging using a multispectral structured lighting endoscope
Author(s)
Lin, Jianyu
Type
Thesis
Abstract
Surgical guidance and decision making could be improved with accurate measurement of intra-operative data including shape and hyper/multispectral information of the tissue surface. In this thesis, a dual-modality endoscopic system based on the structured light (SL) technique has been proposed to enable real-time tissue surface reconstruction and pixel-level dense multispectral imaging. The system is described in a chronological order of development for both the hardware and software aspects.
Starting from the previously developed ICL SL system that enabled tissue surface shape measurement, a naive 3D reconstruction pipeline is described. This includes a SL system calibration method, normalized cut, and local rigid registration based pattern decoding algorithm. In order to enable a faster, denser, and more robust surface reconstruction, further hardware and algorithmic improvements have been proposed, including replacing the RGB camera with a multispectral camera and stroboscopically switching between white light (WL)/SL modes. Software improvements have allowed on-the-fly calibration and reconstruction, and improved accuracy through the integration of local normal information, the use of convolutional neural networks (CNN) based pattern decoding with a specially designed feature matching algorithm, and the combination of SL and structure-from-motion (SfM).
The system has also been extended to enable sparse hyperspectral signal measurements, by integrating a slit hyperspectral camera. A CNN based model has been developed to estimate pixel-level dense multispectral images from RGB images and the captured sparse hyperspectral signals.
The whole system has been validated using phantoms and ex vivo tissue, with in vivo demonstrations in a large animal trial and in patients, demonstrating its potential in real-world clinical applications.
Starting from the previously developed ICL SL system that enabled tissue surface shape measurement, a naive 3D reconstruction pipeline is described. This includes a SL system calibration method, normalized cut, and local rigid registration based pattern decoding algorithm. In order to enable a faster, denser, and more robust surface reconstruction, further hardware and algorithmic improvements have been proposed, including replacing the RGB camera with a multispectral camera and stroboscopically switching between white light (WL)/SL modes. Software improvements have allowed on-the-fly calibration and reconstruction, and improved accuracy through the integration of local normal information, the use of convolutional neural networks (CNN) based pattern decoding with a specially designed feature matching algorithm, and the combination of SL and structure-from-motion (SfM).
The system has also been extended to enable sparse hyperspectral signal measurements, by integrating a slit hyperspectral camera. A CNN based model has been developed to estimate pixel-level dense multispectral images from RGB images and the captured sparse hyperspectral signals.
The whole system has been validated using phantoms and ex vivo tissue, with in vivo demonstrations in a large animal trial and in patients, demonstrating its potential in real-world clinical applications.
Version
Open Access
Date Issued
2018-01
Date Awarded
2018-07
Copyright Statement
Attribution NoDerivatives 4.0 International Licence (CC BY-ND)
Advisor
Elson, Daniel
Yang, Guang-Zhong
Sponsor
Imperial College London
Grant Number
ERC 242991
Publisher Department
Computing
Publisher Institution
Imperial College London
Qualification Level
Doctoral
Qualification Name
Doctor of Philosophy (PhD)