Dual-modality endoscopic probe for tissue surface shape reconstruction and hyperspectral imaging enabled by deep neural networks.

File Description SizeFormat 
MedIA2018_revision_v3 submitted.docxAccepted version2.81 MBMicrosoft WordView/Open
Title: Dual-modality endoscopic probe for tissue surface shape reconstruction and hyperspectral imaging enabled by deep neural networks.
Authors: Lin, J
Clancy, NT
Qi, J
Hu, Y
Tatla, T
Stoyanov, D
Maier-Hein, L
Elson, DS
Item Type: Journal Article
Abstract: Surgical guidance and decision making could be improved with accurate and real-time measurement of intra-operative data including shape and spectral information of the tissue surface. In this work, a dual-modality endoscopic system has been proposed to enable tissue surface shape reconstruction and hyperspectral imaging (HSI). This system centers around a probe comprised of an incoherent fiber bundle, whose fiber arrangement is different at the two ends, and miniature imaging optics. For 3D reconstruction with structured light (SL), a light pattern formed of randomly distributed spots with different colors is projected onto the tissue surface, creating artificial texture. Pattern decoding with a Convolutional Neural Network (CNN) model and a customized feature descriptor enables real-time 3D surface reconstruction at approximately 12 frames per second (FPS). In HSI mode, spatially sparse hyperspectral signals from the tissue surface can be captured with a slit hyperspectral imager in a single snapshot. A CNN based super-resolution model, namely "super-spectral-resolution" network (SSRNet), has also been developed to estimate pixel-level dense hypercubes from the endoscope cameras standard RGB images and the sparse hyperspectral signals, at approximately 2 FPS. The probe, with a 2.1 mm diameter, enables the system to be used with endoscope working channels. Furthermore, since data acquisition in both modes can be accomplished in one snapshot, operation of this system in clinical applications is minimally affected by tissue surface movement and deformation. The whole apparatus has been validated on phantoms and tissue (ex vivo and in vivo), while initial measurements on patients during laryngeal surgery show its potential in real-world clinical applications.
Issue Date: 15-Jun-2018
Date of Acceptance: 7-Jun-2018
ISSN: 1361-8415
Publisher: Elsevier
Start Page: 162
End Page: 176
Journal / Book Title: Medical Image Analysis
Volume: 48
Copyright Statement: © 2018 Elsevier Ltd. All rights reserved. This manuscript is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International
Sponsor/Funder: Commission of the European Communities
Imperial College Healthcare NHS Trust- BRC Funding
Imperial College Healthcare NHS Trust- BRC Funding
Engineering & Physical Science Research Council (E
Deutsche Forschungsgemeinschaft ( German Research
Cancer Research UK
Funder's Grant Number: 242991
RDB04 79560
Keywords: 3D reconstruction
Deep learning
Hyperspectral imaging
Intra-operative imaging
Structured light
09 Engineering
11 Medical And Health Sciences
Nuclear Medicine & Medical Imaging
Publication Status: Published
Conference Place: Netherlands
Appears in Collections:Division of Surgery
Faculty of Medicine

Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.

Creative Commonsx