Altmetric

Dual-modality endoscopic probe for tissue surface shape reconstruction and hyperspectral imaging enabled by deep neural networks.

File Description SizeFormat 
MedIA2018_revision_v3 submitted.docxFile embargoed until 15 June 20192.81 MBMicrosoft Word    Request a copy
Title: Dual-modality endoscopic probe for tissue surface shape reconstruction and hyperspectral imaging enabled by deep neural networks.
Authors: Lin, J
Clancy, NT
Qi, J
Hu, Y
Tatla, T
Stoyanov, D
Maier-Hein, L
Elson, DS
Item Type: Journal Article
Abstract: Surgical guidance and decision making could be improved with accurate and real-time measurement of intra-operative data including shape and spectral information of the tissue surface. In this work, a dual-modality endoscopic system has been proposed to enable tissue surface shape reconstruction and hyperspectral imaging (HSI). This system centers around a probe comprised of an incoherent fiber bundle, whose fiber arrangement is different at the two ends, and miniature imaging optics. For 3D reconstruction with structured light (SL), a light pattern formed of randomly distributed spots with different colors is projected onto the tissue surface, creating artificial texture. Pattern decoding with a Convolutional Neural Network (CNN) model and a customized feature descriptor enables real-time 3D surface reconstruction at approximately 12 frames per second (FPS). In HSI mode, spatially sparse hyperspectral signals from the tissue surface can be captured with a slit hyperspectral imager in a single snapshot. A CNN based super-resolution model, namely "super-spectral-resolution" network (SSRNet), has also been developed to estimate pixel-level dense hypercubes from the endoscope cameras standard RGB images and the sparse hyperspectral signals, at approximately 2 FPS. The probe, with a 2.1 mm diameter, enables the system to be used with endoscope working channels. Furthermore, since data acquisition in both modes can be accomplished in one snapshot, operation of this system in clinical applications is minimally affected by tissue surface movement and deformation. The whole apparatus has been validated on phantoms and tissue (ex vivo and in vivo), while initial measurements on patients during laryngeal surgery show its potential in real-world clinical applications.
Issue Date: 15-Jun-2018
Date of Acceptance: 7-Jun-2018
URI: http://hdl.handle.net/10044/1/61341
DOI: https://dx.doi.org/10.1016/j.media.2018.06.004
ISSN: 1361-8415
Publisher: Elsevier
Start Page: 162
End Page: 176
Journal / Book Title: Medical Image Analysis
Volume: 48
Copyright Statement: © 2018 Elsevier Ltd. All rights reserved. This manuscript is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International http://creativecommons.org/licenses/by-nc-nd/4.0/
Sponsor/Funder: Commission of the European Communities
Imperial College Healthcare NHS Trust- BRC Funding
Imperial College Healthcare NHS Trust- BRC Funding
Engineering & Physical Science Research Council (E
Deutsche Forschungsgemeinschaft ( German Research
Cancer Research UK
Funder's Grant Number: 242991
RDB04 79560
RD207
EP/N50869X/1
637960
C24523/A25147
Keywords: 3D reconstruction
Deep learning
Hyperspectral imaging
Intra-operative imaging
Structured light
Super-spectral-resolution
3D reconstruction
Deep learning
Hyperspectral imaging
Intra-operative imaging
Structured light
Super-spectral-resolution
09 Engineering
11 Medical And Health Sciences
Nuclear Medicine & Medical Imaging
Publication Status: Published
Conference Place: Netherlands
Embargo Date: 2019-06-15
Appears in Collections:Division of Surgery
Faculty of Medicine



Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.

Creative Commonsx